Re: [Numpy-discussion] ANN: Scipy 0.15.0 beta 1 release

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [Numpy-discussion] ANN: Scipy 0.15.0 beta 1 release

Matthew Brett
Hi,

On Sun, Nov 23, 2014 at 3:13 PM, Pauli Virtanen <[hidden email]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Dear all,
>
> We have finally finished preparing the Scipy 0.15.0 beta 1 release.
> Please try it and report any issues on the scipy-dev mailing list,
> and/or on Github.
>
> If no surprises turn up, the final release is planned on Dec 20 in
> three weeks.
>
> Source tarballs and full release notes are available at
> https://sourceforge.net/projects/scipy/files/SciPy/0.15.0b1/
> Binary installers should also be up soon.
>
> Best regards,
> Pauli Virtanen
>
>
> - --------------------------------------------
>
> SciPy 0.15.0 is the culmination of 6 months of hard work. It contains
> several new features, numerous bug-fixes, improved test coverage and
> better documentation.  There have been a number of deprecations and
> API changes in this release, which are documented below.  All users
> are encouraged to upgrade to this release, as there are a large number
> of bug-fixes and optimizations.  Moreover, our development attention
> will now shift to bug-fix releases on the 0.16.x branch, and on adding
> new features on the master branch.
>
> This release requires Python 2.6, 2.7 or 3.2-3.3 and NumPy 1.5.1 or
> greater.
>
>
> New features
> ============
>
> Linear Programming Interface
> - - ----------------------------
>
> The new function ``scipy.optimize.linprog`` provides a generic
> linear programming similar to the way ``scipy.optimize.minimize``
> provides a generic interface to nonlinear programming optimizers.
> Currently the only method supported is *simplex* which provides
> a two-phase, dense-matrix-based simplex algorithm. Callbacks
> functions are supported,allowing the user to monitor the progress
> of the algorithm.
>
> Differential_evolution, a global optimizer
> - - ------------------------------------------
>
> A new ``differential_evolution`` function is available in the
> ``scipy.optimize``
> module.  Differential Evolution is an algorithm used for finding the
> global
> minimum of multivariate functions. It is stochastic in nature (does
> not use
> gradient methods), and can search large areas of candidate space, but
> often
> requires larger numbers of function evaluations than conventional gradient
> based techniques.
>
> ``scipy.signal`` improvements
> - - -----------------------------
>
> The function ``max_len_seq`` was added, which computes a Maximum
> Length Sequence (MLS) signal.
>
> ``scipy.integrate`` improvements
> - - --------------------------------
>
> It is now possible to use ``scipy.integrate`` routines to integrate
> multivariate ctypes functions, thus avoiding callbacks to Python and
> providing better performance.
>
> ``scipy.linalg`` improvements
> - - -----------------------------
>
> Add function ``orthogonal_procrustes`` for solving the procrustes
> linear algebra problem.
>
> ``scipy.sparse`` improvements
> - - -----------------------------
>
> ``scipy.sparse.linalg.svds`` can now take a ``LinearOperator`` as its
> main input.
>
> ``scipy.special`` improvements
> - - ------------------------------
>
> Values of ellipsoidal harmonic (i.e. Lame) functions and associated
> normalization constants can be now computed using ``ellip_harm``,
> ``ellip_harm_2``, and ``ellip_normal``.
>
> New convenience functions ``entr``, ``rel_entr`` ``kl_div``,
> ``huber``, and ``pseudo_huber`` were added.
>
> ``scipy.sparse.csgraph`` improvements
> - - -------------------------------------
>
> Routines ``reverse_cuthill_mckee`` and ``maximum_bipartite_matching``
> for computing reorderings of sparse graphs were added.
>
> ``scipy.stats`` improvements
> - - ----------------------------
>
> Added a Dirichlet distribution as multivariate distribution.
>
> The new function ``scipy.stats.median_test`` computes Mood's median test.
>
> The new function ``scipy.stats.combine_pvalues`` implements Fisher's
> and Stouffer's methods for combining p-values.
>
> ``scipy.stats.describe`` returns a namedtuple rather than a tuple,
> allowing
> users to access results by index or by name.
>
> Deprecated features
> ===================
>
> The ``scipy.weave`` module is deprecated.  It was the only module
> never ported
> to Python 3.x, and is not recommended to be used for new code - use Cython
> instead.  In order to support existing code, ``scipy.weave`` has been
> packaged
> separately: `https://github.com/scipy/weave`_.  It is a pure Python
> package, and
> can easily be installed with ``pip install weave``.
>
> ``scipy.special.bessel_diff_formula`` is deprecated.  It is a private
> function,
> and therefore will be removed from the public API in a following release.
>
>
> Backwards incompatible changes
> ==============================
>
> scipy.ndimage
> - - -------------
>
> The functions ``scipy.ndimage.minimum_positions``,
> ``scipy.ndimage.maximum_positions`` and ``scipy.ndimage.extrema`` return
> positions as ints instead of floats.
>
> scipy.integrate
> - - ---------------
>
> The format of banded Jacobians in ``scipy.integrate.ode`` solvers is
> changed. Note that the previous documentation of this feature was
> erroneous.

I ran against current scipy stack packages, and get these errors:

https://travis-ci.org/MacPython/scipy-stack-osx-testing/jobs/42022929#L324

======================================================================
ERROR: test_netcdf.test_byte_gatts
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py",
line 197, in runTest
    self.test(*self.arg)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/tests/test_netcdf.py",
line 268, in test_byte_gatts
    f = netcdf_file(filename, 'w')
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/netcdf.py",
line 226, in __init__
    self.fp = open(self.filename, '%sb' % omode)
IOError: [Errno 13] Permission denied:
'/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/tests/data/g_byte_atts.nc'
======================================================================
ERROR: test_netcdf.test_open_append
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py",
line 197, in runTest
    self.test(*self.arg)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/tests/test_netcdf.py",
line 284, in test_open_append
    f = netcdf_file(filename, 'w')
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/netcdf.py",
line 226, in __init__
    self.fp = open(self.filename, '%sb' % omode)
IOError: [Errno 13] Permission denied:
'/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/tests/data/append_dat.nc'

This turns out to be because the netcdf tests are trying to write into
the scipy/io/tests/data directory:

https://github.com/scipy/scipy/pull/4198

Otherwise, it looks clean.  I uploaded the OSX wheels to sourceforge.

Cheers,

Matthew
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: [Numpy-discussion] ANN: Scipy 0.15.0 beta 1 release

David Cournapeau
Shall we consider https://github.com/scipy/scipy/issues/4168 to be a blocker (the issue arises on scipy master as well as 0.14.1) ?

On Sun, Nov 23, 2014 at 11:13 PM, Pauli Virtanen <[hidden email]> wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dear all,

We have finally finished preparing the Scipy 0.15.0 beta 1 release.
Please try it and report any issues on the scipy-dev mailing list,
and/or on Github.

If no surprises turn up, the final release is planned on Dec 20 in
three weeks.

Source tarballs and full release notes are available at
https://sourceforge.net/projects/scipy/files/SciPy/0.15.0b1/
Binary installers should also be up soon.

Best regards,
Pauli Virtanen


- --------------------------------------------

SciPy 0.15.0 is the culmination of 6 months of hard work. It contains
several new features, numerous bug-fixes, improved test coverage and
better documentation.  There have been a number of deprecations and
API changes in this release, which are documented below.  All users
are encouraged to upgrade to this release, as there are a large number
of bug-fixes and optimizations.  Moreover, our development attention
will now shift to bug-fix releases on the 0.16.x branch, and on adding
new features on the master branch.

This release requires Python 2.6, 2.7 or 3.2-3.3 and NumPy 1.5.1 or
greater.


New features
============

Linear Programming Interface
- - ----------------------------

The new function ``scipy.optimize.linprog`` provides a generic
linear programming similar to the way ``scipy.optimize.minimize``
provides a generic interface to nonlinear programming optimizers.
Currently the only method supported is *simplex* which provides
a two-phase, dense-matrix-based simplex algorithm. Callbacks
functions are supported,allowing the user to monitor the progress
of the algorithm.

Differential_evolution, a global optimizer
- - ------------------------------------------

A new ``differential_evolution`` function is available in the
``scipy.optimize``
module.  Differential Evolution is an algorithm used for finding the
global
minimum of multivariate functions. It is stochastic in nature (does
not use
gradient methods), and can search large areas of candidate space, but
often
requires larger numbers of function evaluations than conventional gradient
based techniques.

``scipy.signal`` improvements
- - -----------------------------

The function ``max_len_seq`` was added, which computes a Maximum
Length Sequence (MLS) signal.

``scipy.integrate`` improvements
- - --------------------------------

It is now possible to use ``scipy.integrate`` routines to integrate
multivariate ctypes functions, thus avoiding callbacks to Python and
providing better performance.

``scipy.linalg`` improvements
- - -----------------------------

Add function ``orthogonal_procrustes`` for solving the procrustes
linear algebra problem.

``scipy.sparse`` improvements
- - -----------------------------

``scipy.sparse.linalg.svds`` can now take a ``LinearOperator`` as its
main input.

``scipy.special`` improvements
- - ------------------------------

Values of ellipsoidal harmonic (i.e. Lame) functions and associated
normalization constants can be now computed using ``ellip_harm``,
``ellip_harm_2``, and ``ellip_normal``.

New convenience functions ``entr``, ``rel_entr`` ``kl_div``,
``huber``, and ``pseudo_huber`` were added.

``scipy.sparse.csgraph`` improvements
- - -------------------------------------

Routines ``reverse_cuthill_mckee`` and ``maximum_bipartite_matching``
for computing reorderings of sparse graphs were added.

``scipy.stats`` improvements
- - ----------------------------

Added a Dirichlet distribution as multivariate distribution.

The new function ``scipy.stats.median_test`` computes Mood's median test.

The new function ``scipy.stats.combine_pvalues`` implements Fisher's
and Stouffer's methods for combining p-values.

``scipy.stats.describe`` returns a namedtuple rather than a tuple,
allowing
users to access results by index or by name.

Deprecated features
===================

The ``scipy.weave`` module is deprecated.  It was the only module
never ported
to Python 3.x, and is not recommended to be used for new code - use Cython
instead.  In order to support existing code, ``scipy.weave`` has been
packaged
separately: `<a href="https://github.com/scipy/weave`_" target="_blank">https://github.com/scipy/weave`_.  It is a pure Python
package, and
can easily be installed with ``pip install weave``.

``scipy.special.bessel_diff_formula`` is deprecated.  It is a private
function,
and therefore will be removed from the public API in a following release.


Backwards incompatible changes
==============================

scipy.ndimage
- - -------------

The functions ``scipy.ndimage.minimum_positions``,
``scipy.ndimage.maximum_positions`` and ``scipy.ndimage.extrema`` return
positions as ints instead of floats.

scipy.integrate
- - ---------------

The format of banded Jacobians in ``scipy.integrate.ode`` solvers is
changed. Note that the previous documentation of this feature was
erroneous.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iEYEARECAAYFAlRyaf8ACgkQ6BQxb7O0pWC7XQCeNtdJD4ZNDXvFeNFs7N3KjQn6
8QkAoK3pFmhMrTwCrgkusl+fRNMboN2r
=WSpM
-----END PGP SIGNATURE-----
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/numpy-discussion


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: [Numpy-discussion] Scipy 0.15.0 beta 1 release

David Cournapeau


On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden <[hidden email]> wrote:
David Cournapeau <[hidden email]> wrote:
> Shall we consider <a
> href="https://github.com/scipy/scipy/issues/4168">https://github.com/scipy/scipy/issues/4168</a>
> to be a
> blocker (the issue arises on scipy master as well as 0.14.1) ?
>

It is really bad, but does anyone know what is really going on?

Yes, it is in the bug report.

David
 

Which changes to NumPy set this off?

Sturla

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/numpy-discussion


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: [Numpy-discussion] ANN: Scipy 0.15.0 beta 1 release

Nathaniel Smith
In reply to this post by David Cournapeau
On Tue, Nov 25, 2014 at 12:53 PM, David Cournapeau <[hidden email]> wrote:
> Shall we consider https://github.com/scipy/scipy/issues/4168 to be a blocker
> (the issue arises on scipy master as well as 0.14.1) ?

Do you think there's anything scipy can do about it? It looks like a
pure numpy bug to me.

-n

--
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: [SciPy-Dev] [Numpy-discussion] ANN: Scipy 0.15.0 beta 1 release

Nathaniel Smith
On Tue, Nov 25, 2014 at 7:07 PM, Sturla Molden <[hidden email]> wrote:

> Nathaniel Smith <[hidden email]> wrote:
>> On Tue, Nov 25, 2014 at 12:53 PM, David Cournapeau <[hidden email]> wrote:
>>> Shall we consider https://github.com/scipy/scipy/issues/4168 to be a blocker
>>> (the issue arises on scipy master as well as 0.14.1) ?
>>
>> Do you think there's anything scipy can do about it? It looks like a
>> pure numpy bug to me.
>
> I think we should assume it is not a bug in NumPy. There is nothing that
> dictates that an ndarray's buffer must be aligned. It can just as well be
> an unaligned view or wrap an external buffer. An ndarray is valud with any
> alignment, but it does not mean it can be used with intent(inout) in f2py.
>
> Then by deduction if it is not a NumPy bug it must be a SciPy bug, which
> means the bug must be the unprotected use of intent(inout) in the f2py
> wrapper. I would think it is the responsibility of SciPy to ensure that
> whatever is passed with intent(inout) in SciPy is properly aligned.

In general you have a point, but in this case the reason the array is
"unaligned" is that numpy is being overly strict about its definition
of "aligned". In fact, it's so strict that numpy itself doesn't
provide any standard and reliable way to *create* an aligned array by
this definition.

(I guess scipy could create an overallocated copy and then take a
slice at the right offset, but asking scipy to use such hacks to work
around our bugs is clearly wrong.)

-n

--
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: [SciPy-Dev] [Numpy-discussion] ANN: Scipy 0.15.0 beta 1 release

Sturla Molden-3
Nathaniel Smith <[hidden email]> wrote:

> (I guess scipy could create an overallocated copy and then take a
> slice at the right offset, but asking scipy to use such hacks to work
> around our bugs is clearly wrong.)

Theoretically this would work, but it wouldn't be nice.

Another solution is to replace malloc() with an aligned malloc. There is a
widely known one which can be used on top of malloc (just a few lines of
extra code).

On later Visual Studio versions there us also _aligned_malloc in the CRT,
but I prefer not to use it to support older compilers. On Linux and OS X
there is of course posix_memalign.

Couldn't we just ask npy_malloc to use an aligned allocator? It would just
be a tiny PR.


Sturla

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: [SciPy-Dev] [Numpy-discussion] ANN: Scipy 0.15.0 beta 1 release

Sturla Molden-3
Sturla Molden <[hidden email]> wrote:

> Couldn't we just ask npy_malloc to use an aligned allocator? It would just
> be a tiny PR.

Sorry I meant PyArray_malloc.

If we use an aligned malloc on top of PyMem_Malloc it would just be a
change of the PyArray_malloc definition in ndarraytypes.h to a slightly
more complicated macro.

Sturla

_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: [SciPy-Dev] [Numpy-discussion] ANN: Scipy 0.15.0 beta 1 release

Sturla Molden-3
On 26/11/14 00:07, Sturla Molden wrote:

> If we use an aligned malloc on top of PyMem_Malloc it would just be a
> change of the PyArray_malloc definition in ndarraytypes.h to a slightly
> more complicated macro.

Here is a suggestion for an aligned allocator:

https://github.com/numpy/numpy/issues/5312



Sturla


_______________________________________________
SciPy-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user