Article(s) on Test Driven Development for Science

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Article(s) on Test Driven Development for Science

Timmie
Administrator
Hello,
from many blogs, in the book "Expert Python Programming" and in the code I read
about test driven development.
It's said to help preventing future breakage or failures of the code.

Is see on major diffuculty when adopting this for science scripts:

In Science we not only have to control the program flow but also to validate the
output.
I think such a validation needs to be included in tests.

I may change something in my code and still pass tests from the software side
but the result data may be totally wrong.

Are there already concepts for such testing?

The tests I have seen so far play mostely with generated random data.

Are there presentations or articles about this?

Regards,
Timmie

_______________________________________________
SciPy-user mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Article(s) on Test Driven Development for Science

josef.pktd
On Thu, Mar 12, 2009 at 9:13 AM, Timmie <[hidden email]> wrote:

> Hello,
> from many blogs, in the book "Expert Python Programming" and in the code I read
> about test driven development.
> It's said to help preventing future breakage or failures of the code.
>
> Is see on major diffuculty when adopting this for science scripts:
>
> In Science we not only have to control the program flow but also to validate the
> output.
> I think such a validation needs to be included in tests.
>
> I may change something in my code and still pass tests from the software side
> but the result data may be totally wrong.
>
> Are there already concepts for such testing?
>
> The tests I have seen so far play mostely with generated random data.
>
> Are there presentations or articles about this?
>
> Regards,
> Timmie
>

I don't think validating the results needs much of a special
discussion besides the regular testing tools, and it would be very
field specific.

For example in stats, and similar in my other work, I use four types
of (unit) tests:

* validating special cases, where I know what the right results are
supposed to be. This is usually my first step to get the basic
mistakes fixed.
* comparison with other implementation: often I have several
implementation available to calculate the same results, e.g. in
stats.distributions, numerical integration versus explicit formula, or
unoptimized version of a function with loops and simple structure,
second version with optimized matrix algebra.
* comparison with validated numbers: e.g. comparing with results from
R, from publications or certified examples as the ones from NIST,
* using theoretical properties: the random tests in stats are based on
the statistical properties of the statistic, distribution or
estimator, either from the definition or, for example, from the law of
large numbers. If I can simulate a large enough sample or run a Monte
Carlo with enough replications, I can test that the computed results
correspond to the theoretical results.

To simplify the actual tests, I also use regression tests after
verifying the results, but regression tests don't validate the results
if they were wrong in the first place.

For big models, I have often found nothing better than visual
inspection and relying on intuition whether it looks correct. I try to
verify the individual pieces with unit tests, but whether everything
works correctly together, I don't have formally tested.

So my impression is that, validating the results in tests should just
be part of the regular testing strategy, which should not be
restricted to tests that verify whether the function runs and the
result has the correct shape and type.

Josef
_______________________________________________
SciPy-user mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Article(s) on Test Driven Development for Science

Timmie
Administrator
> I don't think validating the results needs much of a special
> discussion besides the regular testing tools, and it would be very
> field specific.
....
Thank you very much for your extensive answer!
Comming from a science and not a programmer background, I am still happy
that my code actually produces output ;-)
Therefore, using tests is still not a habit for me.
But as the code base grows, I think I cannot affort not to have.
It's simply impossible to monitor all code manually.

I hope that the following speech will be made available as PDF:

Building tests for large, untested codebases by C. Titus Brown

Covering large codebases with automated tests is a challenging and
frustrating task that can be made much easier by approaching the problem
systematically and choosing good tools. I will discuss a number of
approaches and freely available tools that can help people "tame"
pre-existing codebases with automated tests, and show the evolution of
tests for a complex codebase under these approaches and tools.
http://us.pycon.org/2009/conference/keynotes/

Kind regards,
Timmie

_______________________________________________
SciPy-user mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user
Reply | Threaded
Open this post in threaded view
|

Re: Article(s) on Test Driven Development for Science

Bruce Southey
In reply to this post by Timmie
Really can not to what Josef said.

On Thu, Mar 12, 2009 at 8:13 AM, Timmie <[hidden email]> wrote:

> Hello,
> from many blogs, in the book "Expert Python Programming" and in the code I read
> about test driven development.
> It's said to help preventing future breakage or failures of the code.
>
> Is see on major diffuculty when adopting this for science scripts:
>
> In Science we not only have to control the program flow but also to validate the
> output.
> I think such a validation needs to be included in tests.

I would presume that the code was written to give the correct results
in the first place.

>
> I may change something in my code and still pass tests from the software side
I thought that it is what you hope happens

> but the result data may be totally wrong.

In that case there are really a few reasons:
1) the test is wrong such as not allowing sufficient iterations or is
too lax in setting convergence so the test output is actually
incorrect but code would have given the correct answer if given say
more iterations.
2) your original code had a bug like bad logic
3) the algorithm does not work for that data or may need more iterations.
4) the data is work or inappropriate - may not have sufficient
information or numerical issues to do what you want.

>
> Are there already concepts for such testing?
Writing even more tests that cover any special corner cases.

>
> The tests I have seen so far play mostely with generated random data.
>
> Are there presentations or articles about this?
You might find some references under software quality.

> Regards,
> Timmie

In Numpy there is usually a request for people to write new tests for
the corner cases that get found or any unexpected behavior. I am also
sure that people will ask or file bug reports when they try various
functions and do not get the expected answer.

More fundamental are tests that really test accuracy by giving complex
or difficult problems to solve (not just to show the errors in
Microsoft Excel). I only know statistics and I know there some
datasets and models that really test software. I do not recall the
papers but one attempt is The Statistical Reference Datasets Project:
http://www.itl.nist.gov/div898/strd/

Regards
Bruce
_______________________________________________
SciPy-user mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/scipy-user