Metadata-Version: 1.1
Name: goftests
Version: 0.2.6
Summary: Goodness of fit tests for general datatypes
Home-page: https://github.com/posterior/goftests
Author: Fritz Obermeyer
Author-email: fritz.obermeyer@gmail.com
License: Revised BSD
Description: |Build Status| |Code Quality| |Latest Version|
        
        Goftests
        ========
        
        Generic goodness of fit tests for random plain old data.
        
        Goftests is intended for unit testing random samplers that generate
        arbitrary plain-old-data, and focuses on robustness rather than
        statistical efficiency. In contrast to
        `scipy.stats <http://docs.scipy.org/doc/scipy/reference/stats.html#statistical-functions>`__
        and
        `statsmodels <http://statsmodels.sourceforge.net/stable/stats.html#goodness-of-fit-tests-and-measures>`__,
        goftests does not make assumptions on the distribution being tested, and
        requires only a simple (sample, prob) interface provided by MCMC
        samplers.
        
        Installing
        ----------
        
        ::
        
            pip install goftests
        
        Using goodness of fit tests
        ---------------------------
        
        Goftests implements generic statistical tests for Monte Carlo samplers
        that generate (sample, probability) pairs. The four basic generic tests
        are
        
        -  ``discrete_goodness_of_fit(samples, probs_dict, ...)`` - for
           discrete-valued data with most of the mass on a few values.
        
        -  ``density_goodness_of_fit(samples, probs, ...)`` - for continuously
           distributed univariate data with no discrete component.
        
        -  ``vector_density_goodness_of_fit(samples, probs, ...)`` - for
           continuously distributed multivariate data with no discrete
           component.
        
        -  ``mixed_density_goodness_of_fit(samples, probs, ...)`` - for
           discretely-indexed continuously distributed data, subsuming the all
           the other tests.
        
        Each test outputs a goodness of fit number, say ``gof``, which should be
        uniformly distributed in the interval [0,1], and which tends to fail
        with numbers close to zero. Thus to test a sampler we usually write
        
        ::
        
            TEST_FAILURE_RATE = 1e-3
        
            def test_my_sampler(count=100):
                seed_all(0)
                samples = [my_sampler.rvs() for _ in xrange(count)]
                probs = [my_sampler.pdf(x) for x in samples]
                gof = mixed_density_goodness_of_fit(samples, probs)
                assert gof > TEST_FAILURE_RATE
        
        By specifying a global ``TEST_FAILURE_RATE`` for a suite of tests, we
        can tune the number of tests expected to fail. In a large suite of
        tests, some may fail at the specified seeds despite the distribution
        being correct, typically with gof just below the threshold. In this case
        we usually increment the seed on those tests. In a suite of 1000 tests
        with ``TEST_FAILURE_RATE = 1e-3``, we should expect to have to increment
        the seed on 1 test, on average.
        
        Diagnosing common errors
        ~~~~~~~~~~~~~~~~~~~~~~~~
        
        Too few samples
        ^^^^^^^^^^^^^^^
        
        The discrete tests only look at the most-likely few values, and requires
        enough samples in the top few values to be able to use Pearson's χ2
        test. This is typically at least 100 samples.
        
        The univariate tests require around 100 samples, and the multivariate
        require at least 1000 per dimension.
        
        Too many samples
        ^^^^^^^^^^^^^^^^
        
        When testing for millions of samples or more, tests can fail due to
        numeric imprecision rather than poor distribution. We have had success
        testing with 102-105 samples.
        
        Poorly mixing Markov chains
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        
        When testing samples generated by MCMC, the samples are often correlated
        so that testing the entire chain fails. In this case, try running the
        chain for longer and subsampling only every kth step in the chain.
        
        Sticky Markov chains
        ^^^^^^^^^^^^^^^^^^^^
        
        The density tests are sensitive to Sticky Markov chains. As with poorly
        mixing chains, try running for longer.
        
        Spuriously failing tests
        ^^^^^^^^^^^^^^^^^^^^^^^^
        
        Be sure to set a deterministic seed via ``seed_all(...)`` before
        generating each dataset. In particular, some test runners run tests in a
        nondeterministic order, so setting the seed before generating each
        sample ensures deterministic behavior.
        
        Implementing new tests
        ----------------------
        
        The goodness of fit tests are mostly implemented by reduction to other
        tests, eventually reducing to the multinomial goodness of fit test which
        uses Pearson's χ2 test on each of the multinomial's bins.
        
        .. figure:: /doc/reductions.png
           :alt: Reductions
        
           Reductions
        
        To implement a new test, you can implement from scratch, reduce to
        another test in goftests, or reduce to standard tests in another package
        like
        `scipy.stats <http://docs.scipy.org/doc/scipy/reference/stats.html#statistical-functions>`__
        or
        `statsmodels <http://statsmodels.sourceforge.net/stable/stats.html#goodness-of-fit-tests-and-measures>`__.
        
        References
        ----------
        
        -  [1] Peter J Bickel, Leo Breiman (1983) "Sums of functions of nearest
           neighbor distancess, moment bounds, limit theorems and a goodness of
           fit test"
           `pdf <http://projecteuclid.org/download/pdf_1/euclid.aop/1176993668>`__
        
        -  [2] Mike Williams (2010) "How good are your fits? Unbinned
           multivariate goodness-of-fit tests in high energy physics"
           `pdf <http://arxiv.org/pdf/1006.3019v2.pdf>`__
        
        License
        -------
        
        Copyright (c) 2014 Salesforce.com, Inc. All rights reserved. Copyright
        (c) 2015 Gamelan Labs, Inc. Copyright (c) 2016 Google, Inc. Licensed
        under the Revised BSD License. See `LICENSE.txt <LICENSE.txt>`__ for
        details.
        
        .. |Build Status| image:: https://travis-ci.org/posterior/goftests.svg?branch=master
           :target: https://travis-ci.org/posterior/goftests
        .. |Code Quality| image:: http://img.shields.io/scrutinizer/g/posterior/goftests.svg
           :target: https://scrutinizer-ci.com/g/posterior/goftests/code-structure/master/hot-spots
        .. |Latest Version| image:: https://badge.fury.io/py/goftests.svg
           :target: https://pypi.python.org/pypi/goftests
        
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.3
Classifier: Programming Language :: Python :: 3.4
