Package csb :: Package test |
[frames] | no frames]

# Package test

source code

This is a top level package, hosting the entire CSB test framework. It is divided into several major parts:

- test cases, located under csb.test.cases
- test data, in `/csb/test/data` (not a package)
- test console, in `/csb/test/app.py`

This module, csb.test, contains all the glue-code functions, classes and decorators you would need in order to write tests for CSB.

1. Configuration and Tree

   Config is a common config object shared between CSB tests. Each config instance contains properties like:

   - data: the data folder, automatically discovered and loaded in csb.test.Config.DATA at module import time
   - temp: a default temp folder, which test cases can use

   Each Config provides a convenient way to retrieve files from `/csb/test/data`. Be sure to check out Config.getTestFile and Config.getPickle. In case you need a temp file, use Config.getTempStream or have a look at csb.io.TempFile and csb.io.TempFolder.

   All test data files should be placed in the `data` folder. All test modules must be placed in the root package: csb.test.cases. There is a strict naming convention for test modules: the name of a test module should be the same as the name of the CSB API package it tests. For example, if you are writing tests for `csb/bio/io/__init__.py`, the test module must be `csb/test/cases/bio/io/__init__.py`. `csb.test.cases` is the root package of all test modules in CSB.

2. Writing Tests

   Writing a test is easy. All you need is to import csb.test and then create your own test cases, derived from csb.test.Case:

       >>> import csb.test
       >>> @csb.test.unit
           class TestSomeClass(csb.test.Case):
               def setUp(self):
                   super(TestSomeClass, self).setUp()
                   # do something with self.config here...

   In this way your test case instance is automatically equipped with a reference to the test config, so your test method can be:

       >>> @csb.test.unit
           class TestSomeClass(csb.test.Case):
               def testSomeMethod(self):
                   myDataFile = self.config.getTestFile('some.file')
                   self.assert...

   The "unit" decorator marks a test case as a collection of unit tests. All possibilities are: csb.test.unit, csb.test.functional, csb.test.custom, and csb.test.regression.

   Writing custom (a.k.a. "data", "slow", "dynamic") tests is a little bit more work. Custom tests must be functions, not classes. Basically a custom test is a function, which builds a unittest.TestSuite instance and then returns it when called without arguments.

   Regression tests are usually created in response to reported bugs. Therefore, the best practice is to mark each test method with its relevant bug ID:

       >>> @csb.test.regression
           class SomeClassRegressions(csb.test.Case)
               def testSomeFeature(self)
               """
               @see: [CSB 000XXXX]
               """
               # regression test body...

3. Style Guide:
   - name test case packages as already described
   - group tests in csb.test.Case-s and name them properly
   - prefix test methods with "test", like "testParser" - very important
   - use camelCase for methods and variables. This applies to all the code under csb.test (including test) and does not apply to the rest of the library!
   - for functional tests it's okay to define just one test method: runTest
   - for unit tests you should create more specific test names, for example: "testParseFile" - a unit test for some method called "parse_file"
   - use csb.test decorators to mark tests as unit, functional, regression, etc.
   - make every test module executable:

            if __name__ == '__main__':
                csb.test.Console()   # Discovers and runs all test cases in the module

4. Test Execution

   Test discovery is handled by `test builders` and a test runner `app`. Test builders are subclasses of AbstractTestBuilder. For every test type (unit, functional, regression, custom) there is a corresponding test builder. AnyTestBuilder is a special builder which scans for unit, regression and functional tests at the same time.

   Test builder classes inherit the following test discovery methods:

   - `loadTests` - load tests from a test namespace. Wildcard namespaces are handled by `loadAllTests`
   - `loadAllTests` - load tests from the given namespace, and from all sub-packages (recursive)
   - `loadFromFile` - load tests from an absolute file name
   - `loadMultipleTests` - calls `loadTests` for a list of namespaces and combines all loaded tests in a single suite

   Each of those return test suite objects, which can be directly executed with python's unittest runner.

   Much simpler way to execute a test suite is to use our test app (`csb/test/app.py`), which is simply an instance of csb.test.Console:

          $ python csb/test/app.py --help

   The app has two main arguments:

   - test type - tells the app which TestBuilder to use for test dicsovery ("any" triggers AnyTestBuilder, "unit" - UnitTestBuilder, etc.)
   - test namespaces - a list of "dotted" test modules, for example:

             csb.test.cases.bio.io.*   # io and sub-packages
             csb.test.cases.bio.utils  # only utils
             .                         # current module

   In addition to running the app from the command line, you can run it also programmatically by instantiating csb.test.Console. You can construct a test console object by passing a list of test namespace(s) and a test builder class to the Console's constructor.

5. Commit Policies

   Follow these guidelines when making changes to the repository:

   - no bugs in "trunk": after fixing a bug or implementing a new feature, make sure at least the default test set passes by running the test console without any arguments. This is equivalent to: app.py -t any "csb.test.cases.*". (If no test case from this set covers the affected code, create a test case first, as described in the other policies)
   - no recurrent issues: when a bug is found, first write a regression test with a proper "@see: BugID" tag in the docstring. Run the test to make sure it fails. After fixing the bug, run the test again before you commit, as required by the previous policy
   - test all new features: there should be a test case for every new feature we implement. One possible approach is to write a test case first and make sure it fails; when the new feature is ready, run the test again to make sure it passes

Warning: for compatibility reasons do NOT import and use the unittest module directly. Always import unittest from csb.test, which is guaranteed to be python 2.7+ compatible. The standard unittest under python 2.6 is missing some features, that's why csb.test will take care of replacing it with unittest2 instead.

Submodules
csb.test.app: CSB Test Runner app.

Classes
| AbstractTestBuilder This is a base class, defining a test loader which exposes the `loadTests` method.
| AnyTestBuilder Build a test suite of cases, marked as either unit, functional or regression tests.
| Attributes
| Case Base class, defining a CSB Test Case.
| Config General CSB Test Config.
| Console Build and run all tests of the specified namespace and kind.
| CustomTestBuilder Build a test suite of cases, marked as custom tests.
| FunctionalTestBuilder Build a test suite of cases, marked as functional tests.
| InvalidNamespaceError
| RegressionTestBuilder Build a test suite of cases, marked as regression tests.
| UnitTestBuilder Build a test suite of cases, marked as unit tests.

Functions
custom(function) A function decorator, used to mark functions which build custom (dynamic) test suites when called. | source code
functional(klass) A class decorator, used to label functional test cases. | source code
regression(klass) A class decorator, used to label regression test cases. | source code
skip(reason, condition=None) Mark a test case or method for skipping. | source code
unit(klass) A class decorator, used to label unit test cases. | source code

Function Details

### custom(function) | source code
A function decorator, used to mark functions which build custom (dynamic) test suites when called. Parameters: `function` (callable) - a callable object, which returns a dynamically compiled `unittest.TestSuite`

### functional(klass) | source code
A class decorator, used to label functional test cases. Parameters: `klass` (type) - a `unittest.TestCase` class type

### regression(klass) | source code
A class decorator, used to label regression test cases. Parameters: `klass` (type) - a `unittest.TestCase` class type

### skip(reason, condition=None) | source code
Mark a test case or method for skipping. Parameters: `reason` (str) - message `condition` (bool/expression) - skip only if the specified condition is True

### unit(klass) | source code
A class decorator, used to label unit test cases. Parameters: `klass` (type) - a `unittest.TestCase` class type