Kamaelia: Testing Framework
What is it?
NOTE/WARNING: this is a speculative project that may become "the" testing framework. It does not have that status yet.
Testing Framework aims to be a testing framework that will let
developers write automated tests of kamaelia components easily.
NOTE: this documentation is related to the first stage of the project, which has been suggested to be rewritten, so it may change completely in the near future.
Where do I get it & install it?
It is currenty available in the private_PO_Tests branch. So, step by step:
How do I use it?
- Once the system is installed, you can do the tests you want wherever you want (i.e. in your /Sketches, in /tmp, etc.).
- You need to import the class "KamTestCase" from the package Kamaelia.Testing.KamTestCase and subclass it
- In this subclass, you can use setUp and tearDown just as you would do in PyUnit.
- You can test code that has nothing to do with Axon.
- If using Axon (which is the normal thing):
- You will first create the component under test
- Then, you will call "initializeSystem" with the component under test as an argument
- Then, you will be able to put messages to inboxes with the "put" method. This is, self.put('message','inbox').
- You can also retrieve messages from the outboxes with the "get" method. You may use only a single parameter (the outbox name), or two parameter (the outbox name and the timeout in seconds). This is, self.get('outbox',timeout=2).
- You can also check that a certain outbox of a component is empty after a period of time. This is, self.assertOutboxEmpty('outbox',timeout=5)
- You can check that a component has or has not finished in a certain period of time this is, self.assertFinished(timeout=5) or self.assertNotFinished(timeout=5)
- You can use all the assert* and fail* methods of a unittest.TestCase, which are documented here, to ensure that the output of the component is the one expected
- You can skip output from a component (i.e. waiting from a certain output). In order to define the output you are waiting for, you can use a simple KamExpectMatcher.Matcher object, which just checks the == operator, or you can use KamExpectMatcher.RegexpMatcher, which uses regular expressions, or you can subclass KamExpectMatcher.Matcher for other behaviour. Then you can use it as of self.expect(matcherObject, 'outbox', timeout=5).
- You can provide mock components to avoid dealing with resources that may be difficult to handle with during testing [i.e. disk or networks usage]
- In order to provide some working examples, you can check:
- A simple example that tests a very simple component, but does all the very basic steps to test it. Here.
- A simple example that interacts with a very simple component, providing inputs and outputs. Here.
- A simple example that interacts with the same component in the previous example, but that skips some outputs using the KamExpectMatcher.* matchers. Here.
- A simple example that interacts with a component that uses another component which would be difficult to deal with and may be mocked. Here.
- There are some other (more complex) tests added:
- A clone of the first block of tests found in here can be found here.
- A couple of tests of HTTPClient can be found here.
- Note that the tests added here test the testing framework, and thus they don't rely on it (and thus they are not tested as the tests above).
- Since it relies on PyUnit, it can easily be integrated in continuous integration servers with the existing tools for integrating PyUnit on them.
- The existing tools for running the tests nightly in Kamaelia will work as expected. Since the runner used is unittest, the -v flag still works and thus you can run the tests and it will return the __doc__ and the state of the test (i.e. "Check Addition and Deletion of Inboxes ... ok").
- The existing tools used for integrating PyUnit in CruiseControl may be used "out of the box" to integrate Kamaelia tests in this continuous integration server.
- The same probably applies to other existing automated code testers (i.e. apycot)
Planned Schedule (July 7th: redefining it):
(not added Logging Framework schedule)
- June 6
- Finish Unit-testing KamPlanet using the first approach
(modify the approach as problems arise).
- Start mocking "problematic" components (i.e. those which
require network/disk/... access; file reader and http client) to test
the interaction of all the components (except for the mocked ones) of
- June 13
- Finish integration-testing KamPlanet by using these mock
- For this case (KamPlanet), add also integration testing with
real HTTP servers (i.e. using python's SimpleHTTPServer etc.).
- Start testing other kamaelia apps/libs reusing the same
approach/objects, adapting them when problems arise. Order/number of
these applications not yet decided (the most different the applications
are, the best; GUI, network...), the number mainly depends on how many
new problems raise while testing each new application.
- June 20
- (Keep testing other kamaelia apps/libs). Note that the aim of
this task is not to test apps/libs but to check that the approach being
used is good enough by testing apps/libs; a couple of apps/libs getting
tested is only a collateral benefit. But, *at this stage*, I wouldn't
worry too much if while improving the approach/framework when testing
the 3rd app/lib I break the tests on the 1st app/lib.
- June 27
- (Keep testing other kamaelia apps/libs).
- July 4
- Freeze the testing framework API and complete/clean the
tests of the tested apps/libs so that all of them work correctly with
the current approach.
- Finish code-documenting the framework and write a small
tutorial on how to use it.
- Start integrating the configuration on a continuous
integration server (i.e. CruiseControl; it should be quite easy) or similar frameworks (APyCoT).
- July 11
- Finish integrating the testing framework with a continuous
integration server (i.e. CruiseControl) or similar frameworks (APyCoT).
- Complete last week's tutorial with a simple sample app
developed using Test-Driven-Development and Continuous Integration. This
(now bigger) tutorial will assume that Kamaelia installed and basic
notions of Kamaelia; it will include a small introduction to TDD/CI; it
will only reference the installation of the CI server but it will
include its configuration for the sample app; the main point is
including a big. The simple app should use the components that were
especially difficult to work with in June.
Progress (dates related to the weekly meeting - these are the "DONE" lines)
- First approach:
- scheduler steps-based.
- It counts how many steps it takes to do a task. If it takes too many steps, the testing framework asserts that it failed.
- Advantage: when using a single processor, the results should be the same in different machines, so it becomes quite deterministic, which is very interesting when unit testing. This doesn't happen whenever you are testing components that use intrinsically not-deterministic components (such as those using the network, file system, external processes, etc.)
- Available prior to r4391 (with examples in KamPlanet tests), still available as KamPlanetTestCaseOld.py
- Second approach:
- time-based and expect command based
- Suggested by ms-
- Now it asserts that the task has failed when certain seconds have passed
- You can also assert that you wait for a certain output, discarding all the previous output. Or that the component output is empty (equivalent to expect's EOF).
- For this, you pass a Matcher subclass (which can be a Matcher -such as "I wait for a 5, or a 'hello world', or an instance of an object, for example, and it must be exactly that object-, or a RegexpMatcher -I'm waiting for something like '.*[A-Z]+' message-). In the future other Matchers could be implemented (like one comparing xmls as http://xmlunit.sf.net/ does)
- Advantage: it is more realistic than the previous approach, and more compfortable.