March 2024 - This site, and Kamaelia are being updated. There is significant work needed, and PRs are welcome.

Kamaelia - Concurrency made useful, fun

In Kamaelia you build systems from simple components that talk to each other. This speeds development, massively aids maintenance and also means you build naturally concurrent software. It's intended to be accessible by any developer, including novices. It also makes it fun :)

What sort of systems? Network servers, clients, desktop applications, pygame based games, transcode systems and pipelines, digital TV systems, spam eradicators, teaching tools, and a fair amount more :)


Use

Kamaelia based systems

There's quite a few Kamaelia systems as example applications in the repository on github. Some sample apps:

Kamaelia Grey

Spam reduction through greylisting. Kamaelia Grey is a SMTP proxy for your inbound email, rejecting email likely to be spam. Mail likely to not be spam is forwarded to your normal mail server.

Batch Transcoder

This watches directories for new images and videos to transcode to formats suitable for the web. You can think of it as the backend needed for a youtube/flickr type site. PDF support is likely to be added soon.

Whiteboard

A collaborative whiteboard. 2 or more machines share a display you can write on - either can be a server. Whiteboards are paginated, and therefore can also be used for remote presentations. Really cool with a tablet.

Macro

Records and transcodes digital TV for later viewing (ie timeshifting). It can do this for as many channels at once as you have CPU power and tuners available for it.

Kamaelia Documentation

Build & improve Kamaelia based systems

Documentation

Documentation is a core asset in Kamaelia. As of Jan 2024, this is now being significantly updated.

While this is done you are invited to look at the Component Reference for an indication of the scope of components, and the Cookbook for small examples.

Books

Two major pieces of Kamaelia documentation are booksized PDFs:

  • The Kamaelia Europython tutorial. This is an 84 page book teaching you the ideas behind Kamaelia, building your own core, building components and systems.
  • A newer >400 page API reference book, which documentation autogenerated & extracted from the codebase - covering both Axon and Kamaelia. (This is new as of Jan 2024 and a bit rough around the edges!)

Cookbook

This section contains a number of recipes showing how to build a variety of different types of Kamaelia systems. Sections include:

  • Linking components together
  • Building Network systems, clients, servers, IM, etc
  • Building Digital TV systems
  • Building Multimedia systems
  • Using Kamaelia components & subsystems in non-kamaelia systems.

More recipes always welcome.

Component Reference

Full component reference.

Axon Reference

Full reference for Axon - the core of Kamaelia.

Tutorials

Community

Kamaelia is open source, help us improve it!

Developer Central

Interested in helping out? You're more than welcome! In this area you'll find some pages which cover some areas of interest for ongoing dev, our general development process, project management process, guidelines on contributing (eg smart questions, through to code, and contributor agreements)

Get Help (Contact)

For now, please contact us by opening a github issue on the project repo.

Summer of Code

We've been involved with GSOC now for 3 years, and it's been great. One of the less obvious things about GSOC is that it's generated a wealth of docs and ideas which can be dived into here.

Source

Kamaelia's source is hosted on Github. The latest release is somewhat out of data at this point and is to be used (with caution) with Python 2.7. That release is the 1.0.12.0 release. (Y.Y.M.r). Some apps are also available in the release directory.

Note: That release is being retired as part of the 2024 update, and will be replaced with a python 3 compatible version instead. (Much of Kamaelia has worked with python 3 for a number of years now)

Downloads

Please note that these are somewhat out of date, and to be used with caution with Python 2.7. As of Jan 2024, this site and the codebase are being updated, and this process is being tracked in Kamaelia's issues. In the mean time:

Current Release 1.0.12.0 (December 30th 2010)

See also getting started)

Other releases and downloads.

A quick overview

There is a BBC R&D Whitepaper on Kamaelia.

The project grew significantly after this whitepaper was written, but it's useful to give the original context:

WHP 113: Kamaelia: highly concurrent and network systems tamed (2005)

Kamaelia is a project aimed at building large scale online media delivery systems for the long term. [...] A key aim of Kamaelia is to enable even novice programmers to create scalable and safe concurrent systems, quickly and easily.

Kamaelia is a Python library by BBC Research for concurrent programming using a simple pattern of components that send and receive data from each other. The following is an example of a system made by piping the output of one component into another:

from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.Util.Console import ConsoleReader, ConsoleEchoer

Pipeline(
    ConsoleReader(),
    ConsoleEchoer(),
).run()

Or maybe you want to build a presentation tool? (imports & setup excluded here - full example)

Graphline(
     CHOOSER = Chooser(items = files),
     IMAGE = Image(size=(800,600), position=(8,48)),
     NEXT = Button(caption="Next", msg="NEXT", position=(72,8)),
     PREVIOUS = Button(caption="Previous", msg="PREV",position=(8,8)),
     FIRST = Button(caption="First", msg="FIRST",position=(256,8)),
     LAST = Button(caption="Last", msg="LAST",position=(320,8)),
     linkages = {
        ("NEXT","outbox") : ("CHOOSER","inbox"),
        ("PREVIOUS","outbox") : ("CHOOSER","inbox"),
        ("FIRST","outbox") : ("CHOOSER","inbox"),
        ("LAST","outbox") : ("CHOOSER","inbox"),
        ("CHOOSER","outbox") : ("IMAGE","inbox"),
     }
).run()

That's all well and good, but how is a component written? What's inside it?

from Axon.Component import component
from Axon.Ipc import shutdownMicroprocess, producerFinished

class MyComponent(component):
    Inboxes = {"inbox"        : "some data in",
               "control"      : "stops the component"}
    Outboxes = {"outbox"      : "some data out",
                "signal"      : "Shutdown signal"}

    def __init__(self, **argd):
        super(MyComponent, self).__init__(**argd)

    def main(self):
        while not self.doShutdown():
            if self.dataReady("inbox"):
                data = self.recv("inbox")
                # let's echo what we received...
                self.send(data, "outbox")

            if not self.anyReady():
                self.pause()

            yield 1

    def doShutdown(self):
       if self.dataReady("control"):
            mes = self.recv("control")
            if isinstance(mes, shutdownMicroprocess) or isinstance(mes, producerFinished):
                self.send(producerFinished(), "signal")
                return True
        return False

This is the simplest form a component can take. A component:

  • is a class that inherits from Axon.Component.component
  • has inboxes and outboxes

By inheriting from Axon.Component.component you make your class usable by the Axon library which is at the core of the Kamaelia library. It allows for your class to be used with other components.

Inboxes and outboxes allow your component to be linked to and from by other components.

Then your class defines a main method that simple loop until a specific kind of message is put into the "control" inbox of the component. During the looping it checks for any inboxes and process data read from them. Eventually it yields to the Axon scheduler that goes to the next available component. By using a generator we allow the scheduler to come back to the component's loop eventually.

Note that inboxes and outboxes are pure Python dictionary hence they allow for any Python objects and are not limited to strings. The component described above is simple, complex components have many inboxes and outboxes to link to and from.

Kamaelia

Kamaelia is a library of components for all kind of tasks and topics:

For example taking the previous example we could write:

from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.Util.Consoleimport ConsoleReader, ConsoleEchoer

Pipeline(
         ConsoleReader(),
         MyComponent(),
         ConsoleEchoer(),
).run()

Pipeline is component that automatically links outboxes to inboxes of each provided component. The console components allow for reading and writing data from and to the command line. Because Pipeline is also a component itself it could in turns be used in another component.

Note that calling the run() method on a component blocks the process until it is killed. You can also simply activate a component which will then be in an active state but will run only when eventually run is called on another component.

Now that you have the basics of Kamaelia you should dive into the documentation and have fun with this library.