Kamaelia Project Development

How do you work, how can I join in?

Since Kamaelia is exploring a new way of building software in many respects, much of our development work goes into experimenting with ideas, as well as the core codebase. We've established a simple process to help us do this. More detail below, but here are the headlines:

• Version control using SVN, courtesy of sourceforge
• Experiment in /trunk/Sketches/YOUR_INITIALS
• Main code base in /trunk/Code
• Use Branches used for substantial changes
• Merges should be done by someone else (unless the project admins give you permission)
• ... so also be prepared to merge other people's branches!

This page is largely about doing stuff, so we have a separate page about meetings, and stuff like that.

Why all this? We want to be able to explore ideas in a shared collaborative environment without imposing too many risks (bad builds, bad code, bad design decisions, etc). Hopefully you'll think this a sensible way to get things done (please tell us if you disagree) and you'll do something similar. If it works for you, please do consider asking for access to use the the /Sketches space. In practice this should all be quite light touch - despite the length of this document!

Version control

All development for Kamaelia is done under version control using the sourceforge SVN service. See our info on how to access the repository.

All file paths below are relative to the trunk which is where the bulk of development happens.

For reference, the trunk currently contains the following top level entries:

• /Sketches (dir)
• This is where development of an exploratory nature occurs. ie "we don't know what this code should look like yet".
• /Code (dir)
• This is the main codebase from which releases are periodically built.
• /Tests (dir)
• Becomes relavant after a 1.0 release and/or code maturity.
• /AUTHORS, /COPYING, /CVSROOT (dir), /Other (dir), /Website (dir)

Any re-development of code in /Code (such as large scale refactoring) should be done on a branch (see below). To make life simple for everyone, the branch should be of the entire development tree.

What do I do then?!

So what do you do?

• Experiment in /trunk/Sketches/INITIALS
• Move working things into /trunk/Code
• Solidify into something we can trust
• Add test suites
Preferably re-implement test first
• Make large swathe changes in branches called private_INITIALS_AnythingYouWant
• Be prepared to merge other people's branches
• Remember you are not allowed to merge your own branch into the trunk without the permission of the project admins - to encourage peer review, knowledge sharing and collaboration.
• If all else fails, become an agent of CHAOS :-)
• And yes, if this happens we'll with release scripts.
• Obviously this is a place to edit things and just have fun if the above process isn't your thing :-)

Read on...

Exploratory Development in /Sketches

Why sketches?

Development can be an engineering discipline - if you know your requirements you can start with a spec and work forwards. This applies to both "Big design up front" and to a lesser extent many agile development processes. Alternatively, development can be like writing or painting, where the artist will create many sketches before taking the best and integrating them into a single coherant whole. Long term developers will often experiment, writing small throwaway stubs of code to see how something might work - especially if there are no clear requirements to start off with. You can then often understand the requirements better and then work on an engineered approach.

Kamaelia's development process is designed to support the creative "we don't know what we want, so let's sketch out a few ideas" development through to the fully engineered test-driven development approach. The /Sketches area is a place to do this experimentation and prototyping.

Both viewpoints have merit, and one group may term itself hackers (not to be confused with crackers) or even artists, the other may term themselves engineers or even computer scientists. In reality of development is often a mixture of both. We therefore hope to support both approaches in a reasonable fashion.

Creating working space in /Sketches

Suppose your name is Mr Warren W. King and you want to create QT based components. You probably need to experiment first, after all you don't know exactly what you want, and you may need to change other existing components, or maybe even Axon.

First, create yourslef some working space. You have the right to create a subdirectory named your initials: /Sketches/WWK/ . So do that:

• cd Sketches
• mkdir WWK
• svn add WWK

This directory is yours to do with as you please, without clashing with other people. If WWK is gone (say Wendy Wescot-Krinchbottom took it already) just use your full name instead or something else differentiating!

In this particular case, it would make sense to create the following directories for development at that stage:

• /Sketches/WWK/QT/
• /Sketches/WWK/QT/test

You can then experiement with various implementations. You might do one implementation, or you might do a variety. They may, or may not, become full Kamaelia components or systems. The point is though, you can develop any way you like. The advantage of doing this in the shared repository is that if you abandon your work, someone else can always pick it up later on. They'll know it might not perfect or complete, and might not have been meant seriously - because it's in sketches. But by having a shared sketch book you may save someone else time later on. Obviously you also benefit from the shared code sketch book.

Suppose you're now happy that your code works and is useful, and now you want to see how it might work in the repository.

Migrating from /Sketches into /Code

This is the beginning of changing from an experimental mode to a mode where we can say "this is something we find useful". As a result we take some care in thinking about where code should go and the naming of component before they enter the repository. The guidelines are really as follows:

• As far as possible, the name of the object should as obviously as possible reflect what the thing does. For example, a component that takes other components and bolts them together/plugs them into itself is termed a chassis. This name was not the first such name that was discussed, and there were a

Please note: you might do a sketch, and put it forward to going into /Code, or equally someone else may do so. Don't be offended if the latter happens!

What's the structure of /Code ?

In order to understand the where something might fit, and what it might be named it's useful to know the current structure. This discussion will focus on the Python code tree.

Currently /Code is subdivided by language, then major projects & bindings as follows:

• /Code/CPP - This is reserved largely for work developing a C++ version of Kamaelia/Axon.
• /Code/Java - This is reserved largely for work developing a Java version of Kamaelia/Axon.
• /Code/Ruby - this does not exist as yet, but if someone were to port the ideas in Kamaelia/Axon to Ruby, their implementation would be welcome here.
• /Code/Python - the current focus of work.

/Code/Python is subdivided into the following key directories:

• /Code/Python/Axon - this contains the full distribution/development tree of the kernel of the component system
• /Code/Python/Kamaelia - this contains the full distribution/development tree of the bulk of the component system - the component system user.
• /Code/Python/Bindings - this contains a number of python bindings for C & C++ libraries that are used by a number of components. They are here because they've been developed as part of the project.
• /Code/Python/Releases/
• There's also a handful of scripts in this directory which are designed to simplify the process of building releases.

In the general case, most of the time the result of work on actual components will end up in /Code/Python/Kamaelia. For example Kamaelia.Internet.UDP.SimplePeer resides inside the file /Code/Python/Kamaelia/Kamaelia/Internet/UDP.py (at present).

Before making changes to /Code (on a shared branch - such as a default checkout), please discuss them first on either #kamaelia on freenode's IRC or on the kamaelia mailing list. You will find we are generally accommodating towards requests, but ask you to have some patience, since we might have nefarious plans you're not aware of :-)

Criterion for something going in /Code

Code that goes into /Code is expected to be a candidate for adding to the RELEASE branch at some point, but is not automatically added. Due to this expected next step, it is expected that people committing to the SVN trunk in /Code/Python/Kamaelia will discuss what they're doing either on IRC or on the mailing list, or similar. Should a process be needed, it will be invented as it becomes apparent :-) It will probably take a form similar to that used on python-dev. Project admins act as arbiters if there's a loggerhead.

The basic criterion for something going into /Code is simply "this is something we've found useful for , and would now like to try using it in other things and seeing how it fits".

However, recently we added a simple rule for things that go in /Code :

• must have a simple working example to accompany it:
• You'll probably already have this, since you will have been using it in /Sketches
• It forms a simple acceptance test
• Helps explain how to use your code to others

• should have some docstring documentation:
• see our guidelines

• should declare a global tuple referencing all classes in the file:
__kamaelia_components__ = ( BasicPeer, SimplePeer, TargettedPeer, PostboxPeer, )
• Allows documentation to be automagically picked up and added to the documeation tree.
• Makes your component available to graphical pipeline editors and other tools.
• This declaraction should (obviously?) be a top level statement and come BEFORE any if __name__ == "__main__": ...

Note: Not all components in /Code currently have an example. This is due to the development process evolving over time, and becoming more concrete about what's useful. Also, depending on how useful we percieve your component to be, we may waive this requirement in favour of seeing what can be done.

If you do NOT feel you have time to hang around and wait for us (perhaps only a few hours), please see discussion of the CHAOS branch below.

Use of SVN Branches in Kamaelia

The bulk of day to day development with a view to release occurs on the SVN trunk. It is the natural working point that new developers - especially those new to version control - start from. We prefer to have a situation where you can just get on with development.

If you want to make experimental sweeping changes to the main /Code codebase you need to use a branch - either personal one, or the CHAOS branch.

RELEASE branch

No editting at all happens on the release branch. This tracks the actual releases. It's function is to simplify the automated release build. Files are manually added to the release branch, but automatically mirrored and updated from trunk at release point.

Criterion (as of April 2006) for adding to the release branch:

• Shown to be stable
• Has an example in /Code/Python/Kamaelia/Examples
• Has pydoc style documentation of the code

Desirable when adding to the release branch:

• Full coverage test suite. Note, this is desirable. Anything added without full coverage testing is considered either alpha or beta quality. This would be expected to reside initially in a subdirectory called test. As time matures, this will move out to /Tests/Python/Kamaelia - in the same way as /Tests/Python/Axon today. Tests should NOT move into /Tests/Python/Kamaelia until they are older than 1 year, and not expected to change. The reason for this delay is to find out whether they're a mistake or not! (As a result 1 year is meant to signify "a significant time period", rather than be prescriptive)

Expected development regarding code on the RELEASE branch:

• Code added without full coverage testing is expected to gain a full coverage test suite over time. This may lead to reimplementation of the modue based on experience of using the module. Experience has shown that modules reimplemented test first after this process have a level of use & maturity which is extremely useful.

Adding something to the release branch should only be done with the knowledge of a project admin. (Hey, that's part ofthe idea of project admin after all. Whilst this may seem restrictive, it's simply to ensure a level of consistency, and in practice will often be a very short discussion.

Personal Branches

Much like having a personal namespace in /Sketches is useful, having the ability to change anything and everything in the entire code tree is extremely useful. It's also often by far the best way of demonstrating why your point is right - why your way is better than the current method or a proposed method. It's not always the best way, but sufficiently often to be worth supporting.

As was hinted above in "Version Control", if you do make a branch it should be across the whole code tree, and we have a naming mechanism to simplify things. Suppose once again your name is Warren W King. Suppose also you were the first person with those initials to start developing things in Kamaelia, so your subdirectory name in /Sketches is /Sketches/WWK/ . ie your personal "tag" is WWK.

For sake of argument, lets say you're doing integrations that require a variety of changes in the code tree in various locations, based on some changes in perspective the implementation of the QT module we described above. You might consider calling that INTEGRATION_BOOSTS . However this precludes someone else using the same name.

As a result the suggested approach is to name your personal branches as:

• private_YOURTAG_Anything_You_Like

So in this case, the branch name would be:

• private_WWK_INTEGRATION_BOOSTS

Again, personally I'd suggested something more descriptive/precise like private_WWK_QT_INTEGRATION_BOOSTS, but any branch with a name starting private_YOURTAG is yours to do as you will. This allows others to see what's happening and share ideas, and see your ideas as they develop.

There is a special rule here though: you are NOT allowed to merge a branch you create (unless all project admins agree). The reason for this is this means someone else has to merge it. This automatically gains 2 things - peer review of code, and also it means that at least 2 people understand the changes embodied in the code. Where possible, please include documentation since it's a pain to add later on!

If you don't know how to create branches, please just say it's relatively simple, but remember the rule - the branch must be across the entire tree.

Agent of CHAOS branch

If you feel the need to try out radical new ideas, AND also want to have shared development of them, please use the CHAOS branch. Co-ordinate amongst yourselves what you want to do and what the ground rules are. If you're making someone upset here, you're doing something wrong.

If the main CVS head ends up having too much control, the CHAOS branch is design to allow you to bypass that control and produce your own version that potentially has different goals from the project admin. If this idea sounds familiar, it's because it's the approach that governs the development of content on wikis, and we'd like to run the CHAOS branch in that way - you get to add things and work on things your way. If you break things, your code may get reverted by someone else, or changed by someone else.

In theory this sounds like complete anarchy and like it wouldn't work. Experience with another project has shown that if it's taken on board in the manner as described above (never ask permission on CHAOS) then you can get great creative drives and experience from trying out a variety of ideas quickly and effectively.

Note: This branch does not exist at time of writing, but this document allows for its existance. If you feel the need to use the CHAOS branch, please just create and use it.

Finally ... Some General Guiding Principles!

• Descriptions of things optimised for the 80% of programmers are preferable to those with only the top 20% will understand. If this seems "dumbing down", consider that it is often harder to describe something complex simply, than to describe something simple in a complex fashion. Remember that a key aim is to make life easier for the bulk of programmers, with a recognition that its worth optimising for maintenance.
  
• Fun descriptions, ideas and code tend to win arguments :-)
  
• You're going to get it wrong. Whether you like it or not, something you do will go wrong and be very wrong. We'd like to aim for a very high hit rate of "good" decisions, but often the only way to find out what the "right" answer is to actually try something and find out. Please don't be afraid of pointing out the emperor has new clothes. We might be embarrassed (possibly even upset) for a while, but we'd rather be told and know when we've got something wrong. We're human too after all :-) For a good discourse on why it's worth taking this risk, read Paul Arden's excellent (and brief!) read "Whatever you think, think opposite".
  
• A corrollary of this is that it might make sense occasionally to do sketches and branches of ideas that you think are really bad ideas and see how they pan out!
  
• Implementations change, but interfaces stay the same. If the interfaces change, you've written something new. ie you may add new inboxes and outboxes to components, but if you change behaviours of existing in-/out-boxes you should consider whether you're actually writing something new. If in doubt, talk to the team.
  
• Quick, simple, clear, justified decisions (But not hasty).
  
• All rules have exceptions.
  
• import this :-)
  
• Oh, and if it's not blatantly obvious, yes we'd like to allow you to check in. We have cvs_acls we could use to protect our code if we think you're acting badly, so we're not afraid to share our workspace, we'd love for you to find Kamaelia as useful as we do.