April 2024 - This site, and Kamaelia are being updated. There is significant work needed, and PRs are welcome.


Simple line-of-text tokeniser

This component takes a line of text and splits it into space character separated tokens. Tokens can be encapsulated with single or double quote marks, allowing spaces to appear within a token.

Example Usage

A simple pieline that takes each line you type and splits it into a list of tokens, showing you the result:

Pipeline( ConsoleReader(),
At runtime::
>>> Hello world "how are you" 'john said "hi"' "i replied "hi"" "c:\windows" end
[ 'Hello',
  'how are you',
  'john said "hi"',
  'i replied "hi"',
  'end' ]

How does it work?

lines_to_tokenlists receives individual lines of text on its "inbox" inbox. A line is converted to a list of tokens, which is sent out of its "outbox" outbox.

Space characters are treated as the token separator, however a token can be encapsulated in single or double quotes allowing space characters to appear within it.

If you need to use a quote mark or backslash within a token encapsulated by quote marks, it must be escaped by prefixing it with a backslash. Only do this if the token is encapsulated.

encapsulating quote marks are removed when the line is tokenised. Escaped backslashes and quote marks are converted to plain backslashes and quote marks.

If a producerFinished() or shutdownMicroprocess() message is received on this component's "control" inbox, then it will send it on out of its "signal" outbox and immediately terminate. It will not flush any whole lines of text that may still be buffered.


class lines_to_tokenlists(Axon.Component.component)

lines_to_tokenlists() -> new lines_to_tokenlists component.

Takes individual lines of text and separates them into white space separated tokens. Tokens can be enclosed with single or double quote marks.


  • control : Shutdown signalling
  • inbox : Individual lines of text


  • outbox : list of tokens making up the line of text
  • signal : Shutdown signalling

Methods defined here


You should be using the inbox/outbox interface, not these methods (except construction). This documentation is designed as a roadmap as to their functionalilty for maintainers and new component developers.


x.__init__(...) initializes x; see x.__class__.__doc__ for signature

lineToTokens(self, line)

linesToTokens(line) -> list of tokens.

Splits a line into individual white-space separated tokens. Tokens can be enclosed in single or double quotes to allow spaces to be used in them.

Escape backslash and single or double quotes by prefixing them with a backslash only if used within an quote encapsulated string.


Main loop.


Returns True if a shutdownMicroprocess or producerFinished message was received.


Got a problem with the documentation? Something unclear that could be clearer? Want to help improve it? Constructive criticism is very welcome - especially if you can suggest a better rewording!

Please leave you feedback here in reply to the documentation thread in the Kamaelia blog.

-- Automatic documentation generator, 05 Jun 2009 at 03:01:38 UTC/GMT