Simple line-of-text tokeniser
This component takes a line of text and splits it into space character separated tokens. Tokens can be encapsulated with single or double quote marks, allowing spaces to appear within a token.
A simple pieline that takes each line you type and splits it into a list of tokens, showing you the result:
Pipeline( ConsoleReader(), lines_to_tokenlists(), ConsoleEchoer() ).run()
lines_to_tokenlists receives individual lines of text on its "inbox" inbox. A line is converted to a list of tokens, which is sent out of its "outbox" outbox.
Space characters are treated as the token separator, however a token can be encapsulated in single or double quotes allowing space characters to appear within it.
If you need to use a quote mark or backslash within a token encapsulated by quote marks, it must be escaped by prefixing it with a backslash. Only do this if the token is encapsulated.
encapsulating quote marks are removed when the line is tokenised. Escaped backslashes and quote marks are converted to plain backslashes and quote marks.
If a producerFinished() or shutdownMicroprocess() message is received on this component's "control" inbox, then it will send it on out of its "signal" outbox and immediately terminate. It will not flush any whole lines of text that may still be buffered.
lines_to_tokenlists() -> new lines_to_tokenlists component.
Takes individual lines of text and separates them into white space separated tokens. Tokens can be enclosed with single or double quote marks.
Methods defined here
x.__init__(...) initializes x; see x.__class__.__doc__ for signature
linesToTokens(line) -> list of tokens.
Splits a line into individual white-space separated tokens. Tokens can be enclosed in single or double quotes to allow spaces to be used in them.
Escape backslash and single or double quotes by prefixing them with a backslash only if used within an quote encapsulated string.
Returns True if a shutdownMicroprocess or producerFinished message was received.
Got a problem with the documentation? Something unclear that could be clearer? Want to help improve it? Constructive criticism is very welcome - especially if you can suggest a better rewording!
Please leave you feedback here in reply to the documentation thread in the Kamaelia blog.
-- Automatic documentation generator, 05 Jun 2009 at 03:01:38 UTC/GMT
This is an ongoing community based development site. As a result the contents of this page is the opinions of the contributors of the pages involved not the organisations involved. Specificially, this page may contain personal views which are not the views of the BBC. (the site is powered by a wiki engine)
(C) Copyright 2008 Kamaelia Contributors, including the British Broadcasting Corporation, All Rights Reserved.