[00:29] < TheBashar> @ms: I'm really hitting my head against the wall. I send a shutdownMicroprocess to a ServerCore's control channel, but it keeps answering new connections. Any idea why it might not be shutting down?

[00:30] *** eikenberry has parted #kamaelia

[01:18] *** TheBashar has parted #kamaelia

[08:47] *** Uraeus has joined #kamaelia

[11:11] *** Uraeus has joined #kamaelia

[13:03] *** TheBashar_web has joined #kamaelia

[13:24] < TheBashar_web> @ms - I boiled my program down to a small test case. Could you please take a look and tell me if you have any idea why the port listener is not shutting down?

[13:24] < TheBashar_web> http://pastebin.com/m46e39336

[13:40] < ms-afk> hiya

[13:40] < ms-afk> just taken a look

[13:41] < ms-afk> Turns out we used a different message there: serverShutdown

[13:42] < ms-afk> Which makes sense logically speaking, but as time has gone on we decided that what you've written should also work.

[13:42] < ms-afk> (part of the issue of an evolving project I guess)

[13:42] < ms-afk> What you want is:

[13:42] < ms-afk> from Kamaelia.IPC import serverShutdown

[13:42] < ms-afk> self.send(serverShutdown(),'sigkill')

[13:44] < ms-afk> I think I'll change the definition of serverShutdown though

[13:44] < ms-afk> (in a way that won't change behaviour, but make it more consistent with what you expected)

[13:50] < TheBashar_web> Thank you! Do I understand correctly that in the future shutdownMicroprocess will work similarly to serverShutdown? Or is there a subtle difference that I should understand so that I know when I _should_ use shutdownMicroprocess?

[13:54] < TheBashar_web> Also, I cannot import serverShutdown(). Did you mean simply the Axon.Ipc.shutdown() ?

[13:57] < ms-afk> >>> from Kamaelia.IPC import serverShutdown

[13:57] < ms-afk> >>>

[13:57] < ms-afk> In the future ServerCore will understand shutdownMicroprocess to mean serverShutdown

[13:59] < ms-afk> Basically at time T, we needed a means to shutdown the server, so created a message for that

[13:59] < ms-afk> At some later time T' we decided on some reusable meanings for producerFinished and shutdownMicroprocess

[13:59] < ms-afk> Since T' was later than T, it means that there are probably some places we need to go back to to ensure components understand this

[14:00] < ms-afk> But in a way that doesn't break current code.

[14:01] < TheBashar_web> :) Great, thank you, again! This worked. And I was going to ask about Axon.Ipc vs Kamaelia.IPC. But I take it that Axon are the common parts you've factored out of Kamaelia? Like the now almost common shutdownMicroprocess.

[14:02] < ms-afk> Essentially yes. The idea is to keep all the concurrency / core bits inside Axon, and the application or library specific stuff in Kamaelia.

[14:02] < ms-afk> As time has gone on it's become clearer where divisions lie

[14:02] < TheBashar_web> Ahh, I understad the state of the world again. Thanks for clearing that up.

[14:02] < ms-afk> you're welcome

[14:48] < TheBashar_web> I'm doing a little refactoring of my code. Is there any danger in giving a regular component a threaded component child?

[14:50] < ms-afk> Nope :)

[14:50] < ms-afk> A lot of thought has gone into making that sort of thing safe to do

[14:52] < TheBashar_web> Thanks. Multi-threaded can be very confusing especially when you're not real familiar with the framework.

[14:52] < TheBashar_web> Glad that most of that has been padded already.

[14:53] < ms-afk> That's the idea really - in many respects its just like working with pipelines on the unix shell

[14:54] < ms-afk> except rather than just linear shapes (stdout->stdin) you can have arbitrary shapes and rather than serialised data you pass any python object

[14:54] < ms-afk> The idea really to get back to the stage that people have with unix - after all people don't tend to think of pipelines as concurrency - just as a bunch of connected tools

[14:55] < TheBashar_web> If one can achieve that simplicity and flexibillity without sacrificing performance, than great! Sadly, it's usualy a pick 2 out of 3 thing.

[14:56] < ms-afk> Indeed.

[14:57] < ms-afk> kamaelia's model is equivalent in many respects to something like twisted, except that we use generators rather than deferreds

[14:57] < ms-afk> and that we have explicit buffers between things

[14:57] < ms-afk> Inboxes/outboxes have zero copy for delivery btw

[14:58] < ms-afk> Since when you link an inbox to an outbox, that gives the outbox storage space

[14:58] < TheBashar_web> zero copy even with threaded components, or just non-threaded?

[14:58] < ms-afk> Just non-threaded

[14:59] < TheBashar_web> Generators fit my mind a lot closer than deffereds.

[14:59] < ms-afk> There's an extra copy in the case of non-threaded->threaded

[14:59] < ms-afk> me too. I understand deferreds and have worked on a large scale piece of network software that operated that way (C++ rather than python), but I still prefer the generator approach

[15:00] < ms-afk> The explicit inboxes/outboxes thing makes it relatively easy to trace through someone else's code as well, which helps alot IMO

[15:00] < TheBashar_web> Agreed.

[15:01] < TheBashar_web> Are you enthusiastic about expanding Kamaelia with multiprocess support?

[15:01] < TheBashar_web> Or is that not something as interesting for you?

[15:02] < ms-afk> Interested yes. Not sure about enthusiastic, but definitely planning it - we do have a process based component already in experimental form for example

[15:02] < ms-afk> but that's based on Paul Boddie's pprocess library

[15:02] < ms-afk> It's sufficiently complete to work within certain constraints though

[15:02] < TheBashar_web> I'm really excited to start working with the new multiprocess library.

[15:03] < TheBashar_web> It seems straght-forward, similar to Kamaelia, and able to take advantage of multi-core more than multithreaded components.

[15:03] < TheBashar_web> I looked at pprocess before, but it didn't really fit my head as well.

[15:04] *** eikenberry has joined #kamaelia

[15:04] < ms-afk> I picked it because it was pretty small, and because Jesse's code wasn't in the core at that point

[15:04] < ms-afk> It allows things like multi-window pygame apps which is quite fun:

[15:04] < ms-afk> http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=viewpost&nodeid=1205626569

[15:05] < ms-afk> (title is a little over enthusiastic :) )

[15:06] < ms-afk> The nice thing from a kamaelia perspective though is that once you componentise it, which library is used under the hood is largely irrelevant

[15:06] < TheBashar_web> I worked with the multiprocessing library before Jesse championed it into the stdlib.

[15:06] < ms-afk> Cool

[15:06] < TheBashar_web> Are you planning to continue with pprocessing for the default kamaelia mp?

[15:07] < ms-afk> I'd prefer to move over to multiprocessing

[15:07] < ms-afk> For probably obvious reasons

[15:07] < ms-afk> pprocess was very good to get to a proof of concept stage - something which is reliable and nice

[15:07] < TheBashar_web> I may not be knowledgable enough to help, but I'd sure be happy to help test.

[15:08] < ms-afk> That'd be useful :)

[15:09] < ms-afk> The way we implemented usage last time was to create 2 new chassis components

[15:09] < ms-afk> Rather than have just Pipeline & Graphline

[15:09] < ms-afk> I created ProcessPipeline and ProcessGraphline

[15:09] < ms-afk> Which you used in an identical fashion

[15:10] < ms-afk> But ran all the sub components in separate processes.

[15:10] < TheBashar_web> Makes sense. So the components stay the same. It's just what managing process (pipe / graph) they are loaded into.

[15:10] < ms-afk> Yep

[15:10] < TheBashar_web> Would all sub-components in a PorcessPipeline be run in different process?

[15:11] < ms-afk> Yes

[15:11] < TheBashar_web> Or would all components in s ProcessPipeline be run in the same process. But different ProcessPipelines would be run in different processes?

[15:11] < ms-afk> You can mix and match

[15:11] < ms-afk> since a Pipeline is a component

[15:12] < ms-afk> so you can put pipelines in processpipelines and vice versa

[15:13] < TheBashar_web> So for two independant school buses with kids all in the same process, it would be like ProcessPipeline(Pipeline(kid1, kid2, kid3), Pipeline(kid4, kid5, kid6))?

[15:13] < TheBashar_web> I guess that makes sense.

[15:14] < TheBashar_web> But that example makes it seem like bus1 would talk to bus2

[15:14] < TheBashar_web> For completely independant buses, how would you run two completely independant pipelines in separate processes?

[15:19] < ms-afk> ProcessPipeline( A )

[15:19] < ms-afk> ProcessPipeline( B )

[15:19] < ms-afk> ProcessPipeline( C )

[15:19] < ms-afk> :)

[15:20] *** Davbo has joined #kamaelia

[15:21] < TheBashar_web> D'oh

[15:22] < Davbo> Hey all

[15:22] < Davbo> Could anyone recommend a library for working with an excel spreadsheet from Python?

[15:26] < ms-afk> pyExcelerator

[15:26] < Davbo> cheers ms-afk

[15:27] < ms-afk> TheBashar_web: It would be nice to be able to do this though:

[15:27] < ms-afk> ProcessPar( A, B, C )

[15:27] < ms-afk> for the same thing

[15:28] < ms-afk> I've been considering implementing some occam style language constructs as components

[15:28] < ms-afk> Which has the 3 main constucts of SEQ, PAR, and ALT

[15:29] < ms-afk> We already have Seq, Alt partially exists in the form of TPipe, and PAR is like a Graphline with no linkages.

[15:29] < ms-afk> But picking up those 3 forms properly would enable people to pick up a large body of knowledge from the past as well

[15:30] < ms-afk> (I always quite liked occam, but never had a good use/need for it/system for using it)

[15:31] < TheBashar_web> ProcessParallel(A, B, C) makes sense to me.

[15:32] < TheBashar_web> Not knowing occaml though, the abbreviations are a little confusing at first.

[15:32] *** ms-afk nods

[15:34] < TheBashar_web> What is ALT? Like a round-robin?

[15:35] < ms-afk> It's like a parallel if

[15:35] < Davbo> IIRC ALT is like a guarded statement you'd see in haskell or something similar

[15:35] < TheBashar_web> alternative - d'oh

[15:36] < ms-afk> You set a bunch of conditions and code - one bit of code per condition

[15:36] < ms-afk> and when a condition is ready, it fires

[15:36] < Davbo> alternative / parallel / sequence /me can remember looking them up during gsoc :)

[15:37] < TheBashar_web> I don't work with our low level software, but my company has a programming package that translates graphical sequential function charts into programmable logic controller code.

[15:37] *** ms-afk nods

[15:37] < TheBashar_web> With alt, par, and seq you could almost make a graphical programmer for kamaelia systems.

[15:38] < ms-afk> There is a nascent Compose application...

[15:38] < ms-afk> but it needs TLC

[15:38] *** Davbo goes back to pulling his hair out over parsing this excel file

[15:38] < ms-afk> Alt, par and seq could probably simplify things somewhat

[15:39] < TheBashar_web> It would be great to have, no doubt. It would just be a shame to lose any apparent simplicity.

[15:39] < ms-afk> It's in the Tools directory. Along with the AxonVisualiser

[15:40] < ms-afk> Largely as a "I want to be able to do this graphically too please" view from my perspective

[15:41] < ms-afk> I did this as a test to see if doing a more natural front end was doable:

[15:41] < ms-afk> http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=viewpost&nodeid=1214609362

[15:41] < TheBashar_web> I meant the alt and par constructs would be great to have. But I'd hate to see Axon appear to get twisted in complexity.

[15:41] < ms-afk> Which uses some gesture recognition code for links

[15:41] *** ms-afk nods

[15:42] < ms-afk> I'm always very reticent when it comes to extending Axon for that very reason

[15:42] < TheBashar_web> Sweet screenshot.

[15:43] < ms-afk> :)

[15:44] < TheBashar_web> When I said it looks like our graphical sequential function chart designer I misspoke.

[15:44] < TheBashar_web> It looks exactly like our function block diagram programmer.

[15:45] < ms-afk> :)

[15:45] < ms-afk> Many moons ago, as a student between my second and third years I worked with manchester Uni's CS department on a hardware compiler

[15:46] < ms-afk> something for taking essentially a parse tree of an HDL and generating the netlist

[15:46] < ms-afk> The thing that stuck with me was that the parse tree was part of a language that had both textual form and graphical forms which were equivalent

[15:46] < ms-afk> I've wanted something like that, which is efficient, for software for a long time :)

[15:47] < TheBashar_web> I still would prefer using a structured text representation, but the graphical visualizer/composer would be great for teaching young kids or for higher-level WBI type integration software people.

[15:47] *** ms-afk nods

[15:48] < ms-afk> It's also useful for showing managers, since you can do this:

[15:48] < ms-afk> ./AxonVisualiser.py --navelgaze

[15:48] < ms-afk> And show what's going on inside a running system

[15:48] < ms-afk> (in Tools)

[15:48] < TheBashar_web> LOL

[15:48] < ms-afk> http://www.kamaelia.org/AxonVisualiser

[15:51] < TheBashar_web> I was having some trouble understanding how things worked yesterday evening.

[15:51] < TheBashar_web> I'd like to suggest bumping up in prominence his particular page on the documentation list on the Kamaelia front-page:

[15:52] < TheBashar_web> http://www.kamaelia.org/Docs/Axon/Axon.Component.html

[15:57] < ms-afk> Where, specifically, would you find it better ?

[15:59] < TheBashar_web> Under Documentation, you have 1 - Tuturials, 2 - Cookbook, 3 - Comp Ref, 4 - Axon Ref

[15:59] < TheBashar_web> For writing new Kamaelia software, the tutorials don't help so much.

[16:00] < TheBashar_web> They're good for adapting existing non K* software.

[16:00] < TheBashar_web> For people trying to write new learn the system programs, I would order it:

[16:00] < TheBashar_web> 1 - Cookbook, 2 - Tutorials, 3 - Axon Ref, 4 - Comp Ref

[16:01] < TheBashar_web> And I would put the Axon.Component page near the very top of the cookbook page.

[16:01] < TheBashar_web> Here's how to write a component.

[16:01] < TheBashar_web> Then, here's how to wire a couple components together.

[16:02] < ms-afk> Interesting.

[16:02] < TheBashar_web> Just two cents obviosuly.

[16:02] < TheBashar_web> grain of salt and all that

[16:03] < ms-afk> It does make sense though

[16:04] < Davbo> the source & SVN bit on the front page still has loren ipsum fwiw ms-afk

[16:04] < ms-afk> It's very hard for me to be able to accurately judge what's better/worse for someone new, simply because to me it's not new :) As a result each new set of eyes is useful

[16:05] < TheBashar_web> The first cookbooks receipes were the best tutorials I could find for starting out new. But then once I needed to go outside the small box, going down the cookbook list got into the deep end of the pool.

[16:05] < TheBashar_web> The component ref was good to know what a component could and couldn't do.

[16:06] < ms-afk> It's an interesting point.

[17:16] < TheBashar_web> Are there any automated docs generated that would show the return types of functions?

[17:16] < TheBashar_web> For instance, does microprocess.activate() return anything?

[17:38] < Lawouach> it returns self

[17:38] < Lawouach> if I'm not mistaken :p

[18:24] < ms-afk> yep

[18:25] < ms-afk> Should probably be in the docs mind

[18:26] < ms-afk> (It's not mentioned on http://www.kamaelia.org/Docs/Axon/Axon.Microprocess.html)

[18:26] < ms-afk> fwiw, In the Axon docs...

[18:26] < ms-afk> The section labelled Test documentation

[18:26] < ms-afk> is generated from the test suite

[18:27] < ms-afk> each line - eg

[18:27] < TheBashar_web> I had been looking there, but it didn't seem to contain return types except for functions explictly made to return stuff (like recv)

[18:27] < ms-afk> "After being stopped a microprocess returns true to _isStopped and false to _isRunnable."

[18:27] < ms-afk> Is the docstring from the test

[18:27] < ms-afk> which details precisely what the test covers

[18:29] < ms-afk> Means we probably don't have a test to say ".activate() when called returns the same microprocess"

[18:29] < TheBashar_web> Ahh... I see.

[18:29] < TheBashar_web> I guess I should really browse the source to find that stuff.

[18:31] < ms-afk> I admit that's not the best way to find stuff.

[18:44] < TheBashar_web> The component doc shows addChildren before child.activate(). Is there any reason why I cant activate first? So just parent.addChildren(Child().activate())

[18:57] *** Davbo has joined #kamaelia

[18:58] < ms-afk> These days most components don't tend to use addChildren at all

[18:58] < ms-afk> addChildren tends to be used by components like Pipeline and Graphline whose primary purpose

[18:59] < ms-afk> is to create a bunch of linkages, activate the children and then go to sleep

[18:59] < ms-afk> ... and are awoken (due to the addChildren...) when one of the children shuts down

[19:00] < TheBashar_web> So I want to embed my TimedPump component inside my PortManager (which TimedPump triggers).

[19:00] < TheBashar_web> I could just embed it with regular encapsulation. But I thought addChildren might be more elegant.

[19:01] < ms-afk> You could do that as well. The child stuff was really put in place because we originally thought it might be useful for capturing topology information

[19:01] < ms-afk> cf the visualiser

[19:03] < TheBashar_web> I see. I'll keep it for encapsulation for now because it automates cleanup a bit.

[19:15] < TheBashar_web> Sorry, stupid question. How do I cleanly terminate everything based on some keyboard input?

[19:16] < TheBashar_web> I hooked a ConsoleReader outbox to my main component's control inbox, and that stop's it.

[19:16] < TheBashar_web> But as the docs note, console reader doesn't terminate. I even routed a shutdownMicroprocess message back to the reader's control box.

[19:17] < TheBashar_web> How do I shutdown the whol shebang (cleanly)?

[19:21] < ms-afk> If you want to shutdown the entire system, the easiest way is to ask the scheduler to shutdown.

[19:21] < ms-afk> which you can do using

[19:21] < ms-afk> self.scheduler.stop()

[19:24] < TheBashar_web> Ahh, yes! Thank you! It shutdown cleanly for the first time ever. A beautiful thing.

[19:26] < ms-afk> :)

[19:27] < ms-afk> When all the components exit, the scheduler also stops of course, but it's also handy to be able to just say "stop"

[19:27] < TheBashar_web> Yes, easier than trying to track down all the components.

[19:33] < ms-afk> There is some work slowly happening there (kinda led to the ALT, Seq, etc ideas)

[19:33] < ms-afk> specifically if a message reaches the "control" inbox of a graphline

[19:34] < ms-afk> then it's assumed to be a control message, rather than something that gets changed

[19:34] < ms-afk> and passed on to all the subcomponents of the graphline

[19:34] < ms-afk> (unless the graphlines control inbox has been rewired somewhere else)

[19:35] < ms-afk> So for those tracking where shutdown messages go is handled by the chassis

[19:36] < ms-afk> It's tempting to extend that further to see what gets made simpler

[19:37] < TheBashar_web> It would be nice if there was a PAR you could register all the disparate chains with. Then send shutdown to the PAR and watch everything cleanly cascade to a stop.

[19:37] < ms-afk> interesting idea

[20:40] < Davbo> sqlalchemy is nice

[21:01] *** TheBashar has joined #kamaelia

[21:54] < TheBashar> @ms: The TCPClient doesn't seem to have any connection timeout capability built in.

[21:54] < TheBashar> Would you recommend adding that with a cooperating timer component, or by subclassing the existing tcpclient.