[03:59] *** raz has joined #kamaelia

[03:59] < raz> hi guys

[03:59] < raz> i just looked at kamaelia and the concept sticks :)

[03:59] < raz> just a few questions:

[03:59] < raz> has anyone written an app that uses/integrates with sqlalchemy? any gotchas?

[04:00] < raz> and: i assume for dealing with blocking database interaction (such as sqlalchemy) i'll inevitably need to use threadedcomponents

[04:01] < raz> i'm a bit concerned about performance (GIL), has anyone tried to turn threadedcomponent into a "processcomponent" that uses pyprocessing to spawn processes instead of threads?

[07:10] < MS-> raz: I'm not aware of anyone having using sqlalchemy with Kamaelia, but it should be a big deal. The only issue would be to put the code that uses it into a threaded component due to blocking

[07:10] < MS-> But you already clocked that :)

[07:11] < MS-> As a result you are likely to want to play with what makes sense from a component perspective in updating & accessing the DB.

[07:13] < MS-> However, writing a threaded component is little different from writing a small stub application. (A bit like a small program controlled via stdin/stdout, except you have more options, can sling python objects round & so on)

[07:13] < raz> hmm thx for the input

[07:13] < raz> well, i'm not so eager to try and to it myself as SQLAlchemy performs a lot of magic behind the scenes that could easily come back and bite me

[07:14] < raz> i was just a bit surprised that there seems to be no database support kamaelia so far, how are people persisting their stuff?

[07:16] < MS-> Not every application relates to databases :) Most of the stuff we've been doing has related to stuff that relates to either document like data or media related.

[07:16] < MS-> The closest that's had something database like so far was Macro which extracted meta data about broadcast TV from the broadcast stream.

[07:16] < MS-> But for that it was sufficient to out put that as a seperate file per program since that data was not needed to be searched

[07:16] < MS-> However if it had needed to be searched

[07:17] < MS-> then the seperate files output would've formed an ideal candidate for feeding into an "inserter" component

[07:17] < raz> hmm yea

[07:17] < MS-> To give an idea of the amount of work involved in turning a normal piece of code into a threaded component...

[07:17] < MS-> (ie very little)

[07:18] < MS-> This: http://pastebin.com/m492fdb7f is a simple pygame based program for grabbing & displaying video from a webcam

[07:19] < MS-> And this is the threadedcomponent version - with minimal change: http://pastebin.com/m2ebf49eb

[07:20] < MS-> (OK, a few more changes than there should be in terms of config)

[07:20] < MS-> Those two examples are yanked from here:

[07:20] < MS-> http://kamaelia.googlecode.com/svn/trunk/Sketches/MPS/WebCam.py

[07:20] < MS-> It steps through changing from non-component to fully componentised

[07:20] < raz> well, i fear a "sqlalchemy-component" alone would not do in my case. after all i want all (or most) of my other components to access SA objects in a transparent way.

[07:21] < MS-> If that's threadsafe to do, then that would be fine, but you're then hitting whether that's actually safe to do in sqlalchemy rather than whether it's good/bad in kamaelia :)

[07:22] < MS-> BTW, the GIL is often less of an issue that you might expect.

[07:22] *** MS- 's phone goes

[07:27] < raz> hmm

[07:39] < MS-> back

[07:40] < MS-> Essentially the way to look at kamaelia in this case, is "how can this assist with maintaining my code", rather than "how do I kamaelia-ise it"

[07:40] < MS-> kamaelia is specifically designed for assisting maintenance of systems that happen to be concurrent

[07:41] < raz> hmm yes

[07:41] < MS-> As a result if you're likely to need threads, then hopefully what our code should do is help you there.

[07:41] < raz> well, i'm looking at eventlet at the same time and wonder whether eventlet.tpool.Proxy() could help me dodge my problem here - and whether kamaelia might grow a something similar function in the future. => http://wiki.secondlife.com/wiki/Eventlet/Documentation#Integrating_Blocking_Code_with_Threads

[07:41] < MS-> a) by encouraging safe behaviours - which is what the inbox/outbox model is really about - the metaphor naturally encourages safe behaviour.

[07:41] *** MS- looks

[07:42] < MS-> b) If you do need to share things, then the basic STM code provides a mechanism for doing so

[07:43] < MS-> but regarding b) it's possible that sqlalchemy itself actually deal with any concurrency issues if its designed with threading use in mind (I've never used it)

[07:43] *** MS- still waiting for that page...

[07:43] < raz> yea it seems they just killed their wiki this minute..grmbl

[07:43] < MS-> sods law :)

[07:44] < raz> and yes, sqlalchemy is built for threading. that's part of the problem because SA objects pulled from a database may decide to perform further database operations on attribute access

[07:44] *** MS- nods

[07:45] < raz> basically i want to call and use these objects from my main-thread, but make sure that they cannot block the main thread

[07:45] < MS-> Personally I suspect that what you'd end up with in such a system is dividing it, at least conceptually, into two main chunks - the parts using sqlalchemy and parts not using it

[07:45] < raz> quite an adventure ;)

[07:45] < MS-> sounds like you want proxy objects in that case

[07:45] < raz> yup, i'll have to talk with the SA guys about this

[07:46] < MS-> I'd be interested in hearing how you deal with it BTW

[07:46] < raz> me too ;)

[07:46] < raz> it seems there are at least two nice alternatives to twisted for non-blocking network stuff (kamaelia and eventlet) but neither tackles the issue of integrating with blocking calls

[07:46] < MS-> Generally I've been slinging json encoded objects around from a to b, and so either just serialising it, or serialising with a view to data ending up in something like conchdb at some point

[07:47] < MS-> Well, our approach is "well, if you block, use a thread", which doesn't really work for you here.

[07:48] < raz> i'd prefer to avoid the "slinging around" because the conversion is fairly error prone when datatypes are involved that cannot be easily serialized to json

[07:48] < raz> or rather that cannot be easily deserialized *from* json. stuff like references to other objects...

[07:48] < MS-> Hence, that implies there's a pattern of usage we need to support, or a pattern of usage (of kamaelia) which we ought to recommend...

[07:48] *** MS- nods

[07:48] < MS-> Not all problems are the same.

[07:49] < raz> yup. databases leave nasty scars on every non-blocking architecture :\

[07:49] < MS-> what sort of data are you storing/updating, if you don't mind me asking?

[07:49] < raz> it's a gameserver that we have here. it needs to load games and userdata from the db and persist them back later (along with game-results, game states etc.)

[07:50] < MS-> OK

[07:50] < raz> we cannot really swap out SA because we use the sharding features (our data spans n physical databases) and duplicating that would turn into a maintenance nightmare immediately

[07:50] < MS-> in game as well - or just at beginning / end ?

[07:50] < MS-> I see

[07:50] < MS-> sounds like you do it in game

[07:51] < raz> only beginning/end would suffice for a while

[07:51] < MS-> If you need ingame, that sounds more problematic, and worth solving first

[07:51] < raz> well, we can live without ingame.

[07:52] < raz> right now, with twisted, we perform all db ops in a dedicated thread, convert the results to "intermediate" objects (basically identical to the SA objects but without any SA magic), use that inside the game logic, then the whole thing backwards

[07:52] < MS-> I wonder if you could treat SQL Alchemy as being like an STM store. You check out values and periodically just commit your current values.

[07:53] < raz> stm store sounds like an interesting keyword

[07:53] < MS-> Esp if they're mainly updated by a single player.

[07:53] < raz> can i find examples for that on the homepage?

[07:53] < MS-> http://www.kamaelia.org/STM

[07:54] < MS-> http://kamaelia.googlecode.com/svn/trunk/Code/Python/Axon/Examples/STM

[07:54] < raz> hmm yes that's an interesting pattern, very close to what we'd need

[07:54] < MS-> Adapting this:

[07:54] < MS-> http://kamaelia.googlecode.com/svn/trunk/Code/Python/Axon/Axon/STM.py

[07:54] < MS-> to use sqlalchemy behind the scenes could well work nicely

[07:54] < MS-> very nicely in fact

[07:54] < raz> hmm gotta look into it more, thx for the pointer

[07:54] < MS-> you're welcome :)

[07:55] < MS-> I'll make a note of the problem somewhere - I like interesting problems :)

[07:55] < raz> hehe

[07:56] < raz> it's not a new one at all

[07:56] < MS-> Indeed, but it's just one we've not had to look at before you see :)

[07:56] < raz> when you google for "sqlalchemy twisted" you'll find quite a lot of people have tackled this before

[07:56] *** MS- nods

[07:56] < MS-> I lurk on the twisted list

[07:56] < raz> if kamelia finds a decent way to deal with it then it'd definately be the first

[07:57] < MS-> I suspect that the STM approach is probably the cleanest

[07:58] < MS-> Since it gives you start of game/end of game for free as well

[07:58] < raz> hmm yes

[07:58] < MS-> The problem with starting with start of game/end of game IMO is that it won't necessarily give you in-game, which would be problematic

[07:59] < raz> in-game is not an issue for us. we need to persist the ingame state to a fast flat-file anyways because the frequency is too high for db writes.

[07:59] < MS-> k

[08:00] < raz> we persist like once per second in each game (to survive hardware failures) and that would kill our databases quickly if we sent it to them

[08:00] *** MS- nods

[08:01] < MS-> So, how often do you persist to the backend DB? Or do you do batch updates?

[08:01] < MS-> (ie rolling batch updates)

[08:02] < MS-> I suppose that's not relevant really.

[08:02] < raz> we persist only on game-start and game-end. the in-game saves are really just for resume-after-outage when we have to temporarily shutdown the gc for upgrades or maintenance.

[08:03] < MS-> Ahh, OK, very different order of magnitude

[08:03] < raz> it doesn't kill us if we lose a few games in a server crash, we've written that off as "happens so rarely, not worth the effort"

[08:03] *** Uraeus has joined #kamaelia

[08:03] < MS-> You know, I suspect still taking the same pattern as the STM code would be the cleanest

[08:03] < raz> yup i'll definately look into that.

[08:04] < raz> as said, we have a working twisted impl here, but it's butt-ugly and kamaelia looks very promising in that regard.

[08:04] < MS-> Cool :)

[08:05] < raz> one more question would be, though, if there is a processcomponent equivalent to threadedcomponent? (process instead of thread)

[08:05] < raz> right now we use pyprocssing instead of threads for the actual game logic because we have hundreds of concurrent games and threads were quite expensive memory-wise iirc

[08:06] < raz> or was it cpu contention. i'm not sure what the problem was, we converted about 6 months ago.

[08:07] < MS-> There is a process component, but at present it uses Paul Boddies pprocess code. We're planning on shifting it over to the multiprocess support in 2.6/3.0 at some point

[08:07] < raz> hmm ok

[08:07] < MS-> The nice thing is it works, but I would view it as experimental at present personally.

[08:07] < raz> yep

[08:07] < MS-> It'll probably get redone when we move to the builtin library

[08:08] < raz> ah btw i remember what the real problem was: threads didn't utilize our dual-core cpu's properly :)

[08:08] < MS-> :-)

[08:08] *** MS- nods

[08:08] < MS-> Well, the multiprocess stuff does work

[08:08] < MS-> You either have a ProcessComponent

[08:09] < MS-> or you can use a ProcessPipeline or ProcessGraphline

[08:09] < raz> hmm cool!

[08:09] < MS-> Those distribute all the components in the pipeline/graphline across multiple processes

[08:10] < MS-> ProcessGraphline is slightly more restricted than Graphline in topologies it'll handle at present than I'd like

[08:10] < MS-> But that was chosen deliberately for safety over functionality :)

[08:10] < raz> good choice imho

[08:11] < raz> well, i'll have to see if we can wrap our heads around the programming model at all anyways

[08:12] < MS-> I think the more awkward thing is probably that it can simplify things rather than be harder. That's awkward because the more experienced you are, the more likely you are to go "no, that cannot work"

[08:12] < MS-> :)

[08:12] < raz> hmm

[08:13] < MS-> I might have a look at SQL Alchemy/STM at the weekend.

[08:13] < raz> well, i'd like a few more "down to earth" examples in your otherwise excellent documentation.

[08:13] < raz> the problem with the existing samples is that they import a lot of stuff from the standard library

[08:13] < MS-> What sort of examples?

[08:13] < MS-> k

[08:13] < raz> http://www.kamaelia.org/Examples/FortuneCookieProtocolClientServer.html

[08:14] < raz> that one actually made me laugh out loud because it literally starts with: from Kamaelia.Protocol.FortuneCookieProtocol import FortuneCookieProtocol

[08:14] < raz> ;)

[08:14] < MS-> yeah, that was written a looooong time ago :)

[08:14] < raz> the UDPSimplePeer comes closest to what i want to see

[08:15] < raz> but anyways, i'll just have more digging to do

[08:15] < raz> there are tons of docs, first impression is not everything

[08:16] < MS-> This was a larger example written for someone asking on the list :

[08:16] < MS-> http://groups.google.com/group/kamaelia/msg/387c1209ca99279f

[08:16] < MS-> For some reason my blog isn't responding where the full example is

[08:17] < raz> oh yes, that looks *very* good

[08:19] < MS-> I suspect it would probably be useful to take existing systems apart and document them as examples actually

[08:19] < raz> whatever helps the cause. but ultimately newbies like me can't expect to be spoon-fed.

[08:19] < raz> the docs are already excellent as they are.

[08:20] < MS-> thanks :)

[08:21] < raz> a simple chatserver example might be interesting to provide

[08:21] < MS-> I'm actually looking to try and produce 24 or so examples this month which will be "unlocked" once per day in december as an advent calendar

[08:21] < raz> haha great!

[08:21] < MS-> The upshot of course would then be 24 examples :)

[08:21] < raz> then make a clone of the eventlet example: http://wiki.secondlife.com/wiki/Eventlet/Documentation#Eventlet_Overview

[08:21] < MS-> Chat server is a good idea

[08:22] < MS-> Aha, their wiki has come back :)

[08:22] < raz> yup, i have a heart for chat servers.i always found them the most instructive for any networking framework that i've looked at.

[08:22] < raz> i guess that's because they touch all the basics of in/out without littering the code with too much else

[08:24] < MS-> This contains a simple P2P splitting system for capturing radio off air, and then players at the edges playing it:

[08:24] < MS-> http://kamaelia.googlecode.com/svn/trunk/Sketches/MPS/Examples/LUGRadio/

[08:24] < MS-> System source: http://kamaelia.googlecode.com/svn/trunk/Sketches/MPS/Examples/LUGRadio/SimpleSwarmRadioSource.py

[08:24] < MS-> Very similar in structure to a chat server, but slightly more interesting due to going further by building a network

[08:25] < raz> hmm interesting indeed

[08:25] < raz> ah work's calling, gotta run. thanks for the enlightening conversation :) bbl!

[08:26] < MS-> http://www.slideshare.net/kamaelian/building-systems-with-kamaelia

[08:26] < MS-> details the structure from about slide 42

[08:26] < MS-> cool

[08:26] < MS-> have fun :)

[08:28] < MS-> For whenever you look at the logs (or if you do) : http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=viewpost&nodeid=1223342651

[08:45] < MS-> Simple chat server:

[08:45] < MS-> http://kamaelia.googlecode.com/svn/trunk/Sketches/MPS/Examples/SimpleChatServer.py

[08:45] < MS-> could be simpler, but this one tells you who messages are coming from

[09:25] *** ian_brasil has joined #kamaelia

[09:25] *** ian_brasil_ has joined #kamaelia

[09:31] < MS-> Blogged about this example, and created a basic graphical client for it as well

[09:31] < MS-> http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=viewpost&nodeid=1226574014

[10:25] *** raz has joined #kamaelia

[10:26] *** elvum has joined #kamaelia

[12:02] *** ian_brasil has joined #kamaelia

[12:42] *** salmon_ has joined #kamaelia

[12:56] *** salmon_ has parted #kamaelia

[14:02] *** vmlemon_ has joined #kamaelia

[14:03] < vmlemon_> Hi

[14:25] *** SamHasler has joined #kamaelia

[14:28] < vmlemon_> Hi SamHasler

[14:37] < vmlemon_> kamlogbot: dance

[14:37] Reply: does the macarena

[15:47] *** Davbo has joined #kamaelia

[15:58] *** vmlemon_ has joined #kamaelia

[16:00] < vmlemon_> Hi

[16:36] *** vmlemon_ has joined #kamaelia

[16:41] *** ian_brasil has joined #kamaelia

[17:11] < MS-> SamHasler: ping

[17:12] < SamHasler> Hi MS-

[17:13] < MS-> Hiya - saw your reply on my blog

[17:13] < MS-> I was wondering what the suggestion is ... Sounds a little, cryptic I suppose

[17:13] < MS-> Were you suggesting rewriting that plaxo script?

[17:14] < SamHasler> I don't know python, kamaelia , and I haven't looked at the script, so I can't really answer that

[17:15] < MS-> o..k..

[17:15] < MS-> I'm not sure how to interpret your comment the at all :)

[17:16] < SamHasler> I've used the script to crawl my own social graph and I've put a link to it on my openID endpoint

[17:16] < SamHasler> http://www.plaxo.com/opensocialgraph.py?url=http%3A%2F%2Fsam.haslers.info%2F&output=&verbose=1

[17:16] *** vmlemon_ waits for someone to come up with the Antisocial Graph ;)

[17:16] < MS-> I'm sure I'm being obtuse here, but I don't see how the comment relates to the post it's attached to...

[17:16] < MS-> vmlemon_: That would actually be more useful :)

[17:17] < SamHasler> I noticed it was written in python and that they mentioned it not being threaded and I thought of kamaelia, even before you made that post. I was going to mention it some time

[17:17] < MS-> Much like the "can we offend you", "if so, how much"

[17:17] < MS-> Ahhh, OK, now it makes sense

[17:17] < MS-> maybe I ought to have a random drop box for stuff

[17:18] < vmlemon_> Was just thinking of FriendFeed (albeit with the "Friends" options switched with "Enemies") ;)

[17:18] < vmlemon_> last.fm could get an "I hate this track and never want to hear it ever again!" button, too

[17:20] < MS-> Indeed. It's the one thing that does make a difference

[17:20] < MS-> People are often more than happy to listen to random stuff as long as they don't want to claw their ears out

[17:20] < vmlemon_> (I could think of a lot of music that I'd hit that button on)

[17:21] < MS-> Simply liking something doesn't tell someone what you dislike

[17:21] < vmlemon_> Very true

[17:21] < MS-> knowing negatives as well often simplifies constraint satisfaction

[17:22] < vmlemon_> You could actually get some pretty good quality results (subjective of course) by building a system that records which tracks a user dislikes

[17:22] < vmlemon_> Naturally, you'd want a system for tracking likes too

[17:23] < MS-> I've always thought there's more information in the negatives than in the positives

[17:23] < MS-> But obviously more when you have both

[17:24] < vmlemon_> An "it depends"/"so-so" option might be useful too

[17:24] < vmlemon_> Although machines obviously can't determine a person's mood

[17:26] < MS-> It's been alleged that if you have a keystroke logger that you can, but you have to calibrate based on the person.

[17:26] < MS-> Also, it does rely on a keystroke logger, which is a rather dubious thing to want to have installed :)

[17:26] *** vmlemon_ wonders what happened to dis.gust.in

[17:27] *** Davbo has joined #kamaelia

[17:27] < vmlemon_> Hi Davbo

[17:27] < MS-> evening Davbo

[17:27] < Davbo> Hey all

[17:28] *** Davbo goes to wash pots

[17:29] *** vmlemon_ wonders why he could find the story about Norio Ogha and his defective tape recorder by misspelling his name in a Google search, but can't find it when it's spelt correctly (even though it's spelt correctly in the article)

[17:51] *** vmlemon_ has joined #kamaelia

[19:23] *** vmlemon_ has joined #kamaelia

[20:24] *** vmlemon_ has joined #kamaelia

[20:31] *** Davbo has joined #kamaelia

[20:54] *** vmlemon_ has joined #kamaelia

[21:04] *** MS- has parted #kamaelia

[21:33] *** vmlemon_ has joined #kamaelia

[22:51] *** Davbo is now known as Davbop

[22:59] *** Davbop is now known as Davbo

[23:02] < Davbo> freenode staff are awesome