[06:31] *** salmon_ has joined #kamaelia

[07:10] *** vmlemon_ has joined #kamaelia

[07:23] < vmlemon_> Hi

[07:46] < Lawouach> hi

[08:20] *** musashi has joined #kamaelia

[08:20] < musashi> hello?

[08:20] < vmlemon_> Hi, musashi

[08:20] < musashi> How is going

[08:21] < vmlemon_> OK

[08:21] < musashi> I'm developing a application with Kamaelia, and have a (newbie I think) question...

[08:22] < musashi> Can I ask? :)

[08:23] < vmlemon_> Sure, you don't need to ask, in order to ask a question ;)

[08:23] < musashi> hehehehe just to be polite

[08:23] < musashi> I have a server, built with SimpleServer

[08:23] < musashi> It runs great, and tested connecting with telnet

[08:24] < musashi> but then, I want to open a connection from a client, using TCPClient

[08:24] < musashi> just to send a string and get the results

[08:24] < musashi> I'm using OneShot, but the server doesn't receive any message...

[08:24] < musashi> The code of the client is as follows:

[08:24] < musashi> Pipeline(OneShot("insultam"),

[08:24] < musashi> ConsoleEchoer(True),

[08:24] < musashi> TCPClient("localhost",clientServerTestPort),

[08:24] < musashi> ConsoleEchoer(True),

[08:24] < musashi> ).run()

[08:25] < musashi> any hints?

[08:25] < musashi> I also tried with a DataSource but without results :(

[08:26] < vmlemon_> I'm not sure, since it's a while since I last used the server stuff, but someone else probably knows

[08:27] < musashi> OK Thanks :)

[08:28] < vmlemon_> No problem

[08:29] < vmlemon_> You'll find that although people are "in" the channel, they're usually either sleeping, or busy doing other things

[08:29] < musashi> yes I figure

[08:29] < musashi> there's activity on sundays?

[08:29] < vmlemon_> Sometimes

[08:29] < musashi> okay :)

[08:35] < Lawouach> hmm

[08:35] < Lawouach> I'm afraid the ConsoleEchoer doesn't work as you might expect in this case

[08:35] < Lawouach> have you tried without them?

[08:35] < Lawouach> at least without the one after OneShot

[08:37] < musashi> Hello Lawouach

[08:37] < musashi> Yes, but let me try again...

[08:40] < musashi> No result :(

[08:40] < musashi> Seems like the server doesn't get the message

[08:41] < Lawouach> hmm

[08:41] < musashi> The server protocol starts like that:

[08:41] < musashi> class FuncionsProtocol(Axon.Component.component):

[08:41] < musashi> def main(self):

[08:41] < musashi> self.handler = funcions()

[08:41] < musashi> while not self.shutdown():

[08:41] < musashi> yield 1

[08:41] < musashi> if self.dataReady("inbox"):

[08:41] < musashi> print "rebo dades"

[08:42] < musashi> using telnet I get the print, but using TCPClient I don't

[08:42] < Lawouach> this looks okay though

[08:44] < musashi> If I replace the OneShot with a ConsoleReader it works

[08:44] < Lawouach> right

[08:45] < musashi> But I need to use OneShot or something like that...

[08:45] < Lawouach> oh right

[08:46] < Lawouach> OneShot sends to the "signal" outbox rather than "inbox"

[08:46] < Lawouach> I mean outbox

[08:46] < Lawouach> damn no

[08:46] < Lawouach> grumble I'm confusing myself :)

[08:47] < musashi> hehehe

[08:47] < musashi> Trying also with DataSource(["insultam"]) but same results :(

[08:48] < Lawouach> oh right

[08:48] < Lawouach> your use of ConsoleEchoer was fair since you were setting the forwarder parameter to True

[08:48] < Lawouach> was the message printed to the console then?

[08:50] < musashi> yes, it gets printed

[08:51] < Lawouach> damn that's weird then

[08:51] < musashi> I know :D

[08:51] < Lawouach> I'm wondering if the message doesn't get lost before the client is actually connected

[08:52] < musashi> hmmm I understand...

[08:52] < Lawouach> well it's just a guess of course

[08:56] < musashi> Tried to move the TCPClient before the pipeline: myclient = TCPClient("localhost",port).activate,

[08:56] < musashi> and Pipeline(OneShot("message"),

[08:56] < musashi> myclient,).run()

[08:56] < musashi> but no luck also :(

[08:56] < Lawouach> activate()

[08:56] < musashi> on the pipeline or the TCPClient?

[08:56] < Lawouach> the client

[08:57] < musashi> without luck also :(

[08:57] < Lawouach> grumble

[09:00] < musashi> Ah, and anytime I run a kamaelia file I get a DeprecationWarning: the sets module is deprecated

[09:00] < musashi> import sets

[09:00] < musashi> but haven't considered this important...

[09:02] < Lawouach> this isn't

[09:02] < Lawouach> you're using Python 2.6 or above but that's fine

[09:02] < Lawouach> I do to ;)

[09:04] < musashi> Well, I actually had another question... :D

[09:05] < Lawouach> sure

[09:08] < musashi> I want to use the result of that Pipeline as the parameters of another... The first pipeline returns the host and port of a server, which is used in the next Pipeline:

[09:08] < musashi> Pipeline( ConsoleReader(eol=""),

[09:08] < musashi> TCPClient("localhost", 7000),

[09:08] < musashi> ConsoleEchoer()

[09:08] < musashi> ).run()

[09:08] < musashi> (this is the second)

[09:08] < Lawouach> right

[09:09] < musashi> Do I have to create something like a "Sink" that captures the data, or is there another way to do this? ... More "pipelined"?

[09:11] < Lawouach> a pipeline doesn't return anything in its output IIRC

[09:11] < Lawouach> meaning you might want to look at graphline perhaps

[09:11] < Lawouach> or use a backplane

[09:11] < Lawouach> I don't have a solution off the top of my head

[09:12] < musashi> Ok I'll try :)

[09:15] < musashi> But as I see graphlines, it's a way of specify the connections of the components... what I was looking is something to pass the parameters of host and port to TCPClient("localhost", 7000)... I am wrong about graph?

[09:16] < MS-> That latter question relates to this: http://www.kamaelia.org/Cookbook/Carousels

[09:17] < MS-> I think you can get a race hazard in the way you use TCPClient AFAICT if they're in the same piece of code. Specifically with sending data too soon (before the server has started)

[09:18] < MS-> Though there's something odd there, which I've not seen before as well, so looking into that as well.

[09:18] < MS-> testing with telnet would naturally just work in that scenario because delay of a person resolves the race

[09:19] < musashi> I see...

[09:20] < Lawouach> hello MS-

[09:20] < MS-> Code I'm playing with: http://code.google.com/p/kamaelia/source/browse/trunk/Sketches/MPS/Examples/OneShot_Race.py

[09:20] < MS-> Lawouach: heya

[09:20] < Lawouach> stayed up last night :)

[09:20] < MS-> I did for a while, yes

[09:20] < Lawouach> found that memory leak eventually?

[09:20] < MS-> having a rummage inside code

[09:20] < Lawouach> :)

[09:20] < MS-> I've got a candidate, yes

[09:21] < MS-> It's not due to a component not deactivating itself, it's due to the gc not collecting it.

[09:21] < MS-> Which is actually pretty easy to construct test cases for

[09:21] < MS-> musashi: client at line 35 can connect to the server, and send data

[09:21] < MS-> resulting in line 19 causing this to be emitted:

[09:22] < musashi> looking at it right now :)

[09:22] < MS-> msg 127.0.0.1 50112

[09:22] < musashi> that's just what I want :D

[09:22] < MS-> I'm currently a little inclear as to why ConsoleEchoer() doesn't *appear* to get the response, but I'm sure it'll become clear

[09:22] < musashi> trying it right now

[09:22] < MS-> I suspect buffering

[09:27] < MS-> OK, looks like that's right

[09:28] < MS-> How odd that's not come to light before...

[09:29] < MS-> Updated that example to emit several messages

[09:31] < MS-> I think it must be an issue in TCPClient, for what it's worth, because I've been running servers like this for a fair while (years)

[09:33] < MS-> OK, I'm checking in a version that allows you to telnet to a python console running inside that client/server system...

[09:33] < MS-> Way that is used:

[09:34] < MS-> ~/code.google/kamaelia/trunk/Sketches/MPS/Examples> ./OneShot_Race.py

[09:34] < MS-> Then in another window:

[09:34] < MS-> ~> telnet 127.0.0.1 8765

[09:34] < MS-> Trying 127.0.0.1...

[09:34] < MS-> Connected to 127.0.0.1.

[09:34] < MS-> Escape character is '^]'.

[09:34] < MS-> >>> self

[09:34] < MS-> Component Kamaelia.Experimental.PythonInterpreter.InterpreterTransformer_39 [ inboxes : {'control': [], 'inbox': []} outboxes : {'outbox': [], 'signal': []}

[09:35] < MS-> >>> [ str(x) for x in self.scheduler.threads.keys() ]

[09:35] < MS-> which looks like...

[09:35] < MS-> http://pastebin.com/m57619c00

[09:35] < MS-> From that we can see that the responses

[09:36] < MS-> which are missing, are rather suprisingly stuck inside the Echo component

[09:37] < MS-> I wonder if the client is sleeping...

[09:38] < MS-> Oh, that's wierd, the TCPClient isn't there!

[09:39] < MS-> Oh, I think I know what the problem is... I think it's been passed on a shutdown message, and shutdown

[09:39] < MS-> because the OneShot component has sent its message and shutdown

[09:41] < MS-> And sorted

[09:42] < MS-> Checking in a version that works

[09:44] < MS-> http://code.google.com/p/kamaelia/source/browse/trunk/Sketches/MPS/Examples/OneShot_Race.py Now uses a graphline instead

[09:45] < MS-> http://kamaelia.googlecode.com/svn/trunk/Sketches/MPS/Examples/OneShot_Race.py is the same file, but for anonymous checkouts

[09:46] < MS-> The issue with that code really is that TCPClient doesn't know if it's being used in any particular way

[09:46] < MS-> For sending, receiving or as a request/response thing

[09:46] < MS-> If it's request response you generally want to build something a little higher level to deal with that

[09:47] < MS-> Since OneShot & DataSource both emit producerFinished when they've sent their data

[09:47] < MS-> Whereas what you want to do is to get TCPClient to disconnect *after* it's recieved the response

[09:48] < MS-> Given it can't know what the response is supposed to be, or when it's supposed to get it you need something else controlling it

[09:53] < musashi> Thank you MS-! I understand the problem right now. As I just wanted to emit a message and get a single response I thought OneShot would do :)

[09:59] < MS-> In lots of cases it's fine, but there's occasionally little interactions to be aware of

[09:59] < MS-> could be argued that tcpclient needs more shutdown modes for example :)

[10:08] < MS-> Probably ought to package this up as a prefab which can just be instantiated and used in any piece of code:

[10:08] < MS-> http://pastebin.com/m2000f432

[10:37] < MS-> I am rather curious though as to what is actually going on in the original base case though

[11:48] *** salmon_ has joined #kamaelia

[12:46] < MS-> OK, it's very clear now that the key cause of the problem I'm seeing is due to something retaining a reference to socketobjects and ConnectedSocketAdapters

[12:47] < MS-> which means it's almost certainly related to cleanup issues

[12:47] < MS-> Probably inside servercore or TCPServer

[12:47] < MS-> time to start using "get_refererrers" methinks :)

[12:48] < Lawouach> the memory leak or the racing issue we saw earlier today?

[12:58] < MS-> mem leak

[12:58] < MS-> race issue was more a usage issue

[12:59] < MS-> Points at a usecase which needs a standard answer really. (eg an "expect" type component)

[13:00] < MS-> Digging through this leak is quite interesting. Certainly easier this way than it would be with either standard threads or with an event type system

[13:01] < MS-> For example, I can do this:

[13:01] < MS-> y=[x for x in Z if "Connected" in x.__class__.__name__]

[13:01] < MS-> at a prompt and rummage :)

[13:05] < Lawouach> nice

[13:08] < MS-> I'm then doing stuff like this:

[13:08] < MS-> >>> y[1]

[13:08] < MS-> Component Kamaelia.Internet.ConnectedSocketAdapter.ConnectedSocketAdapter_20

[13:08] < MS-> >>> y[0]

[13:08] < MS-> Component Kamaelia.Internet.ConnectedSocketAdapter.ConnectedSocketAdapter_12

[13:08] < MS-> To find out the specific CSA id

[13:08] < MS-> and then this to look in the scheduler:

[13:09] < MS-> [str(x) for x in self.scheduler.threads.keys() if "ConnectedSock" in str(x)]

[13:09] < MS-> Which tells me ConnectedSocketAdapter_20 is the dangling one

[13:11] < Lawouach> that's excellent

[13:11] < Lawouach> and most needed :p

[13:12] < Lawouach> it'd be nice to have a tool that performs a quick health check without having to do it manually

[13:12] < MS-> True.

[13:12] < MS-> I suppose codifying this particular sort of rummaging would help there

[13:12] < Lawouach> probably yes

[13:14] < MS-> heh, there's a dictionary that is using the actual connected socket (ie socket._socketobject) as a lookup for the CSA , and *that's* not had it's entry deleted

[13:18] *** salmon_ has joined #kamaelia

[15:06] < MS-> OK, socket object count stablised.

[15:06] < MS-> :)

[15:09] < Lawouach> sweet

[15:09] < Lawouach> that's really cool

[15:25] < MS-> Still hunting CSAs though, weirdly

[15:25] < MS-> But at least one part is sorted

[15:44] < MS-> I suppose part of the issue is this is amongst the oldest code in K and as a result has the greatest cruft factor

[15:44] < MS-> When something's not broke you don't fix it :)

[16:02] *** musashi has parted #kamaelia

[16:21] *** salmon_ has joined #kamaelia

[16:47] < MS-> Woo, think I've killed it.

[16:47] < MS-> That took longer than it should

[17:25] < MS-> That's actually quite an important bugfix, assuming I've got everything :-)

[17:37] *** vmlemon_ has joined #kamaelia

[17:55] < MS-> Yay, it really does look like that I've stomped on that memory leak in the basic TCP Server code properly :-D

[18:31] *** Lawouach has joined #kamaelia

[18:43] < MS-> Not sure if those fixes are sufficient to make the greylister stable (may have issues elsewhere), but certainly makes the server (CSA, socketobject) stable.

[20:06] < MS-> This is my bugs queue that I've started:

[20:06] < MS-> http://twitter.com/kamaeliabugs

[20:07] < MS-> When something's fixed, I delete the status update.