[08:24] *** Uraeus has joined #kamaelia

[09:26] *** Uraeus has joined #kamaelia

[09:45] *** Davbo has joined #kamaelia

[10:00] *** MS- has joined #kamaelia

[10:00] < MS-> greetings

[10:43] *** Davbo has joined #kamaelia

[10:48] < Davbo> Good to see activity on the mailing list

[10:50] < MS-> indeed

[10:51] < MS-> Gnat's posting on radar helped I think (there's a time correlation)

[10:51] < MS-> ( http://radar.oreilly.com/2009/02/four-short-links-27-feb-2009.html )

[10:52] < MS-> Though Gloria's interest predates that, and Steve's questions have been interesting/useful

[10:53] < MS-> I generally don't like adding in timeouts in network code if I can help it because it adds complexity to the core code. Putting in a watcher component - ala the TTL component - which the greylister does as well is nice

[10:53] < MS-> But in TCPClient's case the timeout suggested seems sensible

[10:54] < MS-> Anyway, how's things?

[10:54] < Davbo> I think he's realised that he doesn't really need that TTL anyway

[10:55] < Davbo> I'm good thanks, yourself?

[10:55] < MS-> But what I don't think he's realised is that the TTL component is reusable beyond this case :-)

[10:55] < MS-> I'm OK. Bit tired, but I think that's because it's tuesday

[10:57] < MS-> On the upside I've realised that if I time my holidays appropriately I can take 6, or maybe 7 weeks leave this year, which would be really nice.

[10:58] < MS-> (we can buy leave - sacrifice salary for leave :) - contemplating that at the moment)

[10:58] < Davbo> you can buy leave? is that a common thing?

[10:59] < MS-> dunno - essentially its a salary sacrifice really - more of a formal way of handling "I would like X days extra unpaid leave"

[11:00] < MS-> not as good as when I worked at Manchester University though

[11:01] < MS-> There was one really nice aspect of that - the university would shutdown between just before christmas , and only reopen just after the new year - giving about 10 days off

[11:01] < Davbo> I bet you could do with the leave, you always seem so busy

[11:01] < MS-> but it wasn't classed as leave

[11:02] < MS-> I managed to take 3 weeks leave *before* that shutdown when I was there my second year

[11:02] < MS-> Managed to basically get a month off work paid for something like 10 days

[11:02] < MS-> which was fun :)

[11:03] < MS-> last year was manic work wise, which is why I'm thinking about leave. Also they opened a month long window yesterday - which is when we get to decide this stuff

[11:04] < MS-> it was also manic personal wise, given the personal hassles. The combo meant I was over busy really

[11:04] < MS-> this year should be much better on both fronts.

[11:05] < Davbo> you should go for it :)

[14:38] *** MS- has parted #kamaelia

[16:19] *** eikenberry has joined #kamaelia

[21:23] *** MS- has joined #kamaelia

[21:24] < MS-> evening

[22:13] *** TheBashar has joined #kamaelia

[22:14] < TheBashar> MS-: Sorry for all the back and forth on the mailing list. I thought you were off IRC for lent.

[22:14] < TheBashar> +1 to the idea of making non-threaded component pause() semantics match the threaded components.

[22:36] < MS-> I think it will simplify things. FWIW, I think that sort of back and forth on the list is a good place for detailed discussions. IRC is good for bitty things.

[22:36] < MS-> and chattyness :)

[22:37] < MS-> I'm not off irc for lent - just things like twitter/yammer/facebook etc. Stuff that is fun, but has no major usage.

[22:37] < TheBashar> Next year I'm giving up sockets for lent.

[22:39] < MS-> heh.

[22:39] < MS-> In a way, that's what kamaelia gives me

[22:40] < MS-> I've not needed to think about this sort of low level stuff for a long time now

[22:40] < TheBashar> Hehe... in a way that's what I'd like it to give me. Darn microsoft.

[22:40] < MS-> the behaviour you're hitting is precisely because of the way you're using it.

[22:41] < TheBashar> Are you talking about not picking up on the ECONNREFUSED or the other thing with timeouts?

[22:41] < MS-> TCPClient wasn't really designed with opening lots of connections all at once in mind.

[22:41] < MS-> It was designed to play nicely with the server code, but not to worry about that being an issue

[22:42] < MS-> hence the decision to go down the "brute force" approach of retrying in a loop, rather than something more complex

[22:42] < TheBashar> Granted. But the standard socket errors detect refused connections for you right? That WSAEINVALID crud is complicating picking up on refused connections.

[22:42] < TheBashar> With silently ignored connections, I agree totally with you.

[22:43] < MS-> "But the standard socket errors detect refused connections for you right" yep

[22:43] < MS-> Windows being different for no good reason is indeed a pain

[22:43] < TheBashar> Any chance you'd be agreeable to the changes suggested in: http://pastebin.com/m7d8914a6

[22:44] < TheBashar> It works for windows. I was hoping it wouldn't break other OSs.

[22:46] < MS-> Taking lines 35 & 36 first

[22:46] < MS-> You're saying that if errno has the attribute WSAEINVAL to raise the error

[22:47] < MS-> IF the given error isn't that

[22:47] < TheBashar> Those two lines don't really help the connrefused, but it was just good cleanup.

[22:47] < TheBashar> If the chain above doesn't catch the error, and you're on windows but NOT WSAEINVALID, then before we did nothing.

[22:47] < MS-> I think that can be made clearer by doing this:

[22:48] < TheBashar> An exception needs to be raised.

[22:48] < MS-> change:

[22:48] < MS-> elif hasattr(errno, "WSAEINVAL"):

[22:48] < MS->       if errorno == errno.WSAEINVAL:

[22:48] < MS-> to

[22:48] < MS-> elif hasattr(errno, "WSAEINVAL") and errorno == errno.WSAEINVAL:

[22:48] < TheBashar> Agreed.

[22:48] < MS-> since python will short circuit cleanly on the first part.

[22:49] < MS-> Taking the former part

[22:49] < TheBashar> My only reason not to do that was if we ever used blocking sockets (timeouts) than a timed out connection would fall in the else clause I made.

[22:52] < MS-> Adding in lines 20, 23,24 looks like the wrong way to go about fixing the problem you're seeing - especially for unix platforms

[22:53] < MS-> I need to check that before making that change really.

[22:54] < TheBashar> Fair enough. But I'd really like to find some way to fix that. Repeatedly trying a a refused connection is a waste of time.

[22:54] < MS-> re-reading your mails btw

[22:54] < MS-> indeed it is

[22:55] < TheBashar> Had some get lost. Sorry if I double mailed you.

[22:55] < MS-> However, what SHOULD be happening is that when the other end refused connection this:

[22:55] < MS-> except socket.error, socket.msg:

[22:55] < MS-> should fail with a refused connection error

[22:55] < MS-> which falls through to:

[22:55] < TheBashar> But windows doesn't get that.

[22:55] < MS-> else:

[22:55] < MS-> raise socket.msg

[22:56] < TheBashar> It just goes from WSAEINVALID back to EWOULDBLOCK.

[22:56] < TheBashar> That's why I'm a bit upset at microsoft.

[22:57] < MS-> OK, for the moment, assume: any change I make to make this work/portable will be compatible with what you've got

[22:58] < MS-> May not be identical, but compatible in terms of usage

[22:58] < TheBashar> Cool, thanks. I understand the ignored connections are a lot harder to fix, but I'm glad to get the refused connections taken care of.

[22:58] < MS-> I'm going to have to check this under windows

[22:59] < MS-> The refused connections not working properly is rather bizarre

[22:59] < MS-> esp if they're reusing an error message

[22:59] < MS-> At the TCP stack level they get back a TCP RESET message which is rather different to say the least

[23:00] < TheBashar> I think they just don't save the result. The connection gets refused in the background so next time you loop around to call connect you're just making a fresh new connection.

[23:00] < TheBashar> Hence wouldblock.

[23:01] < MS-> Oh that's *NASTY*

[23:01] < TheBashar> MS says handling WSAEINVAL is needed to be "robust", but then again they say they prefer that you not using the looping connect paradigm at all.

[23:01] < TheBashar> Agreed. Nasty.

[23:02] < MS-> largely because they get it wrong by the looks of things

[23:02] < MS-> Reading that page in more detail:

[23:02] < TheBashar> I was working off: http://msdn.microsoft.com/en-us/library/ms737625(VS.85).aspx

[23:02] < MS-> "WSAEALREADY --- Note  In order to preserve backward compatibility, this error is reported as WSAEINVAL to Windows Sockets 1.1 applications that link to either Winsock.dll or Wsock32.dll."

[23:02] < MS-> *sob*

[23:06] < TheBashar> I guess I'm seeing that behavior because python is linking to an older 1.1 version of the winsock library?

[23:06] < MS-> Maybe.

[23:06] < MS-> means that different versions of windows will work differently

[23:07] < TheBashar> Well, I'm on Vista and it's behaving like the described legacy 1.1 behaviour.

[23:07] < MS-> Also, reading between the lines and your experiments, it looks like they reserve WSAECONNREFUSED for blocking sockets only

[23:07] < TheBashar> So when exactly did they fix that/

[23:07] < TheBashar> ?

[23:09] < MS-> Mind you, that said, if it fails over to EALREADY it would just mean they're changing from being different to changing to doing it the same as others

[23:11] < TheBashar> It's the fact that they silently drop the refused connection that is the really pinch though.

[23:12] < MS-> Indeed, I've just tried it in a VM manually, and as you say it's being really dumb

[23:13] < TheBashar> Glad it's not just me being crazy.

[23:17] < MS-> No, just windows being crappy

[23:18] < TheBashar> http://bugs.python.org/issue889544

[23:18] < MS-> As guessed btw, it was a speed thing

[23:18] < TheBashar> Closed ticket for moving python to the winsock 2 api off of old 1. I wonder if it was ever implemented.

[23:18] < MS-> manually repeating connects largely gives you EWOULDBLOCKs

[23:18] < MS-> doing it in a tight loop generally gives you INVALs

[23:19] < TheBashar> Is the python mhammond the same as the Kamaelia mailing list Matt Hammond?

[23:20] < MS-> No, it's a different one - did make me smile though :)

[23:20] < MS-> that one is Mark Hammond :)

[23:20] < TheBashar> I think that might be a big part of the problem, if python is still using the ancient winsock library.

[23:21] < TheBashar> Maybe that's why UDP sockets are broken for me as well.

[23:21] < MS-> no, I think this is a windows issue generally.

[23:22] < MS-> I haven't looked at the UDP thing yet, but I suspect it could be a windows specific issue as well.

[23:22] < TheBashar> I took that MS doc to mean WSAEINVALID should only be seen by clients using the ancient 1.1 sockets library.

[23:22] < MS-> the UDP code doesn't share any code with the TCP code btw

[23:22] < TheBashar> Sorry microsoft doc, not michael sparks doc. :)

[23:22] < MS-> :)

[23:22] < MS-> The WSAEINVALID is probably meaning EALREADY, which is OK

[23:23] < MS-> But the fact it doesn't result in an ECONNREFUSED is rather irritating

[23:24] < MS-> Ho hum. Well the way that gets fixed properly is by passing the socket in as a writeable socket to the selector, and waiting for it to show up as ready to write or in the exception set

[23:24] < MS-> if it shows up in the exception set connection failed, and if it shows up in the writeable it connected

[23:24] < TheBashar> I'm going to bring up task explorer and see if I can find what winsock python is loading.

[23:25] < MS-> That was kinda the more general path to follow, and will fix windows properly - for TCP, so I'll follow that through. Though it'll have to wait until tomorrow at the soonest.

[23:25] < TheBashar> Much appreciated.

[23:25] < MS-> you're welcome - thanks for taking the time to explain the problem :)

[23:26] < TheBashar> My pleasure. And sorry it took so long to narrow it down.

[23:27] < MS-> i was expecting windows to be sane. That was problem

[23:27] < MS-> #1

[23:27] < MS-> :)

[23:27] < MS-> OK, I'll sleep on this and implement that tomorrow at some point.

[23:27] < MS-> cya

[23:27] < TheBashar> LOL Yes, always a bad assimption.

[23:27] *** MS- is now known as ms-afk

[23:27] < TheBashar> Cheers