Monticello/ftp stopped working for me

OK guys, I think you loaded my rollover stuff...
Mea maxima culpa:

#waitForDataUntil: and #waitForDataFor: DID NOT DO THE SAME THING!

the former signals ConnectionClosed, while the later silently answer false...

The reason it works on windws is probably I did not load my change (I
used a different image...).

Strangely, I used Monticello on squeaksource without encountering this bug...

Nicolas

2009/4/29 Nicolas Cellier <[hidden email]>:
_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Could you test if this fix works:

PharoInbox/SLICE-M7343-Socket-Rollover-bug-nice.2

2009/4/29 Nicolas Cellier <[hidden email]>:
_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Does not work for me. The problem is probably different. I still get a  
'Socket status must Unconnected before opening a new connection'

Alexandre


On 29 Apr 2009, at 21:39, Nicolas Cellier wrote:

--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.






_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Maybe I'm just sleep deprived... but, the method isUnconnected looks  
odd. Perhaps it is correct, and there is a good reason for the  
implementation, but "socketHandle == nil ifTrue: [^false ]" seems  
counter-intuitive. If socketHandle is not defined, then, it seems, we  
probably not connected... and hence should answer true.

My apologies if I am not thinking clearly... or if it is simply more  
complex handle management.

Socket >>
isUnconnected
        "Return true if this socket's state is Unconnected."

        socketHandle == nil ifTrue: [^ false].
        ^ (self primSocketConnectionStatus: socketHandle) == Unconnected

-Cam

On Apr 29, 2009, at 5:00 PM, Alexandre Bergel wrote:

> Does not work for me. The problem is probably different. I still get a
> 'Socket status must Unconnected before opening a new connection'
>

Per

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Don't worry, I "get it" from the logical view. I just wondered if it  
was related to 'Socket status must Unconnected before opening a new  
connection'.

-cam

On Apr 29, 2009, at 10:27 PM, Cameron Sanders wrote:


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Er maybe someone doing the testing can stick a
Socket allInstances size inspect
in at the pointer where the exception is signaled. I think it would be  
enlightening what the value is.

On 29-Apr-09, at 7:52 PM, Cameron Sanders wrote:

> Socket status must Unconnected before opening a new
> connection

--
=
=
=
========================================================================
John M. McIntosh <[hidden email]>   Twitter:  
squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
=
=
=
========================================================================





_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Further to this I"m wondering if there is something else "clever"  
going on.

Should the finalization process run at priority 61?  See  
WeakArray>>restartFinalizationProcess
Perhaps someone can check that?

Let me ramble on...

In the past you had the event polling going on at prioirty 40 in the  
'UI process'
That would grind away...

You also had the 'event tickler' that ran every 500 ms at priority 60.

And there in the middle why the key player here, the 'WeakArray  
finalization process' at 50

In this problem area, if we consider the socket creation fails because  
the number of sockets allocated has reached the limit (aka unix limit)  
not seen in windows.
then this is because either (a) we are actualy holding onto thousands  
of sockets on purpose?
Or we have 1, and thousands of zombies that have been tenured to  
OldSpace but not yet GCed, and Unix is unhappy with us.

Now when the socket create fails that causes well another pointless  
attempt at creation (why?) but also a full GC.
The full GC will of course signal to the 'WeakArray finalization  
process' for it to gently destroy sockets.

But let's consider what's NEW here is the 'input events fetching  
process' is now running at 2x the speed of the older UI process and  
consuming 10-30% of the cpu?
I'm not sure why before it was acceptable to look every 1/50 second,  
but now it has to be every 1/100 of a second.

But it's interesting that it''s running at priority 60, which means  
it's sucking CPU away from the weak array finalization process.

Now as we know the weak array finalization process is rather cpu  
intensive, so I wonder if there is just enough CPU taken away from  
finalization process
so that it can't do enough work before the retry for the socket  
allocation leaps in and fails for a final time?

Well of course I'm not sure why the finalization process wouldn't  
finalize all the zombie sockets in one go when the full gc completes,  
but that would require some more testing...


On 29-Apr-09, at 8:09 PM, John M McIntosh wrote:

--
=
=
=
========================================================================
John M. McIntosh <[hidden email]>   Twitter:  
squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
=
=
=
========================================================================





_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Alexandre, you did mention having #port: sent to a ByteArray. It looks  
like the last line of
FTPClient>>openPassiveDataConnection should be:
        self openDataSocket: remoteHostAddress asSocketAddress port: dataPort

instead of (current code):
        self openDataSocket: remoteHostAddress port: dataPort

With that change, the system drills on down and has a primitive  
failure in #port (on a Mac).

I will add "Socket allInstances size inspect" just above the #port:  
method in Socket>>connectTo:port:.

... and the answer is SmallInteger: 7.
Anything else I should inspect in here?

-Cam
PS: I am coming at this from the Monticello/FTP side, because my code  
is on an ftp server.


On Apr 29, 2009, at 5:00 PM, Alexandre Bergel wrote:


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Where do I reduce that priority?
-cam
On Apr 29, 2009, at 11:40 PM, John M McIntosh wrote:

> But let's consider what's NEW here is the 'input events fetching
> process' is now running at 2x the speed of the older UI process and
> consuming 10-30% of the cpu?
> I'm not sure why before it was acceptable to look every 1/50 second,
> but now it has to be every 1/100 of a second.
>
> But it's interesting that it''s running at priority 60, which means
> it's sucking CPU away from the weak array finalization process.


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

On Wed, Apr 29, 2009 at 08:40:37PM -0700, John M McIntosh wrote:
>
> In this problem area, if we consider the socket creation fails because  
> the number of sockets allocated has reached the limit (aka unix limit)  
> not seen in windows.
> then this is because either (a) we are actualy holding onto thousands  
> of sockets on purpose?
> Or we have 1, and thousands of zombies that have been tenured to  
> OldSpace but not yet GCed, and Unix is unhappy with us.

I don't know if this is available on OS X, but on some unices (Linux
for sure), a good way to check this out is by looking at the open
file descriptors listed in the /proc file system. So for example if
you have Squeak running in a process with pid 1234, you can see the
open file descriptors with "$ ls -l /proc/1234/fd/*".

> Now when the socket create fails that causes well another pointless  
> attempt at creation (why?) but also a full GC.
> The full GC will of course signal to the 'WeakArray finalization  
> process' for it to gently destroy sockets.

And of course relying on finalization to close file/socket handles
is a sure-fire way to get into trouble. So if you find lots of open
file descriptors in /proc/1234/fd/* that do not correspond to
anything being used in the image, then yes you probably do have
this kind of issue. On the other hand, if you can see that the
number of open file descriptors is well below 1024 (or whatever
the limit is on your platform) then the problem lies elsewhere.

Dave
 

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

The Socket allInstances size inspect
should go in the exception handler block before it throws the  
execption you don't want to see.


On 29-Apr-09, at 8:58 PM, Cameron Sanders wrote:

--
=
=
=
========================================================================
John M. McIntosh <[hidden email]>   Twitter:  
squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
=
=
=
========================================================================





_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Thanks Johan for such a deep insight.
You're probably very close to the truth. I modified the priority of  
the finalization process to 61. The socket problem just vanished.

It is hard to see whether this is a hack or this process should be at  
a higher priority  by design. I googled a bit about finalization and  
priority processes to see whether java people have a similar issue,  
but I haven't find something relevant yet.

Cheers,
Alexandre



On 30 Apr 2009, at 05:40, John M McIntosh wrote:

--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.






_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Alexandre Bergel wrote:
> Thanks Johan for such a deep insight.
> You're probably very close to the truth. I modified the priority of  
> the finalization process to 61. The socket problem just vanished.

Great news!

And a big thanks to John for the in depth analysis!

So, should we just change the priority by default? What could be
possible side effects?

Michael



_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Hi Cam,

The problem is likely to be the one pointed out by John

Alexandre


On 30 Apr 2009, at 05:58, Cameron Sanders wrote:

--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.






_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Alexandre Bergel wrote:
> Hi Cam,
>
> The problem is likely to be the one pointed out by John

yes and no.
The problem with the ftp is probably a porting problem to the new socket
code.
Will take a look

Michael

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

I checked again, and apparently setting the priority of the  
finalization process to 61 does _not_ solve the socket problem.
Maybe I did a wrong manipulation previously.

Grrr...

Alexandre


On 30 Apr 2009, at 05:40, John M McIntosh wrote:

--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.






_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

I would like to know why does the system rely on finalisation to  
release socket
Apparently david mentioned that this is the source of huge problems.
So what would be the alternative?

Stef

On Apr 30, 2009, at 12:00 PM, Michael Rueger wrote:


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

> I don't know if this is available on OS X, but on some unices (Linux
> for sure), a good way to check this out is by looking at the open file
> descriptors listed in the /proc file system. So for example if you
> have Squeak running in a process with pid 1234, you can see the open
> file descriptors with "$ ls -l /proc/1234/fd/*".

Nope, no /proc on OS/X, but lsof works here, as on other unices, so
continuing your example:

           lsof|awk '1234==$2{print}'

will print info about all open files for process 1234.

../Dave

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Well,  "Socket allInstances size inspect" is giving me numbers like 8,  
6, or 2 from the "ScriptLoader loadOBAlpha" do-it. When I look at the  
process on the Mac (with the activity monitor), it says I have "Ports:  
91". A look at "open files and ports" shows a lengthy list of  
filenames/devices that are open, but only about 35 show up. There are  
a few surprises (to me) in the open files list:

/  -- ??
/Library/Fonts/... -- several of these ... do we really hold the font  
files open? and
/System/Library/Fonts/...
...
/usr/lib/dyld
/private/var/db/dyld/dyld_shared_cache_i386
/dev/null -- i've been out of C/sys programming for a while (15+  
years) and forget why/when we use this -- pipes?
...
*:http-alt
---

using  "lsof | grep Squeak" I get the same list (down below) -- note  
that I generated this list several times while "ScriptLoader  
loadOBAlpha" was running, and it added only one line to the output.
----

Squeak     141 sanders  cwd      DIR      14,2      1156         2 /
Squeak     141 sanders  txt      REG      14,2   2592048   1833085 /
Users/sanders/Documents/Pharo-sudoku/Squeak 4.1.0beta1U.app/Contents/
MacOS/Squeak VM Opt
Squeak     141 sanders  txt      REG      14,2     19336   1070045 /
private/var/folders/3F/3FmnjhVXHi8w4CG9gjzd7U+++TI/-Caches-/
com.apple.IntlDataCache.le.sbdl
Squeak     141 sanders  txt      REG      14,2    100788    688604 /
System/Library/Caches/com.apple.IntlDataCache.le.kbdx
Squeak     141 sanders  txt      REG      14,2    144580     11678 /
System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/
HIToolbox.framework/Versions/A/Resources/HIToolbox.rsrc
Squeak     141 sanders  txt      REG      14,2   2295501     10702 /
System/Library/Fonts/LucidaGrande.dfont
Squeak     141 sanders  txt      REG      14,2    808008     15784 /
System/Library/Keyboard Layouts/AppleKeyboardLayouts.bundle/Contents/
Resources/AppleKeyboardLayouts-L.dat
Squeak     141 sanders  txt      REG      14,2    490410     11654 /
System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/
HIToolbox.framework/Versions/A/Resources/English.lproj/Localized.rsrc
Squeak     141 sanders  txt      REG      14,2   9309920    689553 /
private/var/folders/3F/3FmnjhVXHi8w4CG9gjzd7U+++TI/-Caches-/
com.apple.ATS/annex_aux
Squeak     141 sanders  txt      REG      14,2    138848     31052 /
Library/Fonts/Trebuchet MS.ttf
Squeak     141 sanders  txt      REG      14,2    129360     31050 /
Library/Fonts/Trebuchet MS Bold.ttf
Squeak     141 sanders  txt      REG      14,2    144820     31051 /
Library/Fonts/Trebuchet MS Italic.ttf
Squeak     141 sanders  txt      REG      14,2     44848   1833408 /
Users/sanders/Documents/Pharo-sudoku/Squeak 4.1.0beta1U.app/Contents/
Resources/SpellingPlugin.bundle/Contents/MacOS/SpellingPlugin
Squeak     141 sanders  txt      REG      14,2     46608   1833402 /
Users/sanders/Documents/Pharo-sudoku/Squeak 4.1.0beta1U.app/Contents/
Resources/SpellingPlugin.bundle/Contents/Frameworks/SpellCheck.bundle/
Contents/MacOS/SpellCheck
Squeak     141 sanders  txt      REG      14,2     34952   1833093 /
Users/sanders/Documents/Pharo-sudoku/Squeak 4.1.0beta1U.app/Contents/
Resources/ClipboardExtendedPlugin.bundle/Contents/MacOS/
ClipboardExtendedPlugin
Squeak     141 sanders  txt      REG      14,2    137008     31049 /
Library/Fonts/Trebuchet MS Bold Italic.ttf
Squeak     141 sanders  txt      REG      14,2   5425538     11671 /
System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/
HIToolbox.framework/Versions/A/Resources/Extras2.rsrc
Squeak     141 sanders  txt      REG      14,2    946176   1880828 /
Library/Caches/com.apple.LaunchServices-023501.csstore
Squeak     141 sanders  txt      REG      14,2  10243824     12079 /
usr/share/icu/icudt36l.dat
Squeak     141 sanders  txt      REG      14,2   3537336   1833127 /
Users/sanders/Documents/Pharo-sudoku/Squeak 4.1.0beta1U.app/Contents/
Resources/FT2Plugin.bundle/Contents/MacOS/FT2Plugin
Squeak     141 sanders  txt      REG      14,2   1832384      7707 /
System/Library/PrivateFrameworks/CoreUI.framework/Versions/A/Resources/
SArtFile.bin
Squeak     141 sanders  txt      REG      14,2  10860368    692208 /
System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
Squeak     141 sanders  txt      REG      14,2   5836820      7536 /
System/Library/PrivateFrameworks/CoreUI.framework/Versions/A/Resources/
ArtFile.bin
Squeak     141 sanders  txt      REG      14,2   1059792     21872 /
usr/lib/dyld
Squeak     141 sanders  txt      REG      14,2 136196096    706084 /
private/var/db/dyld/dyld_shared_cache_i386
Squeak     141 sanders    0r     CHR       3,2       0t0 119087748 /
dev/null
Squeak     141 sanders    1     PIPE 0x7232644     16384
Squeak     141 sanders    2     PIPE 0x7232644     16384
Squeak     141 sanders    3r     REG      14,2   5425538     11671 /
System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/
HIToolbox.framework/Versions/A/Resources/Extras2.rsrc
Squeak     141 sanders    4r     REG      14,2    144580     11678 /
System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/
HIToolbox.framework/Versions/A/Resources/HIToolbox.rsrc
Squeak     141 sanders    5r     REG      14,2    490410     11654 /
System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/
HIToolbox.framework/Versions/A/Resources/English.lproj/Localized.rsrc
Squeak     141 sanders    6r     DIR      14,2      5814   1102398 /
Users/sanders/Documents/Pharo-sudoku/package-cache
Squeak     141 sanders    7r     REG      14,2  17584742   1102378 /
Users/sanders/Documents/Pharo-sudoku/SqueakV39.sources
Squeak     141 sanders    8u     REG      14,2  32481960   1873656 /
Users/sanders/Documents/Pharo-sudoku/pharo0.1-cs-sudoku-1.5.changes
Squeak     141 sanders    9u    IPv4 0x7c3ce64       0t0       TCP  
*:http-alt (LISTEN)



_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

I forgot to add that my image has seaside (even though i don't use  
it). Perhaps I should step back and make certain I have the right  
starting image for playing around with these sockets.

As I said in a prior email FTPClient appears to have a code error:

It looks  like the last line of

FTPClient>>openPassiveDataConnection should be:
        self openDataSocket: remoteHostAddress asSocketAddress port: dataPort

instead of (current code):
        self openDataSocket: remoteHostAddress port: dataPort


By making this change, I can get down to Socket primitive errors with  
the FTP side of Monticello.

I am not set up to contribute code, so, it would be nice if somebody  
else could check this out, and submit the fix.

-cam

>


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project