Re: [squeak-dev] Low Space Warning

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Low Space Warning

Eliot Miranda-2
 


On Wed, Mar 10, 2010 at 6:39 AM, Jon Hylands <[hidden email]> wrote:
On Fri, 19 Feb 2010 16:25:20 -0500, Jon Hylands <[hidden email]> wrote:

>I'm getting a weird problem, and its been so many years since I saw
>this, I'm at a loss as to what to do...
>
>On my robot Brainbot, I have a 2.83 GHz Core 2 Quad with 2 GB RAM and
>a 32 GB solid state HD. It is running Windows XP Pro.
>
>I'm getting a Low Space Notifier whenever I try and copy text to the
>clipboard, or when I try and paste. I can do other development things,
>writing a saving code, and opening browsers and windows with no
>problems.
>
>Task Manager says Squeak is taking up 37,980 K, and the machine has
>1.2 GB of free physical memory.
>
>I'm running the following VM:
>
>Squeak 3.10.6 (release) from Aug 30 2007
>
>And the following image:
>
>Squeak 3.10.2
>latest update: #7179

Okay, so this is still bothering me, and I dug a little deeper, and
came across something that blew me away...

http://www.huv.com/Clipboard-OutOfMemory.png

Basically, when I do 'copy', the primitive returns a ByteString that
is 4 billion bytes long -- 2^32 - 1 to be exact. Clearly that is not
really allocated, because even when I have this inspector open, Squeak
is still only taking about 37 MB of memory.

However, the method String >> #withSqueakLineEndings is trying to
create a new string that is 4 billion bytes long, and that is where I
run into the low space warning.

Primitive 141 appears to be causing the issue.

The primitive looks like this in my quite possibly obsolete VM sources:

primitiveClipboardText
"When called with a single string argument, post the string to 
the clipboard. When called with zero arguments, return a 
string containing the current clipboard contents."
| s sz |
argumentCount = 1
ifTrue: [s := self stackTop.
(self isBytes: s) ifFalse: [^ self primitiveFail].
self successful
ifTrue: [sz := self stSizeOf: s.
self clipboardWrite: sz From: s + BaseHeaderSize At: 0.
self pop: 1]]
ifFalse: [sz := self clipboardSize.
(self sufficientSpaceToAllocate: sz) ifFalse:[^self primitiveFail].
s := self instantiateClass: (self splObj: ClassString) indexableSize: sz.
self clipboardRead: sz Into: s + BaseHeaderSize At: 0.
self pop: 1 thenPush: s]

I'm willing to bet that clipboardSize is returning -1.

It probably needs to read something like

primitiveClipboardText
"When called with a single string argument, post the string to 
the clipboard. When called with zero arguments, return a 
string containing the current clipboard contents."
| s sz |
argumentCount = 1
ifTrue: [s := self stackTop.
(self isBytes: s) ifFalse: [^ self primitiveFail].
self successful
ifTrue: [sz := self stSizeOf: s.
self clipboardWrite: sz From: s + BaseHeaderSize At: 0.
self pop: 1]]
ifFalse: [sz := self clipboardSize.
>> (sz >= 0 and: [self sufficientSpaceToAllocate: sz]) ifFalse:[^self primitiveFail].
s := self instantiateClass: (self splObj: ClassString) indexableSize: sz.
self clipboardRead: sz Into: s + BaseHeaderSize At: 0.
self pop: 1 thenPush: s] 

I'm not much of a VM person, so hopefully someone who is can look at
that primitive and figure out what case is causing it.

One more thing to note - I'm running Windows in a Remote Desktop, from
a Linux machine. To clarify, my development machine is a desktop,
running Ubuntu linux. The robot is a mini-itx running Windows XP Pro.
I use remote desktop from linux to access the Windows GUI on the
robot.

The Windows VM's clipboardText function is a tad unsafe:

int clipboardSize(void) {
  HANDLE h;
  WCHAR *src;
  unsigned char *tmp;
  int i, count, bytesNeeded;

  /* Do we have text in the clipboard? */
  if(!IsClipboardFormatAvailable(CF_UNICODETEXT))
    return 0;

  if(!OpenClipboard(stWindow))
    return 0;

  /* Get it in unicode format. */
  h = GetClipboardData(CF_UNICODETEXT);
  src = GlobalLock(h);

  /* How many bytes do we need to store those unicode chars in UTF8 format? */
  bytesNeeded = WideCharToMultiByte(CP_UTF8, 0, src, -1,
                    NULL, 0, NULL, NULL );
  tmp = malloc(bytesNeeded+1);

  /* Convert Unicode text to UTF8. */
  WideCharToMultiByte(CP_UTF8, 0, src, -1, tmp, bytesNeeded , NULL, NULL);

  /* Count CrLfs for which we remove the extra Lf */
  count = bytesNeeded; /* ex. terminating zero */
  for(i=0; i<count; i++) {
    if((tmp[i] == 13) && (tmp[i+1] == 10)) bytesNeeded--;
  }
  bytesNeeded--; /* discount terminating zero */
  free(tmp); /* no longer needed */

  GlobalUnlock(h);
  CloseClipboard();

  return bytesNeeded;
}

If bytesNeeded is zero then the function will return (int)-1. 

From the MS manual page:

Return Value

Returns the number of bytes written to the buffer pointed to by lpMultiByteStr if successful. If the function succeeds and cbMultiByte is 0, the return value is the required size, in bytes, for the buffer indicated by lpMultiByteStr.

The function returns 0 if it does not succeed. To get extended error information, the application can call GetLastError, which can return one of the following error codes:

  • ERROR_INSUFFICIENT_BUFFER. A supplied buffer size was not large enough, or it was incorrectly set to NULL.
  • ERROR_INVALID_FLAGS. The values supplied for flags were not valid.
  • ERROR_INVALID_PARAMETER. Any of the parameter values was invalid.
  • ERROR_NO_UNICODE_TRANSLATION. Invalid Unicode was found in a string.

So something like

  /* How many bytes do we need to store those unicode chars in UTF8 format? */
     bytesNeeded = WideCharToMultiByte(CP_UTF8, 0, src, -1,
                    NULL, 0, NULL, NULL );
     if (bytesNeeded <= 0)
          return 0;

would be wise.
 

Later,
Jon



Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Low Space Warning

Eliot Miranda-2
 
The attached is better (remembers to unlock and close clipboard).  I don't think I have commit rights for http://squeakvm.org/svn/squeak/trunk.  Guess I should go get some :)  Who do I talk to?




On Wed, Mar 10, 2010 at 9:25 AM, Eliot Miranda <[hidden email]> wrote:


On Wed, Mar 10, 2010 at 6:39 AM, Jon Hylands <[hidden email]> wrote:
On Fri, 19 Feb 2010 16:25:20 -0500, Jon Hylands <[hidden email]> wrote:

>I'm getting a weird problem, and its been so many years since I saw
>this, I'm at a loss as to what to do...
>
>On my robot Brainbot, I have a 2.83 GHz Core 2 Quad with 2 GB RAM and
>a 32 GB solid state HD. It is running Windows XP Pro.
>
>I'm getting a Low Space Notifier whenever I try and copy text to the
>clipboard, or when I try and paste. I can do other development things,
>writing a saving code, and opening browsers and windows with no
>problems.
>
>Task Manager says Squeak is taking up 37,980 K, and the machine has
>1.2 GB of free physical memory.
>
>I'm running the following VM:
>
>Squeak 3.10.6 (release) from Aug 30 2007
>
>And the following image:
>
>Squeak 3.10.2
>latest update: #7179

Okay, so this is still bothering me, and I dug a little deeper, and
came across something that blew me away...

http://www.huv.com/Clipboard-OutOfMemory.png

Basically, when I do 'copy', the primitive returns a ByteString that
is 4 billion bytes long -- 2^32 - 1 to be exact. Clearly that is not
really allocated, because even when I have this inspector open, Squeak
is still only taking about 37 MB of memory.

However, the method String >> #withSqueakLineEndings is trying to
create a new string that is 4 billion bytes long, and that is where I
run into the low space warning.

Primitive 141 appears to be causing the issue.

The primitive looks like this in my quite possibly obsolete VM sources:

primitiveClipboardText
"When called with a single string argument, post the string to 
the clipboard. When called with zero arguments, return a 
string containing the current clipboard contents."
| s sz |
argumentCount = 1
ifTrue: [s := self stackTop.
(self isBytes: s) ifFalse: [^ self primitiveFail].
self successful
ifTrue: [sz := self stSizeOf: s.
self clipboardWrite: sz From: s + BaseHeaderSize At: 0.
self pop: 1]]
ifFalse: [sz := self clipboardSize.
(self sufficientSpaceToAllocate: sz) ifFalse:[^self primitiveFail].
s := self instantiateClass: (self splObj: ClassString) indexableSize: sz.
self clipboardRead: sz Into: s + BaseHeaderSize At: 0.
self pop: 1 thenPush: s]

I'm willing to bet that clipboardSize is returning -1.

It probably needs to read something like

primitiveClipboardText
"When called with a single string argument, post the string to 
the clipboard. When called with zero arguments, return a 
string containing the current clipboard contents."
| s sz |
argumentCount = 1
ifTrue: [s := self stackTop.
(self isBytes: s) ifFalse: [^ self primitiveFail].
self successful
ifTrue: [sz := self stSizeOf: s.
self clipboardWrite: sz From: s + BaseHeaderSize At: 0.
self pop: 1]]
ifFalse: [sz := self clipboardSize.
>> (sz >= 0 and: [self sufficientSpaceToAllocate: sz]) ifFalse:[^self primitiveFail].
s := self instantiateClass: (self splObj: ClassString) indexableSize: sz.
self clipboardRead: sz Into: s + BaseHeaderSize At: 0.
self pop: 1 thenPush: s] 

I'm not much of a VM person, so hopefully someone who is can look at
that primitive and figure out what case is causing it.

One more thing to note - I'm running Windows in a Remote Desktop, from
a Linux machine. To clarify, my development machine is a desktop,
running Ubuntu linux. The robot is a mini-itx running Windows XP Pro.
I use remote desktop from linux to access the Windows GUI on the
robot.

The Windows VM's clipboardText function is a tad unsafe:

int clipboardSize(void) {
  HANDLE h;
  WCHAR *src;
  unsigned char *tmp;
  int i, count, bytesNeeded;

  /* Do we have text in the clipboard? */
  if(!IsClipboardFormatAvailable(CF_UNICODETEXT))
    return 0;

  if(!OpenClipboard(stWindow))
    return 0;

  /* Get it in unicode format. */
  h = GetClipboardData(CF_UNICODETEXT);
  src = GlobalLock(h);

  /* How many bytes do we need to store those unicode chars in UTF8 format? */
  bytesNeeded = WideCharToMultiByte(CP_UTF8, 0, src, -1,
                    NULL, 0, NULL, NULL );
  tmp = malloc(bytesNeeded+1);

  /* Convert Unicode text to UTF8. */
  WideCharToMultiByte(CP_UTF8, 0, src, -1, tmp, bytesNeeded , NULL, NULL);

  /* Count CrLfs for which we remove the extra Lf */
  count = bytesNeeded; /* ex. terminating zero */
  for(i=0; i<count; i++) {
    if((tmp[i] == 13) && (tmp[i+1] == 10)) bytesNeeded--;
  }
  bytesNeeded--; /* discount terminating zero */
  free(tmp); /* no longer needed */

  GlobalUnlock(h);
  CloseClipboard();

  return bytesNeeded;
}

If bytesNeeded is zero then the function will return (int)-1. 

From the MS manual page:

Return Value

Returns the number of bytes written to the buffer pointed to by lpMultiByteStr if successful. If the function succeeds and cbMultiByte is 0, the return value is the required size, in bytes, for the buffer indicated by lpMultiByteStr.

The function returns 0 if it does not succeed. To get extended error information, the application can call GetLastError, which can return one of the following error codes:

  • ERROR_INSUFFICIENT_BUFFER. A supplied buffer size was not large enough, or it was incorrectly set to NULL.
  • ERROR_INVALID_FLAGS. The values supplied for flags were not valid.
  • ERROR_INVALID_PARAMETER. Any of the parameter values was invalid.
  • ERROR_NO_UNICODE_TRANSLATION. Invalid Unicode was found in a string.

So something like

  /* How many bytes do we need to store those unicode chars in UTF8 format? */
     bytesNeeded = WideCharToMultiByte(CP_UTF8, 0, src, -1,
                    NULL, 0, NULL, NULL );
     if (bytesNeeded <= 0)
          return 0;

would be wise.
 

Later,
Jon





sqWin32Window.c (126K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Low Space Warning

Bert Freudenberg

On 10.03.2010, at 18:31, Eliot Miranda wrote:
> The attached is better (remembers to unlock and close clipboard).  I don't think I have commit rights for http://squeakvm.org/svn/squeak/trunk.  Guess I should go get some :)  Who do I talk to?

Unless I'm mistaken only Ian, John, and Andreas have commit access to the VM trunk.

And it's Ian's private box this is running on.

- Bert -


Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Low Space Warning

Eliot Miranda-2
 


On Wed, Mar 10, 2010 at 9:38 AM, Bert Freudenberg <[hidden email]> wrote:

On 10.03.2010, at 18:31, Eliot Miranda wrote:
> The attached is better (remembers to unlock and close clipboard).  I don't think I have commit rights for http://squeakvm.org/svn/squeak/trunk.  Guess I should go get some :)  Who do I talk to?

Unless I'm mistaken only Ian, John, and Andreas have commit access to the VM trunk.

And it's Ian's private box this is running on.

Then posting a modified file is fine.  Works for me :)  Thanks, Bert!
 

- Bert -



Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Low Space Warning

David T. Lewis
 
On Wed, Mar 10, 2010 at 10:00:48AM -0800, Eliot Miranda wrote:

>  
> On Wed, Mar 10, 2010 at 9:38 AM, Bert Freudenberg <[hidden email]>wrote:
>
> >
> > On 10.03.2010, at 18:31, Eliot Miranda wrote:
> > > The attached is better (remembers to unlock and close clipboard).  I
> > don't think I have commit rights for http://squeakvm.org/svn/squeak/trunk.
> >  Guess I should go get some :)  Who do I talk to?
> >
> > Unless I'm mistaken only Ian, John, and Andreas have commit access to the
> > VM trunk.
> >
> > And it's Ian's private box this is running on.
> >
>
> Then posting a modified file is fine.  Works for me :)  Thanks, Bert!

Actually if you don't mind a few extra steps, it helps a lot if you
can open an issue on Mantis under category "VM" and attach it there.
Things do have a tendency to get lost if just posted to the list.

Dave

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Low Space Warning

Eliot Miranda-2
 


On Wed, Mar 10, 2010 at 10:56 AM, David T. Lewis <[hidden email]> wrote:

On Wed, Mar 10, 2010 at 10:00:48AM -0800, Eliot Miranda wrote:
>
> On Wed, Mar 10, 2010 at 9:38 AM, Bert Freudenberg <[hidden email]>wrote:
>
> >
> > On 10.03.2010, at 18:31, Eliot Miranda wrote:
> > > The attached is better (remembers to unlock and close clipboard).  I
> > don't think I have commit rights for http://squeakvm.org/svn/squeak/trunk.
> >  Guess I should go get some :)  Who do I talk to?
> >
> > Unless I'm mistaken only Ian, John, and Andreas have commit access to the
> > VM trunk.
> >
> > And it's Ian's private box this is running on.
> >
>
> Then posting a modified file is fine.  Works for me :)  Thanks, Bert!

Actually if you don't mind a few extra steps, it helps a lot if you
can open an issue on Mantis under category "VM" and attach it there.
Things do have a tendency to get lost if just posted to the list.

Quite :)  Of course.
 

Dave


Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Low Space Warning

Eliot Miranda-2
In reply to this post by David T. Lewis
 


On Wed, Mar 10, 2010 at 10:56 AM, David T. Lewis <[hidden email]> wrote:

On Wed, Mar 10, 2010 at 10:00:48AM -0800, Eliot Miranda wrote:
>
> On Wed, Mar 10, 2010 at 9:38 AM, Bert Freudenberg <[hidden email]>wrote:
>
> >
> > On 10.03.2010, at 18:31, Eliot Miranda wrote:
> > > The attached is better (remembers to unlock and close clipboard).  I
> > don't think I have commit rights for http://squeakvm.org/svn/squeak/trunk.
> >  Guess I should go get some :)  Who do I talk to?
> >
> > Unless I'm mistaken only Ian, John, and Andreas have commit access to the
> > VM trunk.
> >
> > And it's Ian's private box this is running on.
> >
>
> Then posting a modified file is fine.  Works for me :)  Thanks, Bert!

Actually if you don't mind a few extra steps, it helps a lot if you
can open an issue on Mantis under category "VM" and attach it there.
Things do have a tendency to get lost if just posted to the list.

Issue 7476. clipboardSize on Win32 can return -1 (4294967295) on failure
 

Dave


Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Low Space Warning

David T. Lewis
 
On Wed, Mar 10, 2010 at 11:34:29AM -0800, Eliot Miranda wrote:
>  
> On Wed, Mar 10, 2010 at 10:56 AM, David T. Lewis <[hidden email]>wrote:
> >
> > Actually if you don't mind a few extra steps, it helps a lot if you
> > can open an issue on Mantis under category "VM" and attach it there.
> > Things do have a tendency to get lost if just posted to the list.
>
> Issue 7476. clipboardSize on Win32 can return -1 (4294967295) on failure

Thank you!