New Cog VMs available...

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
35 messages Options
12
Reply | Threaded
Open this post in threaded view
|

New Cog VMs available...

Eliot Miranda-2
at http://www.mirandabanda.org/files/Cog/VM/VM.r3024/.

I had *forgotten* to implement following forwarders on primitive failure in machine code primitives, hence Cog Spur was badly broken.  It's now a little bit better, but I still need to implement this for named primitives.  Real soon now I hope...

CogVM source as per VMMaker.oscog-eem.791/r3024

Implement following forwarders on primitive failure in machine code interpreter
primitives (still have to implement this in sideways calls of named primitives).

Allow the JIT to not compile primitiveDoNamedPrimitiveWithArgs
to avoid any potential complications.

Rewrite all the semaphore installing primitives to fail if the semaphore arg is
neither a semaphore or nil instead of assuming if its not a semaphore it must be
nil, so as to fail and retry when semaphores are forwarded (as they are when
Semaphore is redefined).

Implement isSemaphoreOop:/Obj: in the object memories to abstract away the code.
Base Spur's on the class index of splObj: ClassSemaphore, avoiding the table
lookup to derive the class.  Make checkForEventsMayContextSwitch: treat all its
semaphores consistently.

Have spur's fetchClassOfNonImm: answer nilObj for
forwarders to avoid assert fails.

On Spur add read barriers to primitiveSuspend and synchronousSignal:'s myList
access, because the process list manipulation routines do no checking.  Add
assert checks for forwarders in the process list manipulation routines.

Fix slip in StackInterpreter>>actuallyFollowNecessaryForwardingInMethod:literalCount:
that corrupts the methodClassAssociation.

Abstract out the call machinery from compileTrampolineFor:numArgs:arg:arg:arg:-
arg:saveRegs:pushLinkReg:resultReg: so it can be used by
maybeCompileRetry:onPrimitiveFail: in implementing following forwarders on
primitive failure in machine code, and the Open PIC miss call.

Have bytecodePCFor:cogMethod:startBcpc: map any pc before the stackCheckOffset
to the initialPC, which applies to primitives in progress.

Fix assert fails in updateStateOfSpouseContextForFrame:WithSP:
and elsewhere with forwarders.

LargeIntegers Plugin:
Fix a latent signed shift bug in cDigitSub:len:with:len:into:
caused by VMMaker.oscog-eem.785's eliminating the
divide-via-shift optimization.

These changes allow Cog Spur to redefine Process and/or Semaphore and not hang.


--
best,
Eliot


Reply | Threaded
Open this post in threaded view
|

Re: New Cog VMs available...

marcel.taeumel (old)
Hmmm... would it be possible to add the following minor modification to the sources?

It is about this file:
http://www.squeakvm.org/svn/squeak/branches/Cog/platforms/win32/vm/sqWin32Alloc.h

Could you please change the definition of MAX_VIRTUAL_MEMORY from

#define MAX_VIRTUAL_MEMORY 512*1024*1024

to

#define MAX_VIRTUAL_MEMORY 512*1024*1024*4

Pretty please? =)

Best,
Marcel
Reply | Threaded
Open this post in threaded view
|

Re: New Cog VMs available...

Eliot Miranda-2
Hi Marcel,

    can you make the change, build a VM and verify that on a windows installation that allows > 2Gb binaries the VM allows one to allocate 2Gb?  Last time I tried to change the define things broke.

Eliot (phone)

On Jul 2, 2014, at 8:58 AM, Marcel Taeumel <[hidden email]> wrote:

> Hmmm... would it be possible to add the following minor modification to the
> sources?
>
> It is about this file:
> http://www.squeakvm.org/svn/squeak/branches/Cog/platforms/win32/vm/sqWin32Alloc.h
>
> Could you please change the definition of MAX_VIRTUAL_MEMORY from
>
> #define MAX_VIRTUAL_MEMORY 512*1024*1024
>
> to
>
> #define MAX_VIRTUAL_MEMORY 512*1024*1024*4
>
> Pretty please? =)
>
> Best,
> Marcel
>
>
>
> --
> View this message in context: http://forum.world.st/New-Cog-VMs-available-tp4765964p4766152.html
> Sent from the Squeak - Dev mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] New Cog VMs available...

Frank Shearar-3
In reply to this post by Eliot Miranda-2
On 1 July 2014 18:03, Eliot Miranda <[hidden email]> wrote:
>
> at http://www.mirandabanda.org/files/Cog/VM/VM.r3024/.

CI updated accordingly! For Spur:
http://build.squeak.org/job/SqueakTrunkOnSpur/29/console

frank

Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] New Cog VMs available...

Eliot Miranda-2



On Wed, Jul 2, 2014 at 1:45 PM, Frank Shearar <[hidden email]> wrote:
On 1 July 2014 18:03, Eliot Miranda <[hidden email]> wrote:
>
> at http://www.mirandabanda.org/files/Cog/VM/VM.r3024/.

CI updated accordingly! For Spur:
http://build.squeak.org/job/SqueakTrunkOnSpur/29/console

and I'm working on the large integer failures right now.  that should fix most of the errors.

--
best,
Eliot


Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] New Cog VMs available...

Frank Shearar-3
On 2 July 2014 22:19, Eliot Miranda <[hidden email]> wrote:

>
>
>
> On Wed, Jul 2, 2014 at 1:45 PM, Frank Shearar <[hidden email]>
> wrote:
>>
>> On 1 July 2014 18:03, Eliot Miranda <[hidden email]> wrote:
>> >
>> > at http://www.mirandabanda.org/files/Cog/VM/VM.r3024/.
>>
>> CI updated accordingly! For Spur:
>> http://build.squeak.org/job/SqueakTrunkOnSpur/29/console
>
>
> and I'm working on the large integer failures right now.  that should fix
> most of the errors.

Excellent! That will end the streak of red in the SqueakTrunk build history.

Thanks!

frank

> --
> best,
> Eliot

Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] New Cog VMs available...

Nicolas Cellier



2014-07-02 23:32 GMT+02:00 Frank Shearar <[hidden email]>:
On 2 July 2014 22:19, Eliot Miranda <[hidden email]> wrote:
>
>
>
> On Wed, Jul 2, 2014 at 1:45 PM, Frank Shearar <[hidden email]>
> wrote:
>>
>> On 1 July 2014 18:03, Eliot Miranda <[hidden email]> wrote:
>> >
>> > at http://www.mirandabanda.org/files/Cog/VM/VM.r3024/.
>>
>> CI updated accordingly! For Spur:
>> http://build.squeak.org/job/SqueakTrunkOnSpur/29/console
>
>
> and I'm working on the large integer failures right now.  that should fix
> most of the errors.

Excellent! That will end the streak of red in the SqueakTrunk build history.

Thanks!


I've got a working solution ready for large int plugin.
digitDiv and montgomery are broken due to the same bug // 256...
I can publish if you want.
 
frank

> --
> best,
> Eliot




Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] New Cog VMs available...

Eliot Miranda-2
Hi Nicolas,


On Wed, Jul 2, 2014 at 2:51 PM, Nicolas Cellier <[hidden email]> wrote:



2014-07-02 23:32 GMT+02:00 Frank Shearar <[hidden email]>:

On 2 July 2014 22:19, Eliot Miranda <[hidden email]> wrote:
>
>
>
> On Wed, Jul 2, 2014 at 1:45 PM, Frank Shearar <[hidden email]>
> wrote:
>>
>> On 1 July 2014 18:03, Eliot Miranda <[hidden email]> wrote:
>> >
>> > at http://www.mirandabanda.org/files/Cog/VM/VM.r3024/.
>>
>> CI updated accordingly! For Spur:
>> http://build.squeak.org/job/SqueakTrunkOnSpur/29/console
>
>
> and I'm working on the large integer failures right now.  that should fix
> most of the errors.

Excellent! That will end the streak of red in the SqueakTrunk build history.

Thanks!


I've got a working solution ready for large int plugin.
digitDiv and montgomery are broken due to the same bug // 256...
I can publish if you want.

Forgive me but I went ahead and fixed it.  I just figured out that I could have the code generator output asserts at every divide by a power of two to check that the results agreed with signed shifts and hence easily caught all the places where there was a bug.  I've also realised that I can have the code generator output #if #else #endif code for remapOop:in: and so not incur any overhead in Spur, which doesn't GC during primitives.

--
best,
Eliot


Reply | Threaded
Open this post in threaded view
|

[Vm-dev] Bug in writing compressed stream when saving an mcz (was: New Cog VMs available...)

David T. Lewis
 
On Wed, Jul 02, 2014 at 03:48:03PM -0700, Eliot Miranda wrote:

>  
> On Wed, Jul 2, 2014 at 3:39 PM, Nicolas Cellier <
> [hidden email]> wrote:
> >
> > Anyway, it happens that I cannot publish because there is a bug in my
> > image (mostly an updated trunk image)...
> > I'm currently stuck with ZipWriteStream(DeflateStream)>>updateHashAt:
> > which tries to write at position 65537 in a buffer of length 65536
> > Huh?
> >
>
> Ah, I've seen this recently too!  I think it's just a corrupted mcz
> package.  Delete the package and download a fresh one and you could find
> the bug evaporates.

I saw it last weekend too. I thought that it was an improbable edge condition,
so I did not mention it on squeak-dev until I could explain the bug.

It is not a corrupted mcz. In my case, the failure occurred while trying to
write a new mcz.

We should move this to squeak-dev and figure out what is going on.

Dave

Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] Bug in writing compressed stream when saving an mcz (was: New Cog VMs available...)

David T. Lewis
On Wed, Jul 02, 2014 at 06:53:09PM -0700, Eliot Miranda wrote:

>  
> On Wed, Jul 2, 2014 at 5:08 PM, David T. Lewis <[hidden email]> wrote:
>
> >
> > On Wed, Jul 02, 2014 at 03:48:03PM -0700, Eliot Miranda wrote:
> > >
> > > On Wed, Jul 2, 2014 at 3:39 PM, Nicolas Cellier <
> > > [hidden email]> wrote:
> > > >
> > > > Anyway, it happens that I cannot publish because there is a bug in my
> > > > image (mostly an updated trunk image)...
> > > > I'm currently stuck with ZipWriteStream(DeflateStream)>>updateHashAt:
> > > > which tries to write at position 65537 in a buffer of length 65536
> > > > Huh?
> > > >
> > >
> > > Ah, I've seen this recently too!  I think it's just a corrupted mcz
> > > package.  Delete the package and download a fresh one and you could find
> > > the bug evaporates.
> >
> > I saw it last weekend too. I thought that it was an improbable edge
> > condition,
> > so I did not mention it on squeak-dev until I could explain the bug.
> >
> > It is not a corrupted mcz. In my case, the failure occurred while trying to
> > write a new mcz.
> >
> > We should move this to squeak-dev and figure out what is going on.
> >
>
> That triggers my memory.  I had a method with corrupted source, perhaps
> wide characters.  That broke the writer.  I fixed it by fixing the method
> source code.  Where the corruption came from I can't say.
I would not be surprised if it is something to do with wide characters. I
vaguely recall that we fixed a similar sounding problem elsewhere in the
system recently. Unfortunately I cannot remember what it was.

In any case, I am attaching a png of the debug window for the problem as I
encountered it last weekend. This will need to be turned into a repeatable
test case.

Dave




Error: subscript is out of bounds: 65537.png (68K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: New Cog VMs available...

marcel.taeumel (old)
In reply to this post by Eliot Miranda-2
Hi Elliot,

okay, I will report about that. At the moment, I am using an older version of Cog with this define changed---custom build. Did not allocate more than 900 MB so far, I guess. I will stress it to the 2 GB boundary.

Best,
Marcel
Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] Bug in writing compressed stream when saving an mcz (was: New Cog VMs available...)

Nicolas Cellier
In reply to this post by David T. Lewis



2014-07-03 4:23 GMT+02:00 David T. Lewis <[hidden email]>:
On Wed, Jul 02, 2014 at 06:53:09PM -0700, Eliot Miranda wrote:
>
> On Wed, Jul 2, 2014 at 5:08 PM, David T. Lewis <[hidden email]> wrote:
>
> >
> > On Wed, Jul 02, 2014 at 03:48:03PM -0700, Eliot Miranda wrote:
> > >
> > > On Wed, Jul 2, 2014 at 3:39 PM, Nicolas Cellier <
> > > [hidden email]> wrote:
> > > >
> > > > Anyway, it happens that I cannot publish because there is a bug in my
> > > > image (mostly an updated trunk image)...
> > > > I'm currently stuck with ZipWriteStream(DeflateStream)>>updateHashAt:
> > > > which tries to write at position 65537 in a buffer of length 65536
> > > > Huh?
> > > >
> > >
> > > Ah, I've seen this recently too!  I think it's just a corrupted mcz
> > > package.  Delete the package and download a fresh one and you could find
> > > the bug evaporates.
> >
> > I saw it last weekend too. I thought that it was an improbable edge
> > condition,
> > so I did not mention it on squeak-dev until I could explain the bug.
> >
> > It is not a corrupted mcz. In my case, the failure occurred while trying to
> > write a new mcz.
> >
> > We should move this to squeak-dev and figure out what is going on.
> >
>
> That triggers my memory.  I had a method with corrupted source, perhaps
> wide characters.  That broke the writer.  I fixed it by fixing the method
> source code.  Where the corruption came from I can't say.

I would not be surprised if it is something to do with wide characters. I
vaguely recall that we fixed a similar sounding problem elsewhere in the
system recently. Unfortunately I cannot remember what it was.

In any case, I am attaching a png of the debug window for the problem as I
encountered it last weekend. This will need to be turned into a repeatable
test case.

Dave


Nope. it was a problem of encoding that I changed in .mcz sometimes ago at least in snapshot.st.
Before this change, some buffer started as a ByteString, then become WideString at some point...
Since the bytes of the buffer were picked and copied to another ByteArray with our wonderfull Stream, here is what happened:
- that more or less was corresponding to a change of encoding (CP1252 to UTF32 some endianness) in the middle of the file
- but because the leading byte character did inflate (x4), and the offset of the copy was not updated, the leading bytes were repeated at some point !
- and only n bytes would have been written instead of n characters !
Gasp, a real mess.
After my changes all is first encoded in UTF8 then compressed.

The problem we observe here is completely independent.
We provide a ByteArray to the Deflate layer, and the Deflate layer can't cope with it... We can reproduce with pure bytes, no encoding in the loop.
I've saved the image with debugger opened for being able to reproduce at will.
Unfortunately, my case is several million bytes long (VMMaker), not a joy to debug.



Reply | Threaded
Open this post in threaded view
|

Re: New Cog VMs available...

marcel.taeumel (old)
In reply to this post by marcel.taeumel (old)
Okay, VM crashed at 992,6 MB. LowSpaceWatcher did not show up. I suspect the (copying?) GC needing another 1GB to work fine? I thought that the define considered the sum of both memory spaces. Am I correct?

Basically, I just constructed a very deep tree structure to consume memory.

Best,
Marcel
tty
Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] Bug in writing compressed stream when saving an mcz (was: New Cog VMs available...)

tty
In reply to this post by David T. Lewis

I would not be surprised if it is something to do with wide characters. I
vaguely recall that we fixed a similar sounding problem elsewhere in the
system recently. Unfortunately I cannot remember what it was.

Could it be something as simple as the 'self flush' is needed like we did in WriteStream nextChunkPut?



Reply | Threaded
Open this post in threaded view
|

Re: New Cog VMs available...

Eliot Miranda-2
In reply to this post by marcel.taeumel (old)
Hi Marcel,


On Thu, Jul 3, 2014 at 5:58 AM, Marcel Taeumel <[hidden email]> wrote:
Okay, VM crashed at 992,6 MB. LowSpaceWatcher did not show up. I suspect the
(copying?) GC needing another 1GB to work fine?

That shouldn't be the case.  It should be happy with a headroom of a few hundred k.  The crash looks like its a problem with the LowSpaceWatcher, or a problem with the VM not maintaining enough headroom.
 
I thought that the define
considered the sum of both memory spaces. Am I correct?

That's right.
 
Basically, I just constructed a very deep tree structure to consume memory.

Does anything different happen with the standard VM?  It may be that teh define makes no difference.


BTW, the below is a cheap way to consume memory because a byte object contains no pointers and so if faster to scan, compact, etc:

| them | them := OrderedCollection new. [them addLast: (ByteArray new: 1024*1024*256)] repeat

--
best,
Eliot


Reply | Threaded
Open this post in threaded view
|

Re: New Cog VMs available...

marcel.taeumel (old)
Hi Elliot,

your code fills up memory faster. Mine was just some run-time data collection project. :) With your code, the LowSpaceWatcher kicks in (at about 960 MB).

The standard non-Cog SqueakVM in a Squeak 4.5 image informs at about 510 MB about low memory. Cannot even dismiss the watcher... keeps on bothering me.

In both cases, the VM does not crash. :)

Best,
Marcel
Reply | Threaded
Open this post in threaded view
|

Re: New Cog VMs available...

Eliot Miranda-2



On Thu, Jul 3, 2014 at 8:40 AM, Marcel Taeumel <[hidden email]> wrote:
Hi Elliot,

your code fills up memory faster. Mine was just some run-time data
collection project. :) With your code, the LowSpaceWatcher kicks in (at
about 960 MB).

The standard non-Cog SqueakVM in a Squeak 4.5 image informs at about 510 MB
about low memory. Cannot even dismiss the watcher... keeps on bothering me.

In both cases, the VM does not crash. :)

So just to confirm, you think it is safe to raise MAX_VIRTUAL_MEMORY 

#define MAX_VIRTUAL_MEMORY 512*1024*1024*4

yes?

--
best,
Eliot


Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] Bug in writing compressed stream when saving an mcz (was: New Cog VMs available...)

Nicolas Cellier
In reply to this post by tty
The bug is repeatable, i simply have to execute this snippet with my test file:

(FileStream fileNamed: 'snapshot.bin') binary contentsOfEntireFile asString zipped.

The file is too big for an attachment here - 7.3 Mbytes - or 1.7 Mbytes gzipped by external tool.
If someone can suggest an upload site, or want it by mail, just ask.





2014-07-03 16:05 GMT+02:00 gettimothy <[hidden email]>:

I would not be surprised if it is something to do with wide characters. I
vaguely recall that we fixed a similar sounding problem elsewhere in the
system recently. Unfortunately I cannot remember what it was.

Could it be something as simple as the 'self flush' is needed like we did in WriteStream nextChunkPut?







Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] Bug in writing compressed stream when saving an mcz (was: New Cog VMs available...)

Eliot Miranda-2



On Thu, Jul 3, 2014 at 12:33 PM, Nicolas Cellier <[hidden email]> wrote:
The bug is repeatable, i simply have to execute this snippet with my test file:

(FileStream fileNamed: 'snapshot.bin') binary contentsOfEntireFile asString zipped.

The file is too big for an attachment here - 7.3 Mbytes - or 1.7 Mbytes gzipped by external tool.
If someone can suggest an upload site, or want it by mail, just ask.

You're welcome to put it on ftp.mirandanamda.org, cogftpuser, pw cogging with 0's & 1's. 

I bet you can narrow it down by doing

| file |
file := FileStream fileNamed: 'snapshot.bin'.
file binary.
[[file atEnd] whileFalse: [(file next: 16*1024) asString zipped]] ensure: [file close]

2014-07-03 16:05 GMT+02:00 gettimothy <[hidden email]>:


I would not be surprised if it is something to do with wide characters. I
vaguely recall that we fixed a similar sounding problem elsewhere in the
system recently. Unfortunately I cannot remember what it was.

Could it be something as simple as the 'self flush' is needed like we did in WriteStream nextChunkPut?

--
best,
Eliot


Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] Bug in writing compressed stream when saving an mcz (was: New Cog VMs available...)

Nicolas Cellier



2014-07-04 0:09 GMT+02:00 Eliot Miranda <[hidden email]>:



On Thu, Jul 3, 2014 at 12:33 PM, Nicolas Cellier <[hidden email]> wrote:
The bug is repeatable, i simply have to execute this snippet with my test file:

(FileStream fileNamed: 'snapshot.bin') binary contentsOfEntireFile asString zipped.

The file is too big for an attachment here - 7.3 Mbytes - or 1.7 Mbytes gzipped by external tool.
If someone can suggest an upload site, or want it by mail, just ask.

You're welcome to put it on ftp.mirandanamda.org, cogftpuser, pw cogging with 0's & 1's. 


done, pollution of your site engaged...
Thanks Eliot!
 
I bet you can narrow it down by doing

| file |
file := FileStream fileNamed: 'snapshot.bin'.
file binary.
[[file atEnd] whileFalse: [(file next: 16*1024) asString zipped]] ensure: [file close]

2014-07-03 16:05 GMT+02:00 gettimothy <[hidden email]>:


I would not be surprised if it is something to do with wide characters. I
vaguely recall that we fixed a similar sounding problem elsewhere in the
system recently. Unfortunately I cannot remember what it was.

Could it be something as simple as the 'self flush' is needed like we did in WriteStream nextChunkPut?

--
best,
Eliot






12