GZipWriteStream crashing my VM when serializing ?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

GZipWriteStream crashing my VM when serializing ?

Mariano Martinez Peck
Hi guys. We are doing some experiments with Fuel and GZip streams... Now I have a beatiful test like this:

testMethodContextThisContext

    | materializedContext context |
    context := thisContext copy.
    materializedContext := self resultOfSerializeAndMaterialize: context.

and it crashes like crazy. From the VM crash stack, I can see:


Smalltalk stack dump:
0xbff66ba4 M GZipWriteStream(DeflateStream)>deflateBlock 447551436: a(n) GZipWriteStream
0xbff66bc8 M GZipWriteStream(DeflateStream)>nextPutAll: 447551436: a(n) GZipWriteStream
0xbff66be4 M GZipWriteStream(DeflateStream)>next:putAll:startingAt: 447551436: a(n) GZipWriteStream
0xbff66c08 M FLBufferedWriteStream>flushBuffer 448048664: a(n) FLBufferedWriteStream
0xbff66c20 M FLBufferedWriteStream>nextBytesPutAll: 448048664: a(n) FLBufferedWriteStream
0xbff66c40 M FLBufferedWriteStream>nextStringPut: 448048664: a(n) FLBufferedWriteStream
0xbff66c5c M FLByteStringCluster>serialize:on: 448054196: a(n) FLByteStringCluster
0xbff66c7c M FLByteStringCluster(FLPrimitiveCluster)>serializeInstance:with: 448054196: a(n) FLByteStringCluster
0xbff66ca0 M [] in FLByteStringCluster(FLIteratingCluster)>serializeInstances: 448054196: a(n) FLByteStringCluster
0xbff66cc8 M IdentitySet(Set)>do: 451711680: a(n) IdentitySet
0xbff66ce4 M FLByteStringCluster(FLIteratingCluster)>serializeInstances: 448054196: a(n) FLByteStringCluster
0xbff66d00 M FLByteStringCluster(FLIteratingCluster)>serializeInstancesStepWith: 448054196: a(n) FLByteStringCluster
0xbff66d1c M FLSerialization>clusterInstancesStepOf: 448053428: a(n) FLSerialization
0xbff66d38 M [] in FLSerialization>instancesStep 448053428: a(n) FLSerialization
0xbff66d58 M OrderedCollection>do: 449989736: a(n) OrderedCollection
0xbff66d74 M FLSerialization>instancesStep 448053428: a(n) FLSerialization
0xbff66d8c M FLSerialization>run 448053428: a(n) FLSerialization
0xbff66db0 I [] in FLSerializer>defaultSerialization 447550464: a(n) FLSerializer
0xbff66dd8 M [] in FLSerializer>serialize:on: 447550464: a(n) FLSerializer
0xbff66df8 M BlockClosure>ensure: 448052884: a(n) BlockClosure
0xbff66e18 M FLSerializer>serialize:on: 447550464: a(n) FLSerializer
0xbff66e3c M [] in FLMethodContextSerializationTest(FLSerializationTest)>serialize: 447235380: a(n) FLMethodContextSerializationTest
0xbff66e60 M [] in GZipWriteStream class(PositionableStream class)>on:do: 419845144: a(n) GZipWriteStream class
0xbff66e80 M BlockClosure>ensure: 448048616: a(n) BlockClosure
0xbff66ea0 M GZipWriteStream class(PositionableStream class)>on:do: 419845144: a(n) GZipWriteStream class
0xbff66ec4 M [] in FLGZipStrategy>writeStreamDo: 447550896: a(n) FLGZipStrategy
0xbff66ee0 M FLByteArrayStreamStrategy>writeStreamDo: 447550884: a(n) FLByteArrayStreamStrategy
0xbff66efc M FLGZipStrategy>writeStreamDo: 447550896: a(n) FLGZipStrategy
0xbff66f18 M FLMethodContextSerializationTest(FLSerializationTest)>serialize: 447235380: a(n) FLMethodContextSerializationTest
0xbff66f34 M FLMethodContextSerializationTest(FLSerializationTest)>resultOfSerializeAndMaterialize: 447235380: a(n) FLMethodContextSerializationTest
0xbff66f58 M FLMethodContextSerializationTest>testMethodContextThisContext 447235380: a(n) FLMethodContextSerializationTest


any idea of where I should start to look at?

Thanks


--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: GZipWriteStream crashing my VM when serializing ?

johnmci
Mmm I recall a problem from long ago where the buffer passed into the
zip call was not big enough to store the results.
If you pass in a small chunk of data, why the buffer to hold the
results and the decode data is bigger than the buffer going in.
So the two sizes, input and output are?


On Fri, Dec 2, 2011 at 4:41 PM, Mariano Martinez Peck
<[hidden email]> wrote:

> Hi guys. We are doing some experiments with Fuel and GZip streams... Now I
> have a beatiful test like this:
>
> testMethodContextThisContext
>
>     | materializedContext context |
>     context := thisContext copy.
>     materializedContext := self resultOfSerializeAndMaterialize: context.
>
> and it crashes like crazy. From the VM crash stack, I can see:
>
>
> Smalltalk stack dump:
> 0xbff66ba4 M GZipWriteStream(DeflateStream)>deflateBlock 447551436: a(n)
> GZipWriteStream
> 0xbff66bc8 M GZipWriteStream(DeflateStream)>nextPutAll: 447551436: a(n)
> GZipWriteStream
> 0xbff66be4 M GZipWriteStream(DeflateStream)>next:putAll:startingAt:
> 447551436: a(n) GZipWriteStream
> 0xbff66c08 M FLBufferedWriteStream>flushBuffer 448048664: a(n)
> FLBufferedWriteStream
> 0xbff66c20 M FLBufferedWriteStream>nextBytesPutAll: 448048664: a(n)
> FLBufferedWriteStream
> 0xbff66c40 M FLBufferedWriteStream>nextStringPut: 448048664: a(n)
> FLBufferedWriteStream
> 0xbff66c5c M FLByteStringCluster>serialize:on: 448054196: a(n)
> FLByteStringCluster
> 0xbff66c7c M FLByteStringCluster(FLPrimitiveCluster)>serializeInstance:with:
> 448054196: a(n) FLByteStringCluster
> 0xbff66ca0 M [] in
> FLByteStringCluster(FLIteratingCluster)>serializeInstances: 448054196: a(n)
> FLByteStringCluster
> 0xbff66cc8 M IdentitySet(Set)>do: 451711680: a(n) IdentitySet
> 0xbff66ce4 M FLByteStringCluster(FLIteratingCluster)>serializeInstances:
> 448054196: a(n) FLByteStringCluster
> 0xbff66d00 M
> FLByteStringCluster(FLIteratingCluster)>serializeInstancesStepWith:
> 448054196: a(n) FLByteStringCluster
> 0xbff66d1c M FLSerialization>clusterInstancesStepOf: 448053428: a(n)
> FLSerialization
> 0xbff66d38 M [] in FLSerialization>instancesStep 448053428: a(n)
> FLSerialization
> 0xbff66d58 M OrderedCollection>do: 449989736: a(n) OrderedCollection
> 0xbff66d74 M FLSerialization>instancesStep 448053428: a(n) FLSerialization
> 0xbff66d8c M FLSerialization>run 448053428: a(n) FLSerialization
> 0xbff66db0 I [] in FLSerializer>defaultSerialization 447550464: a(n)
> FLSerializer
> 0xbff66dd8 M [] in FLSerializer>serialize:on: 447550464: a(n) FLSerializer
> 0xbff66df8 M BlockClosure>ensure: 448052884: a(n) BlockClosure
> 0xbff66e18 M FLSerializer>serialize:on: 447550464: a(n) FLSerializer
> 0xbff66e3c M [] in
> FLMethodContextSerializationTest(FLSerializationTest)>serialize: 447235380:
> a(n) FLMethodContextSerializationTest
> 0xbff66e60 M [] in GZipWriteStream class(PositionableStream class)>on:do:
> 419845144: a(n) GZipWriteStream class
> 0xbff66e80 M BlockClosure>ensure: 448048616: a(n) BlockClosure
> 0xbff66ea0 M GZipWriteStream class(PositionableStream class)>on:do:
> 419845144: a(n) GZipWriteStream class
> 0xbff66ec4 M [] in FLGZipStrategy>writeStreamDo: 447550896: a(n)
> FLGZipStrategy
> 0xbff66ee0 M FLByteArrayStreamStrategy>writeStreamDo: 447550884: a(n)
> FLByteArrayStreamStrategy
> 0xbff66efc M FLGZipStrategy>writeStreamDo: 447550896: a(n) FLGZipStrategy
> 0xbff66f18 M
> FLMethodContextSerializationTest(FLSerializationTest)>serialize: 447235380:
> a(n) FLMethodContextSerializationTest
> 0xbff66f34 M
> FLMethodContextSerializationTest(FLSerializationTest)>resultOfSerializeAndMaterialize:
> 447235380: a(n) FLMethodContextSerializationTest
> 0xbff66f58 M FLMethodContextSerializationTest>testMethodContextThisContext
> 447235380: a(n) FLMethodContextSerializationTest
>
>
> any idea of where I should start to look at?
>
> Thanks
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>



--
===========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================

Reply | Threaded
Open this post in threaded view
|

Re: GZipWriteStream crashing my VM when serializing ?

Mariano Martinez Peck


On Sat, Dec 10, 2011 at 2:39 AM, John McIntosh <[hidden email]> wrote:
Mmm I recall a problem from long ago where the buffer passed into the
zip call was not big enough to store the results.
If you pass in a small chunk of data, why the buffer to hold the
results and the decode data is bigger than the buffer going in.
So the two sizes, input and output are?

Hi John. For writing we are using a particular buffered write stream which then delegates to the orignal stream. So at the end we should be doing the equivalent of:
GZipWriteStream on: (ByteArray new writeStream). That's the default for testing, then you could also do GZipWriteStream on a StandardStream...

Anyway, our write buffer has this method:

FLBufferedWriteStream >> flushBuffer
    position = 0 ifTrue: [ ^ self ].
    position = buffer size
        ifTrue: [
            stream nextPutAll: buffer ]
        ifFalse: [
            streamRespondsToNextPutAllStartingAt
                ifTrue: [ stream next: position putAll: buffer startingAt: 1 ]
                ifFalse: [ stream nextPutAll: (buffer copyFrom: 1 to: position) ] ].
    position := 0


From what I can see in the stack, it is crasing here when it sends  "stream next: position putAll: buffer startingAt: 1"
I put some flags here and 'buffer' size is 4096  (we choose that value) and the position was for example 325.

Is this what you asked or did I misunderstand ?

Thanks in advance!

 


On Fri, Dec 2, 2011 at 4:41 PM, Mariano Martinez Peck
<[hidden email]> wrote:
> Hi guys. We are doing some experiments with Fuel and GZip streams... Now I
> have a beatiful test like this:
>
> testMethodContextThisContext
>
>     | materializedContext context |
>     context := thisContext copy.
>     materializedContext := self resultOfSerializeAndMaterialize: context.
>
> and it crashes like crazy. From the VM crash stack, I can see:
>
>
> Smalltalk stack dump:
> 0xbff66ba4 M GZipWriteStream(DeflateStream)>deflateBlock 447551436: a(n)
> GZipWriteStream
> 0xbff66bc8 M GZipWriteStream(DeflateStream)>nextPutAll: 447551436: a(n)
> GZipWriteStream
> 0xbff66be4 M GZipWriteStream(DeflateStream)>next:putAll:startingAt:
> 447551436: a(n) GZipWriteStream
> 0xbff66c08 M FLBufferedWriteStream>flushBuffer 448048664: a(n)
> FLBufferedWriteStream
> 0xbff66c20 M FLBufferedWriteStream>nextBytesPutAll: 448048664: a(n)
> FLBufferedWriteStream
> 0xbff66c40 M FLBufferedWriteStream>nextStringPut: 448048664: a(n)
> FLBufferedWriteStream
> 0xbff66c5c M FLByteStringCluster>serialize:on: 448054196: a(n)
> FLByteStringCluster
> 0xbff66c7c M FLByteStringCluster(FLPrimitiveCluster)>serializeInstance:with:
> 448054196: a(n) FLByteStringCluster
> 0xbff66ca0 M [] in
> FLByteStringCluster(FLIteratingCluster)>serializeInstances: 448054196: a(n)
> FLByteStringCluster
> 0xbff66cc8 M IdentitySet(Set)>do: 451711680: a(n) IdentitySet
> 0xbff66ce4 M FLByteStringCluster(FLIteratingCluster)>serializeInstances:
> 448054196: a(n) FLByteStringCluster
> 0xbff66d00 M
> FLByteStringCluster(FLIteratingCluster)>serializeInstancesStepWith:
> 448054196: a(n) FLByteStringCluster
> 0xbff66d1c M FLSerialization>clusterInstancesStepOf: 448053428: a(n)
> FLSerialization
> 0xbff66d38 M [] in FLSerialization>instancesStep 448053428: a(n)
> FLSerialization
> 0xbff66d58 M OrderedCollection>do: 449989736: a(n) OrderedCollection
> 0xbff66d74 M FLSerialization>instancesStep 448053428: a(n) FLSerialization
> 0xbff66d8c M FLSerialization>run 448053428: a(n) FLSerialization
> 0xbff66db0 I [] in FLSerializer>defaultSerialization 447550464: a(n)
> FLSerializer
> 0xbff66dd8 M [] in FLSerializer>serialize:on: 447550464: a(n) FLSerializer
> 0xbff66df8 M BlockClosure>ensure: 448052884: a(n) BlockClosure
> 0xbff66e18 M FLSerializer>serialize:on: 447550464: a(n) FLSerializer
> 0xbff66e3c M [] in
> FLMethodContextSerializationTest(FLSerializationTest)>serialize: 447235380:
> a(n) FLMethodContextSerializationTest
> 0xbff66e60 M [] in GZipWriteStream class(PositionableStream class)>on:do:
> 419845144: a(n) GZipWriteStream class
> 0xbff66e80 M BlockClosure>ensure: 448048616: a(n) BlockClosure
> 0xbff66ea0 M GZipWriteStream class(PositionableStream class)>on:do:
> 419845144: a(n) GZipWriteStream class
> 0xbff66ec4 M [] in FLGZipStrategy>writeStreamDo: 447550896: a(n)
> FLGZipStrategy
> 0xbff66ee0 M FLByteArrayStreamStrategy>writeStreamDo: 447550884: a(n)
> FLByteArrayStreamStrategy
> 0xbff66efc M FLGZipStrategy>writeStreamDo: 447550896: a(n) FLGZipStrategy
> 0xbff66f18 M
> FLMethodContextSerializationTest(FLSerializationTest)>serialize: 447235380:
> a(n) FLMethodContextSerializationTest
> 0xbff66f34 M
> FLMethodContextSerializationTest(FLSerializationTest)>resultOfSerializeAndMaterialize:
> 447235380: a(n) FLMethodContextSerializationTest
> 0xbff66f58 M FLMethodContextSerializationTest>testMethodContextThisContext
> 447235380: a(n) FLMethodContextSerializationTest
>
>
> any idea of where I should start to look at?
>
> Thanks
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>



--
===========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================




--
Mariano
http://marianopeck.wordpress.com