OpenTalk STST Marshaling Fails for ByteArray Sized 16 MB

Is there a object size limit when using OpenTalk STST Marshaler? I am testing with a large ByteArray, and it seems like the marshaler fails when getting an array where the size is 16 MB (16 * 1024 * 1024 bytes) , or larger.

Kind regards
Runar

Stack trace below:

Break in Promise>>exception: at 233
Protocols.TimedPromise(Promise)>>exception:
Opentalk.STSTRequest(Opentalk.RemoteRequest)>>shutdown
optimized [] in Opentalk.RequestTransport>>postStop
[] in Opentalk.OtWeakValueDictionary(Opentalk.OtWeakKeyDictionary)>>do:
BlockClosure>>ensure:
optimized [] in RecursionLock>>critical:
BlockClosure>>ensure:
Semaphore>>critical:
RecursionLock>>critical:
Opentalk.OtWeakValueDictionary(Opentalk.OtWeakKeyDictionary)>>do:
Opentalk.TCPTransport(Opentalk.RequestTransport)>>postStop
Opentalk.TCPTransport(Opentalk.Transport)>>stop
Opentalk.TCPTransport>>handleIncomingMessageError:
optimized [] in Opentalk.RequestTransport>>handlingIncomingMessage
BlockClosure>>cull:
Opentalk.OtEMarshaling(GenericException)>>performHandler:
Opentalk.OtEMarshaling(GenericException)>>propagatePrivateFrom:
Opentalk.OtEMarshaling(GenericException)>>propagateFrom:
Opentalk.OtEMarshaling(GenericException)>>propagate
Opentalk.OtEMarshaling(GenericException)>>raiseSignal
Opentalk.OtEMarshaling class(GenericException class)>>raiseSignal:
optimized [] in Opentalk.STSTStream>>unmarshalObject
BlockClosure>>cull:
SubscriptOutOfBoundsError(GenericException)>>performHandler:
SubscriptOutOfBoundsError(GenericException)>>propagatePrivateFrom:
SubscriptOutOfBoundsError(GenericException)>>propagateFrom:
SubscriptOutOfBoundsError(GenericException)>>propagate
SubscriptOutOfBoundsError(GenericException)>>raiseRequest
Opentalk.OtWeakValueDictionary(Object)>>subscriptBoundsError:
Opentalk.OtWeakValueDictionary(Opentalk.OtWeakKeyDictionary)>>at:ifAbsent:
Opentalk.TCPTransport(Opentalk.RequestTransport)>>requestById:
Opentalk.STSTStream>>nextSTSTReply
optimized [] in Opentalk.STSTStream class>>initializeTagDispatchTable
Opentalk.STSTStream>>unmarshalObjectTagged:
Opentalk.STSTStream>>unmarshalObjectRecursively
optimized [] in Opentalk.STSTStream>>unmarshalObject
BlockClosure>>on:do:
Opentalk.STSTStream>>unmarshalObject
Opentalk.STSTMarshaler>>unmarshalTransportPayloadFrom:
Opentalk.TCPTransport(Opentalk.RequestTransport)>>createApplicationPackageFrom:
Opentalk.TCPTransport(Opentalk.RequestTransport)>>nextPackage
optimized [] in Opentalk.RequestTransport>>handlingIncomingMessage
BlockClosure>>on:do:
Opentalk.TCPTransport(Opentalk.RequestTransport)>>handlingIncomingMessage
Opentalk.TCPTransport(Opentalk.RequestTransport)>>serverProcessBody
optimized [] in Opentalk.StreamTransport>>serverProcessBody
BlockClosure>>ensure:
Opentalk.TCPTransport(Opentalk.StreamTransport)>>serverProcessBody
optimized [] in Opentalk.Transport>>startServerProcess
BlockClosure>>on:do:
optimized [] in Process class>>forBlock:priority:




----------------------------------------------------------------------
Protocols.TimedPromise(Promise)>>exception:
Receiver:
       a Protocols.TimedPromise
Instance Variables:
       sync = a Semaphore[1]
       value = nil
       hasValue = false
       exception = nil
       delay = a Delay
Arguments:
       anException = an Opentalk.OtEServerError{Connection Closed}
Context PC = 7




----------------------------------------------------------------------
Opentalk.STSTRequest(Opentalk.RemoteRequest)>>shutdown
Receiver:
       a Opentalk.STSTRequest
Instance Variables:
       requestId = 1
       interceptorDispatcher = an Opentalk.InterceptorDispatcher
       target = ObjRef<#service@192.168.2.46:2246>
       message = a Message with selector: #add:and: and arguments: #(1 2)
       timeout = 60000
       promise = a Protocols.TimedPromise
       reply = nil
       serviceContext = an OrderedCollection[1]
Context PC = 27




----------------------------------------------------------------------
optimized [] in Opentalk.RequestTransport>>postStop
Receiver:
       an UndefinedObject
Arguments:
       each = STSTRequest[1]{ObjRef<#service@192.168.2.46:2246> add: 1 and: 2}
Context PC = 3




----------------------------------------------------------------------
[] in Opentalk.OtWeakValueDictionary(Opentalk.OtWeakKeyDictionary)>>do:
Receiver:
       an Opentalk.OtWeakValueDictionary
Instance Variables:
       tally = 1
       keys = an Array[101]
       values = a WeakArray[101]
       accessLock = a RecursionLock
       minimumSize = nil
       growBound = nil
       growRatio = nil
       shrinkBound = 0.1
       shrinkRatio = nil
Temporaries:
       index = 2
       key = 1
       val = STSTRequest[1]{ObjRef<#service@192.168.2.46:2246> add: 1 and: 2}
       .self = an Opentalk.OtWeakValueDictionary
       .aBlock = BlockClosure [] in Opentalk.RequestTransport>>postStop
Context PC = 54




----------------------------------------------------------------------
BlockClosure>>ensure:
Receiver:
       a BlockClosure
Instance Variables:
       method = CompiledBlock [] in Opentalk.OtWeakKeyDictionary>>do:
       outerContext = Opentalk.OtWeakValueDictionary(Opentalk.OtWeakKeyDictionary)>>do:
       copiedValues = an Array[2]
Arguments:
       aBlock = BlockClosure [] in [] in RecursionLock>>critical:
Temporaries:
       result = nil
Context PC = 4




----------------------------------------------------------------------
optimized [] in RecursionLock>>critical:
Receiver:
       an UndefinedObject
Temporaries:
       .activeProcess = a Process in Semaphore>>signal
       .self = a RecursionLock
       .aBlock = BlockClosure [] in Opentalk.OtWeakKeyDictionary>>do:
Context PC = 12




----------------------------------------------------------------------
BlockClosure>>ensure:
Receiver:
       a BlockClosure
Instance Variables:
       method = CompiledBlock [] in RecursionLock>>critical:
       outerContext = nil
       copiedValues = an Array[3]
Arguments:
       aBlock = BlockClosure [] in Semaphore>>critical:
Temporaries:
       result = nil
Context PC = 4




----------------------------------------------------------------------
Semaphore>>critical:
Receiver:
       a Semaphore
Instance Variables:
       firstLink = nil
       lastLink = nil
       excessSignals = 0
Arguments:
       mutuallyExcludedBlock = BlockClosure [] in RecursionLock>>critical:
Context PC = 10




----------------------------------------------------------------------
RecursionLock>>critical:
Receiver:
       a RecursionLock
Instance Variables:
       owner = a Process in Semaphore>>signal
       semaphore = a Semaphore[0]
Arguments:
       aBlock = BlockClosure [] in Opentalk.OtWeakKeyDictionary>>do:
Temporaries:
       activeProcess = a Process in Semaphore>>signal
Context PC = 20




----------------------------------------------------------------------
Opentalk.OtWeakValueDictionary(Opentalk.OtWeakKeyDictionary)>>do:
Receiver:
       an Opentalk.OtWeakValueDictionary
Instance Variables:
       tally = 1
       keys = an Array[101]
       values = a WeakArray[101]
       accessLock = a RecursionLock
       minimumSize = nil
       growBound = nil
       growRatio = nil
       shrinkBound = 0.1
       shrinkRatio = nil
Arguments:
       aBlock = BlockClosure [] in Opentalk.RequestTransport>>postStop
Context PC = 11




----------------------------------------------------------------------
Opentalk.TCPTransport(Opentalk.RequestTransport)>>postStop
Receiver:
       a Opentalk.TCPTransport
Instance Variables:
       handlers = nil
       state = #stopped
       manager = Adaptor<tcp/192.168.2.59:2247>
       serverProcess = nil
       sendLock = a RecursionLock
       marshaler = an Opentalk.STSTMarshaler
       unmarshaler = an Opentalk.STSTMarshaler
       readBuffer = a Opentalk.TransportPackageBytes[16777238]
       writeBuffer = a Opentalk.TransportPackageBytes[1024]
       configuration = TCPTransportConfiguration new
       marshaler: (STSTMarshalerConfiguration new)
       requestRegistry = an Opentalk.OtWeakValueDictionary
       nextRequestId = 1
       sessionObjects = nil
       id = IPSocketAddress(192.168.2.46:2246)
       peerId = nil
       writeStream = nil
       readStream = nil
       socket = nil
       timeout = 1200000
Context PC = 10




----------------------------------------------------------------------
Opentalk.TCPTransport(Opentalk.Transport)>>stop
Receiver:
       a Opentalk.TCPTransport
Instance Variables:
       handlers = nil
       state = #stopped
       manager = Adaptor<tcp/192.168.2.59:2247>
       serverProcess = nil
       sendLock = a RecursionLock
       marshaler = an Opentalk.STSTMarshaler
       unmarshaler = an Opentalk.STSTMarshaler
       readBuffer = a Opentalk.TransportPackageBytes[16777238]
       writeBuffer = a Opentalk.TransportPackageBytes[1024]
       configuration = TCPTransportConfiguration new
       marshaler: (STSTMarshalerConfiguration new)
       requestRegistry = an Opentalk.OtWeakValueDictionary
       nextRequestId = 1
       sessionObjects = nil
       id = IPSocketAddress(192.168.2.46:2246)
       peerId = nil
       writeStream = nil
       readStream = nil
       socket = nil
       timeout = 1200000
Context PC = 20




----------------------------------------------------------------------
Opentalk.TCPTransport>>handleIncomingMessageError:
Receiver:
       a Opentalk.TCPTransport
Instance Variables:
       handlers = nil
       state = #stopped
       manager = Adaptor<tcp/192.168.2.59:2247>
       serverProcess = nil
       sendLock = a RecursionLock
       marshaler = an Opentalk.STSTMarshaler
       unmarshaler = an Opentalk.STSTMarshaler
       readBuffer = a Opentalk.TransportPackageBytes[16777238]
       writeBuffer = a Opentalk.TransportPackageBytes[1024]
       configuration = TCPTransportConfiguration new
       marshaler: (STSTMarshalerConfiguration new)
       requestRegistry = an Opentalk.OtWeakValueDictionary
       nextRequestId = 1
       sessionObjects = nil
       id = IPSocketAddress(192.168.2.46:2246)
       peerId = nil
       writeStream = nil
       readStream = nil
       socket = nil
       timeout = 1200000
Arguments:
       anException = an Opentalk.OtEMarshaling{Subscript out of bounds: }
Context PC = 9




----------------------------------------------------------------------
optimized [] in Opentalk.RequestTransport>>handlingIncomingMessage
Receiver:
       an UndefinedObject
Arguments:
       ex = an Opentalk.OtEMarshaling{Subscript out of bounds: }
Temporaries:
       .self = Transport<tcp/192.168.2.46:2246>
Context PC = 7




----------------------------------------------------------------------
BlockClosure>>cull:
Receiver:
       a BlockClosure
Instance Variables:
       method = CompiledBlock [] in Opentalk.RequestTransport>>handlingIncomingMessage
       outerContext = nil
       copiedValues = Transport<tcp/192.168.2.46:2246>
Arguments:
       anObject = an Opentalk.OtEMarshaling{Subscript out of bounds: }
Context PC = 15




----------------------------------------------------------------------
Opentalk.OtEMarshaling(GenericException)>>performHandler:
Receiver:
       an Opentalk.OtEMarshaling
Instance Variables:
       messageText = 'Subscript out of bounds: '
       originator = nil
       initialContext = optimized [] in Opentalk.STSTStream>>unmarshalObject
       firstUnwindContext = nil
       handlerContext = BlockClosure>>on:do:
       parameter = nil
       proceedBlock = nil
       isResumable = false
       signal = nil
       searchContext = optimized [] in Opentalk.STSTStream>>unmarshalObject
Arguments:
       aContext = BlockClosure>>on:do:
Temporaries:
       result = nil
       oldHandler = BlockClosure>>on:do:
Context PC = 39




----------------------------------------------------------------------
Opentalk.OtEMarshaling(GenericException)>>propagatePrivateFrom:
Receiver:
       an Opentalk.OtEMarshaling
Instance Variables:
       messageText = 'Subscript out of bounds: '
       originator = nil
       initialContext = optimized [] in Opentalk.STSTStream>>unmarshalObject
       firstUnwindContext = nil
       handlerContext = BlockClosure>>on:do:
       parameter = nil
       proceedBlock = nil
       isResumable = false
       signal = nil
       searchContext = optimized [] in Opentalk.STSTStream>>unmarshalObject
Arguments:
       aContext = optimized [] in Opentalk.STSTStr
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

"Runar Jordahl"<[hidden email]> wrote:
> Is there a object size limit when using OpenTalk STST Marshaler? I am
> testing with a large ByteArray, and it seems like the marshaler fails when
> getting an array where the size is 16 MB (16 * 1024 * 1024 bytes) , or
> larger.

I'm not aware of such a limit, but there are some practical limits. Probably the most severe is the fact that STST currently marshals the entire message into memory before it is sent out. This forces the marshaling buffers to grow to accommodate that and they don't shrink. So they'll keep hogging that memory until the broker shuts down or the associated connection expires due to inactivity. So if you need to send something large, it's currently best to chop it up into pieces and reassemble it on the other side.

Anyway, your stack below suggests a different problem, this bit specifically:

...
> SubscriptOutOfBoundsError(GenericException)>>raiseRequest
> Opentalk.OtWeakValueDictionary(Object)>>subscriptBoundsError:
> Opentalk.OtWeakValueDictionary(Opentalk.OtWeakKeyDictionary)>>at:ifAbsent:
> Opentalk.TCPTransport(Opentalk.RequestTransport)>>requestById:
> Opentalk.STSTStream>>nextSTSTReply
....

So the client is trying to unmarshal a reply and is failing to find the corresponding request. Not sure why would that happen, but that's where I'd start looking.

HTH,

Martin

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Thanks for your answer!

My test sets up a connection at the client side like this:
RequestBroker newStstTcpAtPort: nnn

On the server side I answer the following:
ByteArray new: 16 * 1024 * 1024

> So the client is trying to unmarshal a reply and is failing to find the corresponding request. Not sure why would that happen, but that's where I'd start looking.

Well, but if I reduce the size of the ByteArray transferred using
(ByteArray new: 16 * 1024 * 1024 - 1) , the message send goes through
OK. So somewhere in the system, there is a 16 MB limit.

Runar
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Hi Runar,

There is indeed a limit.

On OpenTalk.STSTStream there's this method:

nextPutLength: anInteger

        ( anInteger >= 255 )
                ifTrue: [ self
                                                nextPut: 255;
                                                nextPut: ( ( anInteger bitShift: -16 ) bitAnd: 16rFF );
                                                nextPut: ( ( anInteger bitShift: -8 ) bitAnd: 16rFF );
                                                nextPut: ( anInteger bitAnd: 16rFF )
                                ]
                ifFalse: [ self nextPut: anInteger
                                ]


Which is used by #nextPutByteArray: (and all other collection-like put methods).

As you can see, only 3 bytes are used to store the length of an array on the stream. 3 bytes can only cover 16MB.


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Runar Jordahl
Sent: woensdag 25 januari 2012 19:36
To: [hidden email]
Cc: [hidden email]
Subject: Re: [vwnc] OpenTalk STST Marshaling Fails for ByteArray Sized 16 MB

Thanks for your answer!

My test sets up a connection at the client side like this:
RequestBroker newStstTcpAtPort: nnn

On the server side I answer the following:
ByteArray new: 16 * 1024 * 1024

> So the client is trying to unmarshal a reply and is failing to find the corresponding request. Not sure why would that happen, but that's where I'd start looking.

Well, but if I reduce the size of the ByteArray transferred using (ByteArray new: 16 * 1024 * 1024 - 1) , the message send goes through OK. So somewhere in the system, there is a 16 MB limit.

Runar
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc