HTTP inefficancies

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

HTTP inefficancies

Joerg Beekmann, DeepCove Labs (YVR)

 

Using HTTPClient to post files results in excessive memory consumption. For example posting a 54MB file will grow the image to over 512MB in size. If the memory is not available the upload will fail. The culprit seems to be that the body is composed internally before posting even though we provide and external write stream to the entity as its source. See writeContentsOn: aStream in SimpleBody. Below is the stack.  

 

Joerg

 

 

Unhandled exception: No space available to allocate this object

ByteArray class(Behavior)>>newNoRetry:

ByteArray class(Behavior)>>handleFailedNew:size:

ByteArray class(Behavior)>>new:

ByteArray(Collection)>>copyEmpty:

ByteArray(SequenceableCollection)>>changeSizeTo:

ByteArray(SequenceableCollection)>>growToAtLeast:

WriteStream(InternalStream)>>pastEndPut:

WriteStream>>nextPut:

EncodedStream>>nextPut:

optimized [] in Net.HttpPrintHandler>>printBodyStream:

BlockClosure>>ensure:

Net.HttpPrintHandler>>printBodyStream:

Net.HttpPrintHandler(Net.MessagePrintHandler)>>startBodyStream:

Net.HttpPrintHandler(Net.MessageHandler)>>action:with:

Net.HttpDispatcher(Net.MessageDispatcher)>>action:with:

optimized [] in [] in Net.MessageDispatcher>>acceptSimpleBody:with:

BlockClosure>>ensure:

optimized [] in Net.MessageDispatcher>>acceptSimpleBody:with:

BlockClosure>>ensure:

Net.HttpDispatcher(Net.MessageDispatcher)>>acceptSimpleBody:with:

Net.SimpleBody>>acceptVisit:with:

Net.HttpDispatcher(Net.MessageDispatcher)>>visit:with:

Net.HttpDispatcher(Net.MessageDispatcher)>>visit:

optimized [] in Net.MessageDispatcher>>acceptMimeEntity:with:

BlockClosure>>ensure:

Net.HttpDispatcher(Net.MessageDispatcher)>>acceptMimeEntity:with:

Net.MimeEntity>>acceptVisit:with:

Net.HttpDispatcher(Net.MessageDispatcher)>>visit:with:

Net.HttpDispatcher(Net.MessageDispatcher)>>visit:

optimized [] in [] in [] in Net.MessageDispatcher>>acceptMultipartBody:with:

BlockClosure>>ensure:

optimized [] in [] in Net.MessageDispatcher>>acceptMultipartBody:with:

OrderedCollection>>do:

optimized [] in Net.MessageDispatcher>>acceptMultipartBody:with:

BlockClosure>>ensure:

Net.HttpDispatcher(Net.MessageDispatcher)>>acceptMultipartBody:with:

Net.MultipartBody>>acceptVisit:with:

Net.HttpDispatcher(Net.MessageDispatcher)>>visit:with:

Net.HttpDispatcher(Net.MessageDispatcher)>>visit:

optimized [] in Net.HttpDispatcher>>acceptHttpEntity:with:

BlockClosure>>ensure:

Net.HttpDispatcher>>acceptHttpEntity:with:

Net.HttpRequest(Net.HttpEntity)>>acceptVisit:with:

Net.HttpDispatcher(Net.MessageDispatcher)>>visit:with:

Net.HttpDispatcher(Net.MessageDispatcher)>>visit:

Net.HttpPrintHandler(Net.MessageHandler)>>visit:

Net.HttpPrintHandler(Net.MessagePrintHandler)>>writeMessage:on:

Net.HttpRequest(Net.MimeEntity)>>writeOn:

Net.WebClient(Net.HttpClient)>>privateExecuteRequest:do:

Net.WebClient(Net.HttpClient)>>privateExecuteRequest:

optimized [] in Net.HttpClient>>executeRequestDo:

optimized [] in Net.HttpClient>>safelyExecuteBlock:

BlockClosure>>ensure:

Net.WebClient(Net.HttpClient)>>safelyExecuteBlock:

Net.WebClient(Net.HttpClient)>>executeRequestDo:

Net.WebClient(Net.HttpClient)>>executeAndGetResponse

Net.WebClient>>post:formData:elementName:fileName:source:

 

-----
Joerg Beekmann
DeepCove Labs
4th floor 595 Howe Street
Vancouver, BC, V6C 2T5
voice +1.604.689.0322
fax   +1.604.689.0311
[hidden email]

 


CONFIDENTIALITY NOTICE
Unless otherwise indicated this email contains information that is private
and confidential. If you have received it in error, please notify the sender
and delete this message along with any attachments.

 

Reply | Threaded
Open this post in threaded view
|

Re: HTTP inefficancies

kobetic
Joerg,

This is a known limitation and has been discussed a few times already,
usually referred to as the "streaming" approach to HTTP. It definitely
sounds as an attractive possibility and we would like to try it some
time. However there's quite realistic possibility that it may not be all
that obvious as it seems at first. MIME and HTTP is a peculiar
partnership and being abused the way it gets to be these days it makes
for some very interesting use-cases. Just take the one you're bringing
up, a formerly "hyper-text transfer protocol" is being used to shove
around hundreds of megabytes of binary data (even though there always
have been better alternatives designed specifically for that purpose).

You certainly can design the framework geared towards specific purpose,
but it may not be able to handle the gazillions of other valid options
in MIME/HTTP. MIME generously allows to take wildly different types of
data, combine them into one message, and apply layers of encoding,
compressing, stretching, cutting up to pieces and recombining them
again. To make it even more fun it's not that unusual in practice that
sometimes a message doesn't even have enough information about the
processing that was applied to it, so you may need to go back and try
again (Things like web servers not providing or outright lying about the
content-type and on the other hand HTTP-EQUIV tags buried in HTML
payload, things like that). So I'm seriously concerned that after
spending a year rewriting the framework from the ground up, we'll end up
with something that can shuffle around arbitrarily large files, but
won't be able to support half of the use-cases that the old one could.
I'm not saying it's not worth trying, it would quite likely yield some
interesting insights. But I'm not sure when we (Cincom) will be able to
afford this exercise.

Anyway, I'm not sure how useful my rant here is, I just wanted to
provide my current view of this issue.

Best regards,

Martin

Joerg Beekmann wrote:
>
> Using HTTPClient to post files results in excessive memory consumption.
> For example posting a 54MB file will grow the image to over 512MB in
> size. If the memory is not available the upload will fail. The culprit
> seems to be that the body is composed internally before posting even
> though we provide and external write stream to the entity as its source.
> See writeContentsOn: aStream in SimpleBody. Below is the stack.  
>