HTTP inefficancies
Locked
HTTP inefficancies
 
|
Using HTTPClient to post files results in excessive memory consumption.
For example posting a 54MB file will grow the image to over 512MB in size. If
the memory is not available the upload will fail. The culprit seems to be that the
body is composed internally before posting even though we provide and external
write stream to the entity as its source. See writeContentsOn: aStream in
SimpleBody. Below is the stack. Joerg Unhandled exception: No space available to allocate this
object ByteArray class(Behavior)>>newNoRetry: ByteArray class(Behavior)>>handleFailedNew:size: ByteArray class(Behavior)>>new: ByteArray(Collection)>>copyEmpty: ByteArray(SequenceableCollection)>>changeSizeTo: ByteArray(SequenceableCollection)>>growToAtLeast: WriteStream(InternalStream)>>pastEndPut: WriteStream>>nextPut: EncodedStream>>nextPut: optimized [] in Net.HttpPrintHandler>>printBodyStream: BlockClosure>>ensure: Net.HttpPrintHandler>>printBodyStream: Net.HttpPrintHandler(Net.MessagePrintHandler)>>startBodyStream: Net.HttpPrintHandler(Net.MessageHandler)>>action:with: Net.HttpDispatcher(Net.MessageDispatcher)>>action:with: optimized [] in [] in
Net.MessageDispatcher>>acceptSimpleBody:with: BlockClosure>>ensure: optimized [] in
Net.MessageDispatcher>>acceptSimpleBody:with: BlockClosure>>ensure: Net.HttpDispatcher(Net.MessageDispatcher)>>acceptSimpleBody:with: Net.SimpleBody>>acceptVisit:with: Net.HttpDispatcher(Net.MessageDispatcher)>>visit:with: Net.HttpDispatcher(Net.MessageDispatcher)>>visit: optimized [] in
Net.MessageDispatcher>>acceptMimeEntity:with: BlockClosure>>ensure: Net.HttpDispatcher(Net.MessageDispatcher)>>acceptMimeEntity:with: Net.MimeEntity>>acceptVisit:with: Net.HttpDispatcher(Net.MessageDispatcher)>>visit:with: Net.HttpDispatcher(Net.MessageDispatcher)>>visit: optimized [] in [] in [] in Net.MessageDispatcher>>acceptMultipartBody:with: BlockClosure>>ensure: optimized [] in [] in
Net.MessageDispatcher>>acceptMultipartBody:with: OrderedCollection>>do: optimized [] in
Net.MessageDispatcher>>acceptMultipartBody:with: BlockClosure>>ensure: Net.HttpDispatcher(Net.MessageDispatcher)>>acceptMultipartBody:with: Net.MultipartBody>>acceptVisit:with: Net.HttpDispatcher(Net.MessageDispatcher)>>visit:with: Net.HttpDispatcher(Net.MessageDispatcher)>>visit: optimized [] in
Net.HttpDispatcher>>acceptHttpEntity:with: BlockClosure>>ensure: Net.HttpDispatcher>>acceptHttpEntity:with: Net.HttpRequest(Net.HttpEntity)>>acceptVisit:with: Net.HttpDispatcher(Net.MessageDispatcher)>>visit:with: Net.HttpDispatcher(Net.MessageDispatcher)>>visit: Net.HttpPrintHandler(Net.MessageHandler)>>visit: Net.HttpPrintHandler(Net.MessagePrintHandler)>>writeMessage:on: Net.HttpRequest(Net.MimeEntity)>>writeOn: Net.WebClient(Net.HttpClient)>>privateExecuteRequest:do: Net.WebClient(Net.HttpClient)>>privateExecuteRequest: optimized [] in Net.HttpClient>>executeRequestDo: optimized [] in Net.HttpClient>>safelyExecuteBlock: BlockClosure>>ensure: Net.WebClient(Net.HttpClient)>>safelyExecuteBlock: Net.WebClient(Net.HttpClient)>>executeRequestDo: Net.WebClient(Net.HttpClient)>>executeAndGetResponse Net.WebClient>>post:formData:elementName:fileName:source: -----
|
|
|
Joerg,
This is a known limitation and has been discussed a few times already, usually referred to as the "streaming" approach to HTTP. It definitely sounds as an attractive possibility and we would like to try it some time. However there's quite realistic possibility that it may not be all that obvious as it seems at first. MIME and HTTP is a peculiar partnership and being abused the way it gets to be these days it makes for some very interesting use-cases. Just take the one you're bringing up, a formerly "hyper-text transfer protocol" is being used to shove around hundreds of megabytes of binary data (even though there always have been better alternatives designed specifically for that purpose). You certainly can design the framework geared towards specific purpose, but it may not be able to handle the gazillions of other valid options in MIME/HTTP. MIME generously allows to take wildly different types of data, combine them into one message, and apply layers of encoding, compressing, stretching, cutting up to pieces and recombining them again. To make it even more fun it's not that unusual in practice that sometimes a message doesn't even have enough information about the processing that was applied to it, so you may need to go back and try again (Things like web servers not providing or outright lying about the content-type and on the other hand HTTP-EQUIV tags buried in HTML payload, things like that). So I'm seriously concerned that after spending a year rewriting the framework from the ground up, we'll end up with something that can shuffle around arbitrarily large files, but won't be able to support half of the use-cases that the old one could. I'm not saying it's not worth trying, it would quite likely yield some interesting insights. But I'm not sure when we (Cincom) will be able to afford this exercise. Anyway, I'm not sure how useful my rant here is, I just wanted to provide my current view of this issue. Best regards, Martin Joerg Beekmann wrote: > > Using HTTPClient to post files results in excessive memory consumption. > For example posting a 54MB file will grow the image to over 512MB in > size. If the memory is not available the upload will fail. The culprit > seems to be that the body is composed internally before posting even > though we provide and external write stream to the entity as its source. > See writeContentsOn: aStream in SimpleBody. Below is the stack. > |
| Free forum by Nabble | Edit this page |