Smalltalk › Gemtalk › GLASS

Large request via HTTP - how is this handled ?

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

6 messages Options

GLASS mailing list

Large request via HTTP - how is this handled ?

Hello,

how is this handled in Gemstoine 3.4.x ? I send a request via HTTP to Gemstone - say a 1.5 GB upload. Gemstone/S will never be able to handle this .

How is this handled by the Gemstone/S low level socket ? How large is the internal buffer of that socket ?

Marten

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Large request via HTTP - how is this handled ?

I assume that you are using Zinc? If so then it seems that ZnClient>>downloadTo: allows you to stream the contents of an http request to disk, so I assume that something similar can be done on the server-side ...

It seems that some custom code would be called for, but the specific depends upon what frameworks you are using and where the 1.5GB are coming from.

Dale

On 02/07/2018 11:42 AM, Marten Feldtmann via Glass wrote:

Hello,

how is this handled in Gemstoine 3.4.x ? I send a request via HTTP to Gemstone - say a 1.5 GB upload. Gemstone/S will never be able to handle this .

How is this handled by the Gemstone/S low level socket ? How large is the internal buffer of that socket ?

Marten
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Large request via HTTP - how is this handled ?

We have noticed last week - for the first time since I've used ZeroMQ in Gemstone - that ZMQ throws an exception (telling me, that the external socket handle - held in external non Smalltalk RAM - is not valid any more).

And it happens on a regular way, that the exception is thrown after I aborted (at 8MB) a too large http request coming into Gemstone.

We have now limit the size of the uploaded files in our clients .. to get rid of this

Marten

Dale Henrichs via Glass <[hidden email]> hat am 8. Februar 2018 um 01:49 geschrieben:

I assume that you are using Zinc? If so then it seems that ZnClient>>downloadTo: allows you to stream the contents of an http request to disk, so I assume that something similar can be done on the server-side ...
It seems that some custom code would be called for, but the specific depends upon what frameworks you are using and where the 1.5GB are coming from.
Dale
On 02/07/2018 11:42 AM, Marten Feldtmann via Glass wrote:
Hello,
how is this handled in Gemstoine 3.4.x ? I send a request via HTTP to Gemstone - say a 1.5 GB upload. Gemstone/S will never be able to handle this .
How is this handled by the Gemstone/S low level socket ? How large is the internal buffer of that socket ?

Marten
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Large request via HTTP - how is this handled ?

In reply to this post by GLASS mailing list

There's also this strategy (albeit from 2011 so mYMMV) which uses nginx to
handle the data, bypassing the image entirely

http://nickager.com/blog/2011/07/01/File-upload-using-Nginx-and-Seaside

GLASS mailing list wrote

> <!DOCTYPE html>
>
>
> Hello, how is this handled in Gemstoine 3.4.x ? I send a request via HTTP
> to Gemstone - say a 1.5 GB upload. Gemstone/S will never be able to handle
> this . How is this handled by the Gemstone/S low level socket ? How large
> is the internal buffer of that socket ? Marten
>
>
> _______________________________________________
> Glass mailing list

> Glass@.gemtalksystems

> http://lists.gemtalksystems.com/mailman/listinfo/glass

--
Sent from: http://forum.world.st/GLASS-f1460844.html
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Large request via HTTP - how is this handled ?

> On 08.02.2018, at 18:01, Paul DeBruicker via Glass <[hidden email]> wrote:
>
> There's also this strategy (albeit from 2011 so mYMMV) which uses nginx to
> handle the data, bypassing the image entirely
>
>
> http://nickager.com/blog/2011/07/01/File-upload-using-Nginx-and-Seaside

I'm afraid the "modern" nginx versions do not support the upload module anymore.

However, this can help: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_in_file_only

Best regards
-Tobias

> GLASS mailing list wrote
>> <!DOCTYPE html>
>>
>>
>> Hello, how is this handled in Gemstoine 3.4.x ? I send a request via HTTP
>> to Gemstone - say a 1.5 GB upload. Gemstone/S will never be able to handle
>> this . How is this handled by the Gemstone/S low level socket ? How large
>> is the internal buffer of that socket ? Marten
>>
>>
>> _______________________________________________
>> Glass mailing list
>
>> Glass@.gemtalksystems
>
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
>
>
>
>
> --
> Sent from: http://forum.world.st/GLASS-f1460844.html
> _______________________________________________
> Glass mailing list
> [hidden email]
> http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Large request via HTTP - how is this handled ?

In reply to this post by GLASS mailing list

Marten,

The goal of streams is that you don’t have to handle everything at once, so size shouldn’t matter. Typically one would, in a loop, read a chunk from the socket stream and write that chunk to the file stream. The loop would continue until the entire upload is processed. With this, the internal buffer could be 1KB or 1MB, it wouldn’t matter. Such a loop could keep reading till the disk was filled, so effectively no limit.

As others have mentioned, the actual implementation will depend on your framework and if it follows this model. See https://github.com/jgfoster/WebGS/blob/master/WebGS.gs for the HTTP framework used by GemTalk’s internal bug tracking database (which is always running on the latest release, so now 3.4.1). In this example the “delegate” (or application built on the framework) is passed an HttpRequest (which includes a socket) and can handle it by reading chunks.

James

On Feb 7, 2018, at 11:42 AM, Marten Feldtmann via Glass <[hidden email]> wrote:

Hello,
how is this handled in Gemstoine 3.4.x ? I send a request via HTTP to Gemstone - say a 1.5 GB upload. Gemstone/S will never be able to handle this .
How is this handled by the Gemstone/S low level socket ? How large is the internal buffer of that socket ?

Marten
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass