Managing Large File Uploads in Seaside

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Managing Large File Uploads in Seaside

Esteban A. Maringolo
I'm trying to use the SiouX responder fortification to limit a large file upload from blocking the whole image and/or filling its memory.

Looking at the options I found there is a "save attachments as files", which seems like a similar approach as the one proposed at <https://jbrichau.github.io/blog/large-file-upload-in-seaside > using NGINX or Apache, but apparently that option only works at NetHttpResponder, but it's not applied in Seaside.

Did anybody integrate this feature or a similar one with Seaside in VW? (otherwise I'd have to do it myself).

Regards!

Esteban A. Maringolo

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: [vwnc] Managing Large File Uploads in Seaside

Esteban A. Maringolo
Hi Felix,

That is similar to what Johan proposes in his article, but the goal is to achieve the same thing by using SiouX (VW's latest HTTP server) responders, maybe using a chunked read/write approach (so there is no more than a certain buffer in the object memory).

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 9:46 AM Félix Madrid <[hidden email]> wrote:
Hi Esteban,

Maybe this article (and project) from Nick Ager can help you:


Cheers,

Félix

On Wed, May 8, 2019 at 2:39 PM Esteban Maringolo <[hidden email]> wrote:
I'm trying to use the SiouX responder fortification to limit a large file upload from blocking the whole image and/or filling its memory.

Looking at the options I found there is a "save attachments as files", which seems like a similar approach as the one proposed at <https://jbrichau.github.io/blog/large-file-upload-in-seaside > using NGINX or Apache, but apparently that option only works at NetHttpResponder, but it's not applied in Seaside.

Did anybody integrate this feature or a similar one with Seaside in VW? (otherwise I'd have to do it myself).

Regards!

Esteban A. Maringolo


_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: [vwnc] Managing Large File Uploads in Seaside

Esteban A. Maringolo
I found that SiouX is effectively uploading the contents to an attachment file directory.

Maybe there is a way to avoid instantiating a WAFile and use a WAExternalFile poiting to the file on disk, or simply using my custom SeasideResponder subclass.

This won't save VW from reading the whole file into memory before saving it to disk (or will it?) but it certainly will only keep the reference during the request/response of SiouX, which will be garbage collected faster than anything on a Seaside Component.

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 10:10 AM Esteban Maringolo <[hidden email]> wrote:
Hi Felix,

That is similar to what Johan proposes in his article, but the goal is to achieve the same thing by using SiouX (VW's latest HTTP server) responders, maybe using a chunked read/write approach (so there is no more than a certain buffer in the object memory).

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 9:46 AM Félix Madrid <[hidden email]> wrote:
Hi Esteban,

Maybe this article (and project) from Nick Ager can help you:


Cheers,

Félix

On Wed, May 8, 2019 at 2:39 PM Esteban Maringolo <[hidden email]> wrote:
I'm trying to use the SiouX responder fortification to limit a large file upload from blocking the whole image and/or filling its memory.

Looking at the options I found there is a "save attachments as files", which seems like a similar approach as the one proposed at <https://jbrichau.github.io/blog/large-file-upload-in-seaside > using NGINX or Apache, but apparently that option only works at NetHttpResponder, but it's not applied in Seaside.

Did anybody integrate this feature or a similar one with Seaside in VW? (otherwise I'd have to do it myself).

Regards!

Esteban A. Maringolo


_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: [vwnc] Managing Large File Uploads in Seaside

Karsten Kusche
Hi Esteban,

we have an override in SeasideResponder>>asSeasideFileStream: where we do something differently in „stream isExternalStream ifTrue:[…]“

There we ask the stream for #fileName (returns a Filename object) and test the file size. That’s the file that contains the data that was uploaded. If the file’s size exceeds a certain threshold we return a subclass of WAFile that knows the file and not its contents.

Kind Regards
Karsten

Georg Heeg eK
Wallstraße 22
06366 Köthen

Tel.: 03496/214328
FAX: 03496/214712
Amtsgericht Dortmund HRA 12812

Am 8. Mai 2019 um 16:29:29, Esteban Maringolo ([hidden email]) schrieb:

I found that SiouX is effectively uploading the contents to an attachment file directory.

Maybe there is a way to avoid instantiating a WAFile and use a WAExternalFile poiting to the file on disk, or simply using my custom SeasideResponder subclass.

This won't save VW from reading the whole file into memory before saving it to disk (or will it?) but it certainly will only keep the reference during the request/response of SiouX, which will be garbage collected faster than anything on a Seaside Component.

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 10:10 AM Esteban Maringolo <[hidden email]> wrote:
Hi Felix,

That is similar to what Johan proposes in his article, but the goal is to achieve the same thing by using SiouX (VW's latest HTTP server) responders, maybe using a chunked read/write approach (so there is no more than a certain buffer in the object memory).

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 9:46 AM Félix Madrid <[hidden email]> wrote:
Hi Esteban,

Maybe this article (and project) from Nick Ager can help you:


Cheers,

Félix

On Wed, May 8, 2019 at 2:39 PM Esteban Maringolo <[hidden email]> wrote:
I'm trying to use the SiouX responder fortification to limit a large file upload from blocking the whole image and/or filling its memory.

Looking at the options I found there is a "save attachments as files", which seems like a similar approach as the one proposed at <https://jbrichau.github.io/blog/large-file-upload-in-seaside > using NGINX or Apache, but apparently that option only works at NetHttpResponder, but it's not applied in Seaside.

Did anybody integrate this feature or a similar one with Seaside in VW? (otherwise I'd have to do it myself).

Regards!

Esteban A. Maringolo

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: [vwnc] Managing Large File Uploads in Seaside

Esteban A. Maringolo
Hi Karsten,

I didn't think about an override and went subclassing directly (I'm not keen/not used to method overrides), but your approach of having a threshold seems like a good tradeoff to preserve the simplicity for small files.

What I couldn't find is whether the maximum request body size is enforced with Seaside (I placed a few breakpoints around senders of #requestBodyLimit without any luck).

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 1:59 PM Karsten Kusche <[hidden email]> wrote:
Hi Esteban,

we have an override in SeasideResponder>>asSeasideFileStream: where we do something differently in „stream isExternalStream ifTrue:[…]“

There we ask the stream for #fileName (returns a Filename object) and test the file size. That’s the file that contains the data that was uploaded. If the file’s size exceeds a certain threshold we return a subclass of WAFile that knows the file and not its contents.

Kind Regards
Karsten

Georg Heeg eK
Wallstraße 22
06366 Köthen

Tel.: 03496/214328
FAX: 03496/214712
Amtsgericht Dortmund HRA 12812

Am 8. Mai 2019 um 16:29:29, Esteban Maringolo ([hidden email]) schrieb:

I found that SiouX is effectively uploading the contents to an attachment file directory.

Maybe there is a way to avoid instantiating a WAFile and use a WAExternalFile poiting to the file on disk, or simply using my custom SeasideResponder subclass.

This won't save VW from reading the whole file into memory before saving it to disk (or will it?) but it certainly will only keep the reference during the request/response of SiouX, which will be garbage collected faster than anything on a Seaside Component.

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 10:10 AM Esteban Maringolo <[hidden email]> wrote:
Hi Felix,

That is similar to what Johan proposes in his article, but the goal is to achieve the same thing by using SiouX (VW's latest HTTP server) responders, maybe using a chunked read/write approach (so there is no more than a certain buffer in the object memory).

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 9:46 AM Félix Madrid <[hidden email]> wrote:
Hi Esteban,

Maybe this article (and project) from Nick Ager can help you:


Cheers,

Félix

On Wed, May 8, 2019 at 2:39 PM Esteban Maringolo <[hidden email]> wrote:
I'm trying to use the SiouX responder fortification to limit a large file upload from blocking the whole image and/or filling its memory.

Looking at the options I found there is a "save attachments as files", which seems like a similar approach as the one proposed at <https://jbrichau.github.io/blog/large-file-upload-in-seaside > using NGINX or Apache, but apparently that option only works at NetHttpResponder, but it's not applied in Seaside.

Did anybody integrate this feature or a similar one with Seaside in VW? (otherwise I'd have to do it myself).

Regards!

Esteban A. Maringolo

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: [vwnc] Managing Large File Uploads in Seaside

Karsten Kusche
What do you mean by body size limit? Is that a http thing?

As for subclassing, the method in question isn’t written in a way that would allow subclasses to customize something without copying too much code. In that case an override is better, because there’s tool support to find overrides and inspect them upon migration. 

Karsten




Am 8. Mai 2019 um 19:23:15 MESZ schrieb Esteban Maringolo <[hidden email]>:
Hi Karsten,

I didn't think about an override and went subclassing directly (I'm not keen/not used to method overrides), but your approach of having a threshold seems like a good tradeoff to preserve the simplicity for small files.

What I couldn't find is whether the maximum request body size is enforced with Seaside (I placed a few breakpoints around senders of #requestBodyLimit without any luck).

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 1:59 PM Karsten Kusche <[hidden email]> wrote:
Hi Esteban,

we have an override in SeasideResponder>>asSeasideFileStream: where we do something differently in „stream isExternalStream ifTrue:[…]“

There we ask the stream for #fileName (returns a Filename object) and test the file size. That’s the file that contains the data that was uploaded. If the file’s size exceeds a certain threshold we return a subclass of WAFile that knows the file and not its contents.

Kind Regards
Karsten

Georg Heeg eK
Wallstraße 22
06366 Köthen

Tel.: 03496/214328
FAX: 03496/214712
Amtsgericht Dortmund HRA 12812

Am 8. Mai 2019 um 16:29:29, Esteban Maringolo ([hidden email]) schrieb:

I found that SiouX is effectively uploading the contents to an attachment file directory.

Maybe there is a way to avoid instantiating a WAFile and use a WAExternalFile poiting to the file on disk, or simply using my custom SeasideResponder subclass.

This won't save VW from reading the whole file into memory before saving it to disk (or will it?) but it certainly will only keep the reference during the request/response of SiouX, which will be garbage collected faster than anything on a Seaside Component.

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 10:10 AM Esteban Maringolo <[hidden email]> wrote:
Hi Felix,

That is similar to what Johan proposes in his article, but the goal is to achieve the same thing by using SiouX (VW's latest HTTP server) responders, maybe using a chunked read/write approach (so there is no more than a certain buffer in the object memory).

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 9:46 AM Félix Madrid <[hidden email]> wrote:
Hi Esteban,

Maybe this article (and project) from Nick Ager can help you:


Cheers,

Félix

On Wed, May 8, 2019 at 2:39 PM Esteban Maringolo <[hidden email]> wrote:
I'm trying to use the SiouX responder fortification to limit a large file upload from blocking the whole image and/or filling its memory.

Looking at the options I found there is a "save attachments as files", which seems like a similar approach as the one proposed at <https://jbrichau.github.io/blog/large-file-upload-in-seaside > using NGINX or Apache, but apparently that option only works at NetHttpResponder, but it's not applied in Seaside.

Did anybody integrate this feature or a similar one with Seaside in VW? (otherwise I'd have to do it myself).

Regards!

Esteban A. Maringolo

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: [vwnc] Managing Large File Uploads in Seaside

Esteban A. Maringolo
Hi Karsten, 

In the fortification options of the SeasideServer (or any Listener, for that case) there is an option to set maximum header size and maximum body size in the requests, by default it has a limit of 100Kb (pretty large) for HTTP headers and no limit for the HTTP request body. I set a small limit for the body (5MB), but couldn't get it enforced, the request goes thorugh without problems.

I might need to dive deeper in the inner workings and collaboration between the HTTP server and the Seaside classes.

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 2:30 PM Karsten Kusche <[hidden email]> wrote:
What do you mean by body size limit? Is that a http thing?

As for subclassing, the method in question isn’t written in a way that would allow subclasses to customize something without copying too much code. In that case an override is better, because there’s tool support to find overrides and inspect them upon migration. 

Karsten




Am 8. Mai 2019 um 19:23:15 MESZ schrieb Esteban Maringolo <[hidden email]>:
Hi Karsten,

I didn't think about an override and went subclassing directly (I'm not keen/not used to method overrides), but your approach of having a threshold seems like a good tradeoff to preserve the simplicity for small files.

What I couldn't find is whether the maximum request body size is enforced with Seaside (I placed a few breakpoints around senders of #requestBodyLimit without any luck).

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 1:59 PM Karsten Kusche <[hidden email]> wrote:
Hi Esteban,

we have an override in SeasideResponder>>asSeasideFileStream: where we do something differently in „stream isExternalStream ifTrue:[…]“

There we ask the stream for #fileName (returns a Filename object) and test the file size. That’s the file that contains the data that was uploaded. If the file’s size exceeds a certain threshold we return a subclass of WAFile that knows the file and not its contents.

Kind Regards
Karsten

Georg Heeg eK
Wallstraße 22
06366 Köthen

Tel.: 03496/214328
FAX: 03496/214712
Amtsgericht Dortmund HRA 12812

Am 8. Mai 2019 um 16:29:29, Esteban Maringolo ([hidden email]) schrieb:

I found that SiouX is effectively uploading the contents to an attachment file directory.

Maybe there is a way to avoid instantiating a WAFile and use a WAExternalFile poiting to the file on disk, or simply using my custom SeasideResponder subclass.

This won't save VW from reading the whole file into memory before saving it to disk (or will it?) but it certainly will only keep the reference during the request/response of SiouX, which will be garbage collected faster than anything on a Seaside Component.

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 10:10 AM Esteban Maringolo <[hidden email]> wrote:
Hi Felix,

That is similar to what Johan proposes in his article, but the goal is to achieve the same thing by using SiouX (VW's latest HTTP server) responders, maybe using a chunked read/write approach (so there is no more than a certain buffer in the object memory).

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 9:46 AM Félix Madrid <[hidden email]> wrote:
Hi Esteban,

Maybe this article (and project) from Nick Ager can help you:


Cheers,

Félix

On Wed, May 8, 2019 at 2:39 PM Esteban Maringolo <[hidden email]> wrote:
I'm trying to use the SiouX responder fortification to limit a large file upload from blocking the whole image and/or filling its memory.

Looking at the options I found there is a "save attachments as files", which seems like a similar approach as the one proposed at <https://jbrichau.github.io/blog/large-file-upload-in-seaside > using NGINX or Apache, but apparently that option only works at NetHttpResponder, but it's not applied in Seaside.

Did anybody integrate this feature or a similar one with Seaside in VW? (otherwise I'd have to do it myself).

Regards!

Esteban A. Maringolo

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: [vwnc] Managing Large File Uploads in Seaside

Esteban A. Maringolo
In reply to this post by Karsten Kusche
Overrides are a door to abuse and sometimes an indicator or a bad design, IMO, but as with other powerful tool, it depends on each case.

What I don't like the pragma based configuration of this part, the whole senders/implementors and class references of Smalltalk are not of much use there, since it is hard to find where things are defined, configured, etc. 

E.g. SeasideResponder class>>#configureCurrent

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 2:54 PM Tom Robinson <[hidden email]> wrote:
Hi Karsten,

This appears to be a common issue. As such, rather than just using an override, maybe Cincom should be asked for an implementation change that would ALLOW subclasses to customize this without too much code duplication. In my opinion, if an override is needed, in most cases it indicates that more flexibility in the original implementation is needed.

Tom

On 5/8/2019 11:30 AM, Karsten Kusche wrote:
What do you mean by body size limit? Is that a http thing?

As for subclassing, the method in question isn’t written in a way that would allow subclasses to customize something without copying too much code. In that case an override is better, because there’s tool support to find overrides and inspect them upon migration. 

Karsten




Am 8. Mai 2019 um 19:23:15 MESZ schrieb Esteban Maringolo [hidden email]:
Hi Karsten,

I didn't think about an override and went subclassing directly (I'm not keen/not used to method overrides), but your approach of having a threshold seems like a good tradeoff to preserve the simplicity for small files.

What I couldn't find is whether the maximum request body size is enforced with Seaside (I placed a few breakpoints around senders of #requestBodyLimit without any luck).

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 1:59 PM Karsten Kusche <[hidden email]> wrote:
Hi Esteban,

we have an override in SeasideResponder>>asSeasideFileStream: where we do something differently in „stream isExternalStream ifTrue:[…]“

There we ask the stream for #fileName (returns a Filename object) and test the file size. That’s the file that contains the data that was uploaded. If the file’s size exceeds a certain threshold we return a subclass of WAFile that knows the file and not its contents.

Kind Regards
Karsten

Georg Heeg eK

Wallstraße 22

06366 Köthen



Tel.: 03496/214328

FAX: 03496/214712

Amtsgericht Dortmund HRA 12812

Am 8. Mai 2019 um 16:29:29, Esteban Maringolo ([hidden email]) schrieb:

I found that SiouX is effectively uploading the contents to an attachment file directory.

Maybe there is a way to avoid instantiating a WAFile and use a WAExternalFile poiting to the file on disk, or simply using my custom SeasideResponder subclass.

This won't save VW from reading the whole file into memory before saving it to disk (or will it?) but it certainly will only keep the reference during the request/response of SiouX, which will be garbage collected faster than anything on a Seaside Component.

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 10:10 AM Esteban Maringolo <[hidden email]> wrote:
Hi Felix,

That is similar to what Johan proposes in his article, but the goal is to achieve the same thing by using SiouX (VW's latest HTTP server) responders, maybe using a chunked read/write approach (so there is no more than a certain buffer in the object memory).

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 9:46 AM Félix Madrid <[hidden email]> wrote:
Hi Esteban,

Maybe this article (and project) from Nick Ager can help you:


Cheers,

Félix

On Wed, May 8, 2019 at 2:39 PM Esteban Maringolo <[hidden email]> wrote:
I'm trying to use the SiouX responder fortification to limit a large file upload from blocking the whole image and/or filling its memory.

Looking at the options I found there is a "save attachments as files", which seems like a similar approach as the one proposed at <https://jbrichau.github.io/blog/large-file-upload-in-seaside > using NGINX or Apache, but apparently that option only works at NetHttpResponder, but it's not applied in Seaside.

Did anybody integrate this feature or a similar one with Seaside in VW? (otherwise I'd have to do it myself).

Regards!

Esteban A. Maringolo

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside


_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: [vwnc] Managing Large File Uploads in Seaside

Jerry Kott-3
Estaban,

you will probably want to have a look at FileResponder if you haven’t already. IIRC, it has the functionality to upload large files without loading them all into memory. As for pragmas - I know many find them hard to follow. Assuming you have a Seaside server configured by default, to include a straight FileResponder in your Seaside application, just add a method with a pragma something like:

FileResponder class >> configureForSeaside: aResponder
<server: 'Seaside' path: ‘/upload’>
aResponder
allowGET; “if you want to be able to download the files as well"
allowPUT; “essential to allow uploading”
rootDirectory: <set the directory where you want the uploaded files to go to>

You can of course subclass FileResponder to  implement your own specialized behaviour.

Hope this helps.

Jerry Kott
This message has been digitally signed. 
PGP Fingerprint:
A9181736DD2F1B6CC7CF9E51AC8514F48C0979A5



On 08-05-2019, at 11:09 AM, Esteban Maringolo <[hidden email]> wrote:

Overrides are a door to abuse and sometimes an indicator or a bad design, IMO, but as with other powerful tool, it depends on each case.

What I don't like the pragma based configuration of this part, the whole senders/implementors and class references of Smalltalk are not of much use there, since it is hard to find where things are defined, configured, etc. 

E.g. SeasideResponder class>>#configureCurrent

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 2:54 PM Tom Robinson <[hidden email]> wrote:
Hi Karsten,

This appears to be a common issue. As such, rather than just using an override, maybe Cincom should be asked for an implementation change that would ALLOW subclasses to customize this without too much code duplication. In my opinion, if an override is needed, in most cases it indicates that more flexibility in the original implementation is needed.

Tom

On 5/8/2019 11:30 AM, Karsten Kusche wrote:
What do you mean by body size limit? Is that a http thing?

As for subclassing, the method in question isn’t written in a way that would allow subclasses to customize something without copying too much code. In that case an override is better, because there’s tool support to find overrides and inspect them upon migration. 

Karsten




Am 8. Mai 2019 um 19:23:15 MESZ schrieb Esteban Maringolo [hidden email]:
Hi Karsten,

I didn't think about an override and went subclassing directly (I'm not keen/not used to method overrides), but your approach of having a threshold seems like a good tradeoff to preserve the simplicity for small files.

What I couldn't find is whether the maximum request body size is enforced with Seaside (I placed a few breakpoints around senders of #requestBodyLimit without any luck).

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 1:59 PM Karsten Kusche <[hidden email]> wrote:
Hi Esteban,

we have an override in SeasideResponder>>asSeasideFileStream: where we do something differently in „stream isExternalStream ifTrue:[…]“

There we ask the stream for #fileName (returns a Filename object) and test the file size. That’s the file that contains the data that was uploaded. If the file’s size exceeds a certain threshold we return a subclass of WAFile that knows the file and not its contents.

Kind Regards
Karsten

Georg Heeg eK

Wallstraße 22

06366 Köthen



Tel.: 03496/214328

FAX: 03496/214712

Amtsgericht Dortmund HRA 12812

Am 8. Mai 2019 um 16:29:29, Esteban Maringolo ([hidden email]) schrieb:

I found that SiouX is effectively uploading the contents to an attachment file directory.

Maybe there is a way to avoid instantiating a WAFile and use a WAExternalFile poiting to the file on disk, or simply using my custom SeasideResponder subclass.

This won't save VW from reading the whole file into memory before saving it to disk (or will it?) but it certainly will only keep the reference during the request/response of SiouX, which will be garbage collected faster than anything on a Seaside Component.

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 10:10 AM Esteban Maringolo <[hidden email]> wrote:
Hi Felix,

That is similar to what Johan proposes in his article, but the goal is to achieve the same thing by using SiouX (VW's latest HTTP server) responders, maybe using a chunked read/write approach (so there is no more than a certain buffer in the object memory).

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 9:46 AM Félix Madrid <[hidden email]> wrote:
Hi Esteban,

Maybe this article (and project) from Nick Ager can help you:


Cheers,

Félix

On Wed, May 8, 2019 at 2:39 PM Esteban Maringolo <[hidden email]> wrote:
I'm trying to use the SiouX responder fortification to limit a large file upload from blocking the whole image and/or filling its memory.

Looking at the options I found there is a "save attachments as files", which seems like a similar approach as the one proposed at <https://jbrichau.github.io/blog/large-file-upload-in-seaside > using NGINX or Apache, but apparently that option only works at NetHttpResponder, but it's not applied in Seaside.

Did anybody integrate this feature or a similar one with Seaside in VW? (otherwise I'd have to do it myself).

Regards!

Esteban A. Maringolo

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside


_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: [vwnc] Managing Large File Uploads in Seaside

Esteban A. Maringolo
Hi Jerry,

I'm using FileResponder for static content, although not for uploads.

However using it for uploads would require implementing more than just supporting the upload, because if the resource POSTed/PUT exists it should create another one, and there is no integration with Seaside. I might need to do this in the future, but now I have it working, I only need to find why the SeasideResponder is not enforcing the request body limit.

Regards,

Esteban A. Maringolo


On Thu, May 9, 2019 at 1:36 AM Jerry Kott <[hidden email]> wrote:
Estaban,

you will probably want to have a look at FileResponder if you haven’t already. IIRC, it has the functionality to upload large files without loading them all into memory. As for pragmas - I know many find them hard to follow. Assuming you have a Seaside server configured by default, to include a straight FileResponder in your Seaside application, just add a method with a pragma something like:

FileResponder class >> configureForSeaside: aResponder
<server: 'Seaside' path: ‘/upload’>
aResponder
allowGET; “if you want to be able to download the files as well"
allowPUT; “essential to allow uploading”
rootDirectory: <set the directory where you want the uploaded files to go to>

You can of course subclass FileResponder to  implement your own specialized behaviour.

Hope this helps.

Jerry Kott
This message has been digitally signed. 
PGP Fingerprint:
A9181736DD2F1B6CC7CF9E51AC8514F48C0979A5



On 08-05-2019, at 11:09 AM, Esteban Maringolo <[hidden email]> wrote:

Overrides are a door to abuse and sometimes an indicator or a bad design, IMO, but as with other powerful tool, it depends on each case.

What I don't like the pragma based configuration of this part, the whole senders/implementors and class references of Smalltalk are not of much use there, since it is hard to find where things are defined, configured, etc. 

E.g. SeasideResponder class>>#configureCurrent

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 2:54 PM Tom Robinson <[hidden email]> wrote:
Hi Karsten,

This appears to be a common issue. As such, rather than just using an override, maybe Cincom should be asked for an implementation change that would ALLOW subclasses to customize this without too much code duplication. In my opinion, if an override is needed, in most cases it indicates that more flexibility in the original implementation is needed.

Tom

On 5/8/2019 11:30 AM, Karsten Kusche wrote:
What do you mean by body size limit? Is that a http thing?

As for subclassing, the method in question isn’t written in a way that would allow subclasses to customize something without copying too much code. In that case an override is better, because there’s tool support to find overrides and inspect them upon migration. 

Karsten




Am 8. Mai 2019 um 19:23:15 MESZ schrieb Esteban Maringolo [hidden email]:
Hi Karsten,

I didn't think about an override and went subclassing directly (I'm not keen/not used to method overrides), but your approach of having a threshold seems like a good tradeoff to preserve the simplicity for small files.

What I couldn't find is whether the maximum request body size is enforced with Seaside (I placed a few breakpoints around senders of #requestBodyLimit without any luck).

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 1:59 PM Karsten Kusche <[hidden email]> wrote:
Hi Esteban,

we have an override in SeasideResponder>>asSeasideFileStream: where we do something differently in „stream isExternalStream ifTrue:[…]“

There we ask the stream for #fileName (returns a Filename object) and test the file size. That’s the file that contains the data that was uploaded. If the file’s size exceeds a certain threshold we return a subclass of WAFile that knows the file and not its contents.

Kind Regards
Karsten

Georg Heeg eK

Wallstraße 22

06366 Köthen



Tel.: 03496/214328

FAX: 03496/214712

Amtsgericht Dortmund HRA 12812

Am 8. Mai 2019 um 16:29:29, Esteban Maringolo ([hidden email]) schrieb:

I found that SiouX is effectively uploading the contents to an attachment file directory.

Maybe there is a way to avoid instantiating a WAFile and use a WAExternalFile poiting to the file on disk, or simply using my custom SeasideResponder subclass.

This won't save VW from reading the whole file into memory before saving it to disk (or will it?) but it certainly will only keep the reference during the request/response of SiouX, which will be garbage collected faster than anything on a Seaside Component.

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 10:10 AM Esteban Maringolo <[hidden email]> wrote:
Hi Felix,

That is similar to what Johan proposes in his article, but the goal is to achieve the same thing by using SiouX (VW's latest HTTP server) responders, maybe using a chunked read/write approach (so there is no more than a certain buffer in the object memory).

Regards,

Esteban A. Maringolo


On Wed, May 8, 2019 at 9:46 AM Félix Madrid <[hidden email]> wrote:
Hi Esteban,

Maybe this article (and project) from Nick Ager can help you:


Cheers,

Félix

On Wed, May 8, 2019 at 2:39 PM Esteban Maringolo <[hidden email]> wrote:
I'm trying to use the SiouX responder fortification to limit a large file upload from blocking the whole image and/or filling its memory.

Looking at the options I found there is a "save attachments as files", which seems like a similar approach as the one proposed at <https://jbrichau.github.io/blog/large-file-upload-in-seaside > using NGINX or Apache, but apparently that option only works at NetHttpResponder, but it's not applied in Seaside.

Did anybody integrate this feature or a similar one with Seaside in VW? (otherwise I'd have to do it myself).

Regards!

Esteban A. Maringolo

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside


_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside