Hello all,
Okay, this question belongs to the Apache group, but maybe you know the answer since this is about a GLASS application ;-) We need to have users upload rather big files but we are experiencing crashes without any info in the logs. The code is the simple web upload. ... html form multipart; with: [ html fileUpload callback: [ :value ... That code works fine for files in the 200KB range. However we need now to upload files bigger than 7MB. We are sitting behind Apache 2 and a FastCGI configuration (similar to the one described by James Foster at http://programminggems.wordpress.com/2010/01/12/slicehost-2/). In other words, a Proxy configuration passes the requests to some gems. So, I am suspecting something like the LimitRequestBody parameter. But no luck so far. Any suggestions? Thanks, Thierry Thelliez |
Hi Thierry,
Okay, this question belongs to the Apache group, but maybe you know For large files it makes sense to let the font-end web server stream the file directly to disk, rather than loading the file as a ByteArray into the image and then saving it as a file. I've written how I do that in using Nginx and Seaside (http://nickager.com/blog/file-upload-using-Nginx-and-Seaside), however you are using Apache. It seems Apache has a similar module: http://commons.apache.org/fileupload/
HTH Nick |
In reply to this post by Thelliez
Thierry,
Are you getting a 'gem out of temp object space' error ? If yes, this is because the gem is loading the entire file in memory before it will write it onto disk. Each gem is started with a configuration parameter that limits its temp object space size. In standard GLASS, this size is 50 Mb. So, depending on how much other things your gem has loaded in memory during that request, you will probably run out of temp object space. In my opinion, there are two solutions: - You can bump up the temp object space parameter before starting the seaside gems (look for the GEM_TEMPOBJ_CACHE_SIZE param in the configuration files). This will raise the limit of the size of files you can upload. However, if you upload a file that is large enough, it will crash anyway. - Use a file upload module in your front-end web server that handles the file uploads. I do not know how this can be done in Apache, but we (and others) are happily using the nginx file upload module to handle all file uploads. See http://www.nickager.com/blog/file-upload-using-Nginx-and-Seaside/ for more info. Hope this helps Johan On 17 Aug 2011, at 07:23, Thierry Thelliez wrote: > Hello all, > > Okay, this question belongs to the Apache group, but maybe you know > the answer since this is about a GLASS application ;-) > > We need to have users upload rather big files but we are experiencing > crashes without any info in the logs. The code is the simple web > upload. > > ... > html form > multipart; > with: [ html fileUpload > callback: [ :value > ... > > That code works fine for files in the 200KB range. However we need > now to upload files bigger than 7MB. > > We are sitting behind Apache 2 and a FastCGI configuration (similar to > the one described by James Foster at > http://programminggems.wordpress.com/2010/01/12/slicehost-2/). In > other words, a Proxy configuration passes the requests to some gems. > > So, I am suspecting something like the LimitRequestBody parameter. > But no luck so far. > > Any suggestions? > > Thanks, > Thierry Thelliez |
In reply to this post by Nick
Opps,
I think I was confusing Apache httpd (Web Server) with other a unrelated apache project for file in Java. I found a like of apache server modules (http://httpd.apache.org/docs/2.0/mod/), but a brief search didn't locate anything appropriate, but I'd be surprised if there isn't a similar solution available for Apache.
Nick |
In reply to this post by Johan Brichau-2
On 08/17/2011 09:15 AM, Johan Brichau wrote:
> Thierry, > > Are you getting a 'gem out of temp object space' error ? > > If yes, this is because the gem is loading the entire file in memory before it will write it onto disk. A few years back I wrote a patch for KomHttpServer/SocketStream that read/wrote directly chunk wise to disk and that easily handled my 800Mb test file with same speed all through. Perhaps something to fix in GLASS? regards, Göran |
On 08/17/2011 02:03 PM, Göran Krampe wrote:
> On 08/17/2011 09:15 AM, Johan Brichau wrote: >> Thierry, >> >> Are you getting a 'gem out of temp object space' error ? >> >> If yes, this is because the gem is loading the entire file in memory >> before it will write it onto disk. > > A few years back I wrote a patch for KomHttpServer/SocketStream that > read/wrote directly chunk wise to disk and that easily handled my 800Mb > test file with same speed all through. Perhaps something to fix in GLASS? Here: http://forum.world.st/Final-try-for-Kom-Seaside-file-upload-tuning-td98355.html ...if anyone is interested :) regards, Göran |
Thanks all for your answers. I am going to look into nginx.
(Although nginx is not in this list http://www.entrust.net/ssl-technical/webserver.cfm and that could be a problem here...) Cheers, Thierry |
Just to follow up with what I ended up doing and again thanks for the
suggestions. There are several options: - implement Nick's Nginx trick, - play we a loosely supported mod_upload with Apache, - implement a deep modification in GLASS like Goran suggested, - increase the memory configuration I explored all these options, but since my files are very predictable in size (7 to 8MB), I opted to just increase the memory for now. In fact these files (containing metadata in Excel) were .xls files. Saving them to .xlsx reduced the size to less than 2MB. Increasing GEM_TEMPOBJ_CACHE_SIZE worked. Found in the topaz call in /opt/gemstone/product/seaside/bin/startSeaside30_Adaptor: From cat << EOF | nohup $GEMSTONE/bin/topaz -l -T50000 2>&1 >> $GEMSTONE_LOGDIR/${1}_server-${2}.log & To cat << EOF | nohup $GEMSTONE/bin/topaz -l -T500000 2>&1 >> $GEMSTONE_LOGDIR/${1}_server-${2}.log & That's probably too much, so I have a question: What are the side effects of allocating too much Temp object space? BTW, I had missed the following message in one of the logs... GemStone: Error Fatal VM temporary object memory is full , old space overflow Error Category: 231169 [GemStone] Number: 4067 Arg Count: 1 Context : 20 Arg 1: 20 Cheers, Thierry |
Am 18.08.2011 um 21:07 schrieb Thierry Thelliez: > Just to follow up with what I ended up doing and again thanks for the > suggestions. > > There are several options: > - implement Nick's Nginx trick, > - play we a loosely supported mod_upload with Apache, > - implement a deep modification in GLASS like Goran suggested, > - increase the memory configuration > > I explored all these options, but since my files are very predictable > in size (7 to 8MB), I opted to just increase the memory for now. In > fact these files (containing metadata in Excel) were .xls files. > Saving them to .xlsx reduced the size to less than 2MB. > > Increasing GEM_TEMPOBJ_CACHE_SIZE worked. Found in the topaz call in > /opt/gemstone/product/seaside/bin/startSeaside30_Adaptor: > > From > cat << EOF | nohup $GEMSTONE/bin/topaz -l -T50000 2>&1 >> > $GEMSTONE_LOGDIR/${1}_server-${2}.log & > To > cat << EOF | nohup $GEMSTONE/bin/topaz -l -T500000 2>&1 >> > $GEMSTONE_LOGDIR/${1}_server-${2}.log & > > > That's probably too much, so I have a question: What are the side > effects of allocating too much Temp object space? > > Norbert > > BTW, I had missed the following message in one of the logs... > > GemStone: Error Fatal > VM temporary object memory is full > , old space overflow > Error Category: 231169 [GemStone] Number: 4067 Arg Count: 1 Context : 20 > Arg 1: 20 > > Cheers, > Thierry |
In reply to this post by Thelliez
Hi guys,
Just to know that Swazoo have upload (and download) streaming support, which allows arbitrary large file upload. But it must be supported by web framework too. In Aida we are able to upload few GB DVD movies without problem. And also serve them by output streaming. More about how to add streaming support is here: http://www.swazoo.org/streaming.html Best regards Janko S, Thierry Thelliez piše: > Hello all, > > Okay, this question belongs to the Apache group, but maybe you know > the answer since this is about a GLASS application ;-) > > We need to have users upload rather big files but we are experiencing > crashes without any info in the logs. The code is the simple web > upload. > > ... > html form > multipart; > with: [ html fileUpload > callback: [ :value > ... > > That code works fine for files in the 200KB range. However we need > now to upload files bigger than 7MB. > > We are sitting behind Apache 2 and a FastCGI configuration (similar to > the one described by James Foster at > http://programminggems.wordpress.com/2010/01/12/slicehost-2/). In > other words, a Proxy configuration passes the requests to some gems. > > So, I am suspecting something like the LimitRequestBody parameter. > But no luck so far. > > Any suggestions? > > Thanks, > Thierry Thelliez > -- Janko Mivšek Aida/Web Smalltalk Web Application Server http://www.aidaweb.si |
In reply to this post by Thelliez
Thierry,
GemStone will dynamically increase/decrease the virtual memory requirements for the topaz process ... If you require a large memory footprint to handle the transfer of the file, the temp obj space footprint will dynamically grow, then after you are done with the large buffer, the memory footprint will be reduced. This means that you don't pay a penalty for allocating more temp obj space than you need ... you will be "charged" in terms of swap space, but the virtual memory footprint of the process will only be as big as it needs to be ... If you are committing large chunks of objects, then Norbert's warning definitely applies, since all dirty objects need to be moved to the SPC before the commit can succeed and if there isn't enough space you are toast:) Dale ----- Original Message ----- | From: "Thierry Thelliez" <[hidden email]> | To: "GemStone Seaside beta discussion" <[hidden email]> | Sent: Thursday, August 18, 2011 12:07:46 PM | Subject: Re: [GS/SS Beta] Large file upload | | Just to follow up with what I ended up doing and again thanks for the | suggestions. | | There are several options: | - implement Nick's Nginx trick, | - play we a loosely supported mod_upload with Apache, | - implement a deep modification in GLASS like Goran suggested, | - increase the memory configuration | | I explored all these options, but since my files are very predictable | in size (7 to 8MB), I opted to just increase the memory for now. In | fact these files (containing metadata in Excel) were .xls files. | Saving them to .xlsx reduced the size to less than 2MB. | | Increasing GEM_TEMPOBJ_CACHE_SIZE worked. Found in the topaz call in | /opt/gemstone/product/seaside/bin/startSeaside30_Adaptor: | | From | cat << EOF | nohup $GEMSTONE/bin/topaz -l -T50000 2>&1 >> | $GEMSTONE_LOGDIR/${1}_server-${2}.log & | To | cat << EOF | nohup $GEMSTONE/bin/topaz -l -T500000 2>&1 >> | $GEMSTONE_LOGDIR/${1}_server-${2}.log & | | | That's probably too much, so I have a question: What are the side | effects of allocating too much Temp object space? | | | | BTW, I had missed the following message in one of the logs... | | GemStone: Error Fatal | VM temporary object memory is full | , old space overflow | Error Category: 231169 [GemStone] Number: 4067 Arg Count: 1 Context : | 20 | Arg 1: 20 | | Cheers, | Thierry | |
Free forum by Nabble | Edit this page |