Smalltalk › Gemtalk › GLASS

How do you handle image/document uploads?

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

12 messages Options

Stephan Eggermont-3

How do you handle image/document uploads?

Hello,

We're getting to the point where we want to be able to upload documents to the server.
It is clear that I do not want to store the documents in Gemstone. The documents are
static and too large to handle comfortably. Uploading them I could do through Gemstone,
though that might mean I have to increase the number of gems running, as they are blocked during
the upload. Does anyone use an alternative way to upload documents? I'd been thinking about
using the lighttpd that's in front of it.

Stephan Eggermont

Nick

Re: How do you handle image/document uploads?

Hi Stephan,

We're using Nginx Upload module
(http://www.grid.net.ru/nginx/upload.en.html) and to create a dynamic
upload gauge using Nginx Upload progress module
(http://wiki.nginx.org/HttpUploadProgressModule)

The upload module, ensures that the front end server (Nginx) handles
the upload and saves the file within a specified directory. It then
adds parameters to the request which it passes onto a Gem. The added
parameters specify the filepath of the upload file and size. I then
use an OSProcess call to move and rename the file.

I keep meaning to write a blog on the details. If you think it would
be useful that might spur me into action - knowing I'd have an assured
audience of at least one.

Nick

On 3 March 2011 09:50, Stephan Eggermont <[hidden email]> wrote:

> Hello,
>
> We're getting to the point where we want to be able to upload documents to the server.
> It is clear that I do not want to store the documents in Gemstone. The documents are
> static and too large to handle comfortably. Uploading them I could do through Gemstone,
> though that might mean I have to increase the number of gems running, as they are blocked during
> the upload. Does anyone use an alternative way to upload documents? I'd been thinking about
> using the lighttpd that's in front of it.
>
> Stephan Eggermont

Ivan, Liliana

Re: How do you handle image/document uploads?

Hi Nick
Please do a write-up - I will not use it just yet; but soon :)
Thanks

Liliana
-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Nick Ager
Sent: 03 March 2011 14:46
To: GemStone Seaside beta discussion
Subject: Re: [GS/SS Beta] How do you handle image/document uploads?

Hi Stephan,

We're using Nginx Upload module
(http://www.grid.net.ru/nginx/upload.en.html) and to create a dynamic
upload gauge using Nginx Upload progress module
(http://wiki.nginx.org/HttpUploadProgressModule)

This e-mail is subject to a disclaimer, available at http://www.rmb.co.za/web/elements.nsf/online/disclaimer-communications.html

Stephan Eggermont-3

Re: How do you handle image/document uploads?

In reply to this post by Nick

Hello Nick,

Yes, I would be interested. Why did you choose nginx?
If we use it, we'll have a puppet script for it...
I'd have to replace the lighttpd.

Stephan

On 3 mrt 2011, at 13:46, Nick Ager wrote:

> Hi Stephan,
>
> We're using Nginx Upload module
> (http://www.grid.net.ru/nginx/upload.en.html) and to create a dynamic
> upload gauge using Nginx Upload progress module
> (http://wiki.nginx.org/HttpUploadProgressModule)
>
> The upload module, ensures that the front end server (Nginx) handles
> the upload and saves the file within a specified directory. It then
> adds parameters to the request which it passes onto a Gem. The added
> parameters specify the filepath of the upload file and size. I then
> use an OSProcess call to move and rename the file.
>
> I keep meaning to write a blog on the details. If you think it would
> be useful that might spur me into action - knowing I'd have an assured
> audience of at least one.
>
> Nick
>
> On 3 March 2011 09:50, Stephan Eggermont <[hidden email]> wrote:
>> Hello,
>>
>> We're getting to the point where we want to be able to upload documents to the server.
>> It is clear that I do not want to store the documents in Gemstone. The documents are
>> static and too large to handle comfortably. Uploading them I could do through Gemstone,
>> though that might mean I have to increase the number of gems running, as they are blocked during
>> the upload. Does anyone use an alternative way to upload documents? I'd been thinking about
>> using the lighttpd that's in front of it.
>>
>> Stephan Eggermont

Robert Sirois

Re: How do you handle image/document uploads?

Make that an audience of 3+ ;)

> From: [hidden email]
> Date: Thu, 3 Mar 2011 14:34:38 +0100
> To: [hidden email]
> Subject: Re: [GS/SS Beta] How do you handle image/document uploads?
>
> Hello Nick,
>
> Yes, I would be interested. Why did you choose nginx?
> If we use it, we'll have a puppet script for it...
> I'd have to replace the lighttpd.
>
> Stephan
>
> On 3 mrt 2011, at 13:46, Nick Ager wrote:
>
> > Hi Stephan,
> >
> > We're using Nginx Upload module
> > (http://www.grid.net.ru/nginx/upload.en.html) and to create a dynamic
> > upload gauge using Nginx Upload progress module
> > (http://wiki.nginx.org/HttpUploadProgressModule)
> >
> > The upload module, ensures that the front end server (Nginx) handles
> > the upload and saves the file within a specified directory. It then
> > adds parameters to the request which it passes onto a Gem. The added
> > parameters specify the filepath of the upload file and size. I then
> > use an OSProcess call to move and rename the file.
> >
> > I keep meaning to write a blog on the details. If you think it would
> > be useful that might spur me into action - knowing I'd have an assured
> > audience of at least one.
> >
> > Nick
> >
> > On 3 March 2011 09:50, Stephan Eggermont <[hidden email]> wrote:
> >> Hello,
> >>
> >> We're getting to the point where we want to be able to upload documents to the server.
> >> It is clear that I do not want to store the documents in Gemstone. The documents are
> >> static and too large to handle comfortably. Uploading them I could do through Gemstone,
> >> though that might mean I have to increase the number of gems running, as they are blocked during
> >> the upload. Does anyone use an alternative way to upload documents? I'd been thinking about
> >> using the lighttpd that's in front of it.
> >>
> >> Stephan Eggermont
>

NorbertHartl

Re: How do you handle image/document uploads?

In reply to this post by Nick

Now I'm interested, too :) I would like to know if you realize the _pass_ url using callbacks and how the phrase "passes onto a Gem...use an OSProcess call to move..." fits together.

Norbert

On 03.03.2011, at 13:46, Nick Ager wrote:

Nick

Re: How do you handle image/document uploads?

In reply to this post by Stephan Eggermont-3

Hi

> Why did you choose nginx?

Primarily as I could use it to fake https client support within
Gemstone without having to run stunnel, see:
http://www.monkeysnatchbanana.com/posts/2010/06/22/faking-a-https-client-for-glass.html

Secondly as it appeared to gaining popularity:
http://news.netcraft.com/archives/category/web-server-survey/

That said I started with lighttpd and prefer it's configuration syntax

Nick

Nick

Re: How do you handle image/document uploads?

In reply to this post by NorbertHartl

Hi,

> Now I'm interested, too :) I would like to know if you realize the _pass_ url using callbacks and how the phrase "passes onto a Gem...use an OSProcess call to move..." fits together.

I'll try to write something up this weekend, but in case I don't meet
my schedule the technique I use is to read the post parameters within
a callback on a hiddenInput (as the #fileUpload callback isn't called
as there is no file being passed). Here's a code snippet from my
upload class:

"The hiddenInput callback will contain the file upload fields if
we've loading via nginx
file upload module"
html hiddenInput
value: 'hidden';
callback: [:val | | request postFields fileName uploadFieldName |
request := self requestContext request.
postFields := request postFields.
uploadFieldName := fileUploadField attributeAt: 'name'.
fileName := (postFields at: (uploadFieldName, '.name') ifAbsent: [nil]).

"has nginx file upload module inserted it's post fields in the request?"
fileName notNil ifTrue: [
fileMoveCallback value: postFields value: uploadFieldName ] ] ]

I'll do a more complete write-up soon.

Johan Brichau-2

Re: How do you handle image/document uploads?

In reply to this post by Nick

On 03 Mar 2011, at 15:20, Nick Ager wrote:

>
>> Why did you choose nginx?
>
> That said I started with lighttpd and prefer it's configuration syntax
>

A drawback of lighttpd is that I never managed to get http compression working for fastcgi... did anyone?

Nginx appears to allow this, so I'm thinking of switching as well.

Miguel Cobá

Re: How do you handle image/document uploads?

El jue, 03-03-2011 a las 15:59 +0100, Johan Brichau escribió:
> On 03 Mar 2011, at 15:20, Nick Ager wrote:
>
> >
> >> Why did you choose nginx?
> >
> > That said I started with lighttpd and prefer it's configuration syntax
> >
>
> A drawback of lighttpd is that I never managed to get http compression working for fastcgi... did anyone?

You can't also test if a file exists on disk and if it exists serve it,
otherwise proxy to backend servers. I tried to do it some years ago with
Lua (integrated scripting language of lighttpd) but failed miserably. I
ended using Apache.

>
> Nginx appears to allow this, so I'm thinking of switching as well.
>

Cheers

--
Miguel Cobá
http://twitter.com/MiguelCobaMtz
http://miguel.leugim.com.mx

Paul DeBruicker

Re: How do you handle image/document uploads?

In reply to this post by Johan Brichau-2

On 03/03/2011 09:59 AM, Johan Brichau wrote:

>
> On 03 Mar 2011, at 15:20, Nick Ager wrote:
>
>>
>>> Why did you choose nginx?
>>
>> That said I started with lighttpd and prefer it's configuration syntax
>>
>
> A drawback of lighttpd is that I never managed to get http compression working for fastcgi... did anyone?
>
> Nginx appears to allow this, so I'm thinking of switching as well.
>

Hi Johan,

There's a patch to allow you to use mod_deflate in Lighttpd 1.4.XXX .
Instructions are here:

http://www.howtoforge.com/lighttpd_1.4_mod_deflate_debian_etch

You have to compile your own version of lighttpd.

The patch is at the bottom of this page:
http://redmine.lighttpd.net/wiki/lighttpd/Mod_Deflate

Looks like it only works with a few versions of lighttpd. 1.4.26 being
the latest.

mod_deflate will compress the dynamic http content from Seaside, whether
its from fastcgi or just Swazoo or whatever.

Is that what you had in mind?

Paul

Nick

Re: How do you handle image/document uploads?

In reply to this post by Miguel Cobá

> You can't also test if a file exists on disk and if it exists serve it,
> otherwise proxy to backend servers. I tried to do it some years ago with
> Lua (integrated scripting language of lighttpd) but failed miserably. I
> ended using Apache.

When I was using lighttpd my configuration was something like:

$HTTP["url"] !~ "^/files/" {
# fastcgi.debug=1
fastcgi.server = ("/" =>(
("host" => "127.0.0.1", "port" => 9001, "check-local" => "disable"),
("host" => "127.0.0.1", "port" => 9002, "check-local" => "disable"),
("host" => "127.0.0.1", "port" => 9003, "check-local" => "disable")
)
)
}

which would pass the request onto Gemstone unless the request began
with '^/file' - seemed to work OK.
In Nginx you can use the try_files directive; only if the file doesn't
exist will the request be passed to Gemstone:

location / {
try_files $uri @seaside;
}

More here: http://www.nickager.com/blog/Installing-Gemstone-on-an-Amazon-EC2-Linux-instance/