Using nginx file upload module

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Using nginx file upload module

Johan Brichau-2
Hi all,

I'm calling for people who have got the nginx file upload module to work in their seaside apps...

We are trying to use the Nginx file upload module but are stuck when the request needs to get passed on to seaside.
We are using a configuration with a separate '/upload' location that gets handled by the file upload module. The request then needs to get passed to the upstream backend (seaside) but that request still has the URI '/upload' and we do not seem to get it rewritten to the '/seasideapp' URI.


Is there anyone who has this working completely and could give us some advice on how solve this?

The relevant parts of our nginx conf look like this:
We also tried with a named location, but according to the docs, that makes the request arguments be dropped (and we need the _s & _k args for seaside...)

Johan

----

        location / {
                         include fastcgi_params;
                         fastcgi_pass seaside;
                }
       
        # Upload form should be submitted to this location
        location /upload {
                                       
                # Pass altered request body to this location
                upload_pass /backend;
                upload_pass_args on;
               
               
                #Pass all fields of the form
                upload_pass_form_field ".*";

                # Store files to this directory
                upload_store /var/www/documents/;
       
            # Set specified fields in request body
            upload_set_form_field $upload_field_name.name "$upload_file_name";
            upload_set_form_field $upload_field_name.content_type "$upload_content_type";
            upload_set_form_field $upload_field_name.path "$upload_tmp_path";
       
            upload_cleanup 400 404 499 500-505;
                        }
               
        location /backend
        {
                rewrite ^/upload/(.*) /seasideapp/$1 last;
        }_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: Using nginx file upload module

Nick
Hi Johan,

Sorry for the delay. I have the Nginx file upload module working. In case it's still useful here's the relevant portions of my nginx configuration:

    upstream seaside {
       server 127.0.0.1:9001;
       server 127.0.0.1:9002;
       server 127.0.0.1:9003;
       fair;
    }

    server {
       listen 80 default;
       server_name www.getitmade.com
       # server_name _;
       root /var/www;

       location / {
            try_files $uri @seaside;
        }

       # ensure that files are searched for locally and not passed to the
       # Gems if the file isn't found on the disk
       location ~* \.(gif|jpg|jpeg|png|css|html|htm|js|zip)$ {

       }

       location ~ fileupload {
           # Pass altered request body to this location
           upload_pass   @seaside;

           # if there's no upload file in this request, nginx generates a 405 - we use this
           # to pass the request onto the 'normal' seaside processing
           # The way this is achieved in the other location directives is through a "try_files" method
           # however using "try_files" results in the other directives never being processed
           error_page 405 415 = @seaside;

           # Store files to this directory
           # The directory is hashed, subdirectories 0 1 2 3 4 5 6 7 8 9 should exist
           upload_store /var/nginx/temp 1;

           # Set the file attributes for uploaded files
           upload_store_access user:rw group:rw all:rw;

           # Set specified fields in request body
           upload_set_form_field $upload_field_name.name "$upload_file_name";
           upload_set_form_field $upload_field_name.content_type "$upload_content_type";
           upload_set_form_field $upload_field_name.path "$upload_tmp_path";

           # Inform backend about hash and size of a file
           # upload_aggregate_form_field "$upload_field_name.md5" "$upload_file_md5";
           upload_aggregate_form_field "$upload_field_name.size" "$upload_file_size";

           # seaside automatically assigns sequential integers to fields with callbacks
           # we want to pass those fields to the backend 
           upload_pass_form_field "^\d+$";

           # we don't want files hanging around if the server failed to process them.
           upload_cleanup 500-505;

           # file upload progress tracking - 30s is the timeout (progress tracking is
           # available 30s after the upload has finished)
           # this must be the last directive in the location block.
           track_uploads proxied 30s;
       }

       # used to report upload progress - defined by the Nginx Upload Progress Module
       location  /progress {
          report_uploads proxied;
       }

       location @seaside {
          include /etc/nginx/fastcgi_params;
          fastcgi_intercept_errors on;
          fastcgi_pass seaside;
       }

      location /nginx_status {
         stub_status on;
         access_log   off;
         allow 127.0.0.1;
         deny all;
      }

       error_page 404 /errors/404.html;
       error_page 403 /errors/403.html;
       error_page 500 502 503 504 /errors/50x.html;
    }




On 12 June 2011 10:28, Johan Brichau <[hidden email]> wrote:
Hi all,

I'm calling for people who have got the nginx file upload module to work in their seaside apps...

We are trying to use the Nginx file upload module but are stuck when the request needs to get passed on to seaside.
We are using a configuration with a separate '/upload' location that gets handled by the file upload module. The request then needs to get passed to the upstream backend (seaside) but that request still has the URI '/upload' and we do not seem to get it rewritten to the '/seasideapp' URI.


Is there anyone who has this working completely and could give us some advice on how solve this?

The relevant parts of our nginx conf look like this:
We also tried with a named location, but according to the docs, that makes the request arguments be dropped (and we need the _s & _k args for seaside...)

Johan

----

       location / {
                        include fastcgi_params;
                        fastcgi_pass seaside;
               }

       # Upload form should be submitted to this location
       location /upload {

               # Pass altered request body to this location
               upload_pass /backend;
               upload_pass_args on;


               #Pass all fields of the form
               upload_pass_form_field ".*";

               # Store files to this directory
               upload_store /var/www/documents/;

               # Set specified fields in request body
               upload_set_form_field $upload_field_name.name "$upload_file_name";
               upload_set_form_field $upload_field_name.content_type "$upload_content_type";
               upload_set_form_field $upload_field_name.path "$upload_tmp_path";

               upload_cleanup 400 404 499 500-505;
                       }

       location /backend
       {
               rewrite ^/upload/(.*) /seasideapp/$1 last;
       }_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside


_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: Using nginx file upload module

Johan Brichau-2
Hi Nick,

Thanks for this.

I wonder: did you register a 'fileupload' handler in the seaside backend?

We can now get it to work when we put the file upload module on the / location, but I wonder if that is a good solution.
Your nginx config still catches only a location that matches 'fileupload'.

On 15 Jun 2011, at 13:14, Nick Ager wrote:

> Hi Johan,
>
> Sorry for the delay. I have the Nginx file upload module working. In case it's still useful here's the relevant portions of my nginx configuration:
>
>     upstream seaside {
>        server 127.0.0.1:9001;
>        server 127.0.0.1:9002;
>        server 127.0.0.1:9003;
>        fair;
>     }
>
>     server {
>        listen 80 default;
>        server_name www.getitmade.com;
>        # server_name _;
>        root /var/www;
>
>        location / {
>             try_files $uri @seaside;
>         }
>
>        # ensure that files are searched for locally and not passed to the
>        # Gems if the file isn't found on the disk
>        location ~* \.(gif|jpg|jpeg|png|css|html|htm|js|zip)$ {
>
>        }
>
>        location ~ fileupload {
>            # Pass altered request body to this location
>            upload_pass   @seaside;
>
>            # if there's no upload file in this request, nginx generates a 405 - we use this
>            # to pass the request onto the 'normal' seaside processing
>            # The way this is achieved in the other location directives is through a "try_files" method
>            # however using "try_files" results in the other directives never being processed
>            error_page 405 415 = @seaside;
>
>            # Store files to this directory
>            # The directory is hashed, subdirectories 0 1 2 3 4 5 6 7 8 9 should exist
>            upload_store /var/nginx/temp 1;
>
>            # Set the file attributes for uploaded files
>            upload_store_access user:rw group:rw all:rw;
>
>            # Set specified fields in request body
>            upload_set_form_field $upload_field_name.name "$upload_file_name";
>            upload_set_form_field $upload_field_name.content_type "$upload_content_type";
>            upload_set_form_field $upload_field_name.path "$upload_tmp_path";
>
>            # Inform backend about hash and size of a file
>            # upload_aggregate_form_field "$upload_field_name.md5" "$upload_file_md5";
>            upload_aggregate_form_field "$upload_field_name.size" "$upload_file_size";
>
>            # seaside automatically assigns sequential integers to fields with callbacks
>            # we want to pass those fields to the backend
>            upload_pass_form_field "^\d+$";
>
>            # we don't want files hanging around if the server failed to process them.
>            upload_cleanup 500-505;
>
>            # file upload progress tracking - 30s is the timeout (progress tracking is
>            # available 30s after the upload has finished)
>            # this must be the last directive in the location block.
>            track_uploads proxied 30s;
>        }
>
>        # used to report upload progress - defined by the Nginx Upload Progress Module
>        # see http://wiki.nginx.org/HttpUploadProgressModule
>        location  /progress {
>           report_uploads proxied;
>        }
>
>        location @seaside {
>           include /etc/nginx/fastcgi_params;
>           fastcgi_intercept_errors on;
>           fastcgi_pass seaside;
>        }
>
>       location /nginx_status {
>          # copied from http://blog.kovyrin.net/2006/04/29/monitoring-nginx-with-rrdtool/
>          stub_status on;
>          access_log   off;
>          allow 127.0.0.1;
>          deny all;
>       }
>
>        error_page 404 /errors/404.html;
>        error_page 403 /errors/403.html;
>        error_page 500 502 503 504 /errors/50x.html;
>     }
>
>
>
>
> On 12 June 2011 10:28, Johan Brichau <[hidden email]> wrote:
> Hi all,
>
> I'm calling for people who have got the nginx file upload module to work in their seaside apps...
>
> We are trying to use the Nginx file upload module but are stuck when the request needs to get passed on to seaside.
> We are using a configuration with a separate '/upload' location that gets handled by the file upload module. The request then needs to get passed to the upstream backend (seaside) but that request still has the URI '/upload' and we do not seem to get it rewritten to the '/seasideapp' URI.
>
>
> Is there anyone who has this working completely and could give us some advice on how solve this?
>
> The relevant parts of our nginx conf look like this:
> We also tried with a named location, but according to the docs, that makes the request arguments be dropped (and we need the _s & _k args for seaside...)
>
> Johan
>
> ----
>
>        location / {
>                         include fastcgi_params;
>                         fastcgi_pass seaside;
>                }
>
>        # Upload form should be submitted to this location
>        location /upload {
>
>                # Pass altered request body to this location
>                upload_pass /backend;
>                upload_pass_args on;
>
>
>                #Pass all fields of the form
>                upload_pass_form_field ".*";
>
>                # Store files to this directory
>                upload_store /var/www/documents/;
>
>                # Set specified fields in request body
>                upload_set_form_field $upload_field_name.name "$upload_file_name";
>                upload_set_form_field $upload_field_name.content_type "$upload_content_type";
>                upload_set_form_field $upload_field_name.path "$upload_tmp_path";
>
>                upload_cleanup 400 404 499 500-505;
>                        }
>
>        location /backend
>        {
>                rewrite ^/upload/(.*) /seasideapp/$1 last;
>        }_______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
> _______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: Using nginx file upload module

Nick
Hi Johan,

I wonder: did you register a 'fileupload' handler in the seaside backend?

We can now get it to work when we put the file upload module on the / location, but I wonder if that is a good solution.
Your nginx config still catches only a location that matches 'fileupload'.

Exactly. I think my rationale was that I wanted a way for the backend to communicate with Nginx that it should treat file upload request differently so I append 'fileupload' to the url. I tried adding a url parameter (eg ?fileupload=1&_s=...) but couldn't find a way to trap that within Nginx configuration. Originally I used #updateUrl: to add the 'fileupload' to the url, now I use javascript as I also add an 'X-Progress-ID=' for the upload tracking.

I keep meaning to write this up, I'll try to find time this weekend.

Hope this helps

Nick

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: Using nginx file upload module

Johan Brichau-2

On 15 Jun 2011, at 22:53, Nick Ager wrote:

> Exactly. I think my rationale was that I wanted a way for the backend to communicate with Nginx that it should treat file upload request differently so I append 'fileupload' to the url. I tried adding a url parameter (eg ?fileupload=1&_s=...) but couldn't find a way to trap that within Nginx configuration. Originally I used #updateUrl: to add the 'fileupload' to the url, now I use javascript as I also add an 'X-Progress-ID=' for the upload tracking.

Okay, so it's all a bit trickier than I thought.
I was trying to rewrite the request to '/upload' after nginx has processed the file, such that the normal callback invocation of the seaside application can do its work with the request (except that a hidden field is required in this case).
Unfortunately, that does not seem to work and you thus need to handle the '/upload' request in the seaside backend as well.

I'm still not entirely convinced that a rewrite of the request in nginx after the file processing should not work though ;-) I will give it another try.

But I'm looking forward to reading your write-up too!

thanks again
Johan_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: Using nginx file upload module

Johan Brichau-2
Nick,

After writing my last response, I got the picture...
Because Seaside ignores whatever comes in the path after the application name, the request will be processed by the seaside app anyway. This is what I failed to see.

In our attempts, we _replaced_ the action url on the form by something like '/fileupload?_s=...&k=...'.
Instead, you merely _append_ to the action url such that it has the form: '/<app>/fileupload?_s=...&k=...'

That last url will be correctly handled as an incoming request for <app> by seaside, ignoring whatever comes after in the path.

Great 'aha!' moment ;-) Thanks!
Johan

On 16 Jun 2011, at 08:57, Johan Brichau wrote:

>
> On 15 Jun 2011, at 22:53, Nick Ager wrote:
>
>> Exactly. I think my rationale was that I wanted a way for the backend to communicate with Nginx that it should treat file upload request differently so I append 'fileupload' to the url. I tried adding a url parameter (eg ?fileupload=1&_s=...) but couldn't find a way to trap that within Nginx configuration. Originally I used #updateUrl: to add the 'fileupload' to the url, now I use javascript as I also add an 'X-Progress-ID=' for the upload tracking.
>
> Okay, so it's all a bit trickier than I thought.
> I was trying to rewrite the request to '/upload' after nginx has processed the file, such that the normal callback invocation of the seaside application can do its work with the request (except that a hidden field is required in this case).
> Unfortunately, that does not seem to work and you thus need to handle the '/upload' request in the seaside backend as well.
>
> I'm still not entirely convinced that a rewrite of the request in nginx after the file processing should not work though ;-) I will give it another try.
>
> But I'm looking forward to reading your write-up too!
>
> thanks again
> Johan_______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: Using nginx file upload module

Nick
Hi Johan,

> But I'm looking forward to reading your write-up too!

For what it's worth I started writing it up here: http://www.nickager.com/blog/file-upload-using-Nginx-and-Seaside - though it's little more than notes at the moment..

In the meantime here is a raw dump of some code. Hope it might help, though I'm afraid it's more complex than it needs to be as it includes upload tracking, Iframe submission, additional user feedback.

I set the target of the form to a hidden Iframe:

form := html form.
form
id: #fileUploadForm;
style: 'margin:0px';
attributeAt: 'target' put: 'hiddenImageUploader';
onSubmit: 'startUploadProgressBar(); return true;';

I manipulate the path using javascript:

fileuploadSpecificJavascript
^ String streamContents: [:aStream |
aStream 
nextPutAll: 'var xProgressId="';
nextPutAll:  self xProgressId;
nextPutAll: '";
function checkFileAndSubmitIfImage(actionUrl) {
var filename = $("#fileUploadId").val();
var extensionIndex=filename.lastIndexOf(''.'') + 1;
var extension=filename.substring(extensionIndex).toLowerCase();
if (extension && /^(jpg|png|jpeg|gif)$/.test(extension)) { 
var newUrl = "/fileupload" + actionUrl+"&X-Progress-ID=" + xProgressId; 
$("#fileUploadForm").attr("action", newUrl);
$("#fileUploadForm").submit();
} else {
$("#uploadNotification").append("<ul class=\"errors\"><li>''" + filename + "'' is not a valid image file (jpg, jpeg, gif, png), try another file</li></ul>");
}

$("#fileUploadId").val("");
return false;
}
' ]

You trigger an update with the iframe as a target. The image is uploaded and a minimal response is returned to the iframe. That response triggers the onload. The onload then calls a method in the iframes parent - the main page.   I've called it imageUploadedCallback() then performs the following:
1) clears the upload progress
2) renders any upload errors
3) renders thumb nails of the images uploaded
4) sets a new xProgressId for the next upload.

The imageUploadedCallback()  code looks like:

imageUploadedCallbackScriptOn: html
^ html jQuery ajax script: [ :s |
s << (JSStream on: 'clearUploadProgress()').
s << (s jQuery: #uploadNotification) 
html: [ :renderer | 
errorText notNil 
ifTrue: [ 
renderer div
class: 'error';
with: errorText ]
ifFalse: [
renderer div] ].
s << (s jQuery: #projectUploadedFilesContainer)
html: [ :renderer |
renderer render: self imageRenderer ].
s << (JSStream on: 'xProgressId="', self xProgressId, '"') ]   
 
 
with the form I also render a hiddenInput, whose callback is fired when the form is submitted and I extract the nginx post parameters which tell me the location nginx has stored the file:

"The hiddenInput callback will contain the file upload fields if we've loading via nginx
file upload module"
html hiddenInput
value: 'hidden';
callback: [:val | | request postFields fileName uploadFieldName |
request := self requestContext request.
postFields := request postFields.
uploadFieldName := fileUploadField attributeAt: 'name'.
fileName := (postFields at: (uploadFieldName, '.name') ifAbsent: [nil]).
"has nginx file upload module inserted it's post fields in the request?"
fileName notNil ifTrue: [
fileMoveCallback value: postFields value: uploadFieldName ] ]

I use the OSes 'mv' to move the files:

moveFrom: fromString to: toString
| shellMoveFileCommand |
shellMoveFileCommand := String streamContents: [:stream |
stream 
nextPutAll: 'mv ';
nextPutAll: fromString;
nextPutAll: ' ''';
nextPutAll: toString;
nextPutAll: '''' ].
IZLogger logInfo: 'IZShellScripts>>#moveFrom:to:' with: shellMoveFileCommand.
SpEnvironment runShellCommandString: shellMoveFileCommand.

some more javascript:

imageUploadedCallbackScriptOn: html
^ html jQuery ajax script: [ :s |
s << (JSStream on: 'clearUploadProgress()').
s << (s jQuery: #uploadNotification) 
html: [ :renderer | 
errorText notNil 
ifTrue: [ 
renderer div
class: 'error';
with: errorText ]
ifFalse: [
renderer div] ].
s << (s jQuery: #projectUploadedFilesContainer)
html: [ :renderer |
renderer render: self imageRenderer ].
s << (JSStream on: 'xProgressId="', self xProgressId, '"') ]  

and one more javascript method:

startUploadProgressBarJavascript 
^'
var interval = null;

function startUploadProgressBar() {
$("#progressBar").progressbar({ value: 0 });
$("#progressBar").css("display", "block");
$("#fileUploadContainer").css("display", "none");
interval = window.setInterval( function () {  fetch(xProgressId);  }, 1000);
}

function clearUploadProgress() {
window.clearTimeout(interval);
$("#progressBar").css("display", "none");
$("#fileUploadContainer").css("display", "block");
}

function setUploadStatus (statusText) {
$("#uploadNotification").html("<div class=''status''>" + statusText + "</div>");
}

function fetch(uuid) {
$.ajax({
url: "/progress",
beforeSend: function (xhr) {
xhr.setRequestHeader("X-Progress-ID", uuid);
},
success: function (data, textStatus, xhr) {
var upload = eval(data);

         if (upload.state == "done" || upload.state == "uploading") {
var percentageComplete = Math.floor(upload.received * 100 / upload.size);
$("#progressBar").progressbar({ value: percentageComplete });
if (upload.received != upload.size) {
setUploadStatus("Uploading: " + percentageComplete + "%");
} else {
setUploadStatus("Uploaded: processing");
}
         }
if (upload.state == "starting") {
setUploadStatus("Starting upload");
}
         if (upload.state == "done") {
           clearUploadProgress();
         }
},
error: function(xhr, textError) {
setUploadStatus("Error: " + textError);
}
});
}
'

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: Using nginx file upload module

Nick
In reply to this post by Johan Brichau-2
Johan,

In our attempts, we _replaced_ the action url on the form by something like '/fileupload?_s=...&k=...'.
Instead, you merely _append_ to the action url such that it has the form: '/<app>/fileupload?_s=...&k=...'

Exactly though in my case I cut out the dispatcher and serve the requests directly from my application so I don't need to bother rewriting the incoming request.  More info here: http://nickager.com/blog/serving-seaside-requests-without-the-application-name-in-the-URL/

HTH

Nick

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside