WAFileLibrary / Resource Path

I'm using Amazon S3 to host my static content, which is easy enough  
to do using methods like WAImageTag>>resourceUrl:. But the content of  
the various file libraries is static too, so it seems like it'd make  
sense to host it on S3 as well.

The problem I'm having is that the URLs of file library content isn't  
relative to the resource base -- so it always looks for /seaside/
files/SULibrary/effects.js, where I'd really prefer something like:

        http://mybucket.amazonaws.com/SULibrary/effects.js

or

        http://s3.amazonaws.com/myBucket/SULibary/effects.js

What would be the best way to refactor the code to accomplish this?  
If nobody's interested in this sort of feature, then I guess I can  
hack it however I want. But if other people are interested, I'd like  
to know how to make it as useful as possible.

It seems like this behavior should be configurable. Should I just  
make it use the resource base URL, or keep a separate base URL for  
files?

--
Ken Treis
Miriam Technologies, Inc.

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

2007/8/2, Ken Treis <[hidden email]>:
Right now the simplest thing do so is to remove SULibrary and all the
other libraries you want to be served over  S3 from you application.
Then implement #updateRoot: in your root component :

updateRoot: html
    super updateRoot: html.
    html script resourceUrl: '/SULibary/effects.js'

> If nobody's interested in this sort of feature, then I guess I can
> hack it however I want. But if other people are interested, I'd like
> to know how to make it as useful as possible.

Make it work for you first. Then worry about the rest.

> It seems like this behavior should be configurable. Should I just
> make it use the resource base URL, or keep a separate base URL for
> files?

Cheers
Philippe
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

On 02/08/07, Ken Treis <[hidden email]> wrote:
Libraries are designed to be served from Seaside, and those SULibrary
files are just the standard Scriptaculous distribution that comes
bundled with Seaside. If you don't want Seaside to serve them, you
should remove SULibrary from the app's libraries, grab the
Scriptaculous files from the Scriptaculous site, and use #updateRoot:
to link to them on S3.
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

> It seems like this behavior should be configurable. Should I just
> make it use the resource base URL, or keep a separate base URL for
> files?

I guess what you request are the settings 'Server Hostname', 'Server
Path', 'Server Port' and 'Server Protocol' available for WAApplication
only?

--
Lukas Renggli
http://www.lukas-renggli.ch
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

On Aug 2, 2007, at 1:46 PM, Patrick Collison wrote:

> Libraries are designed to be served from Seaside,

Sure, and it works great that way for development. They keep my small  
static assets in the same version-controlled bundle with my code. But  
when it comes time to deploy, I'd like to be able to squirt them up  
to S3, flip a switch, and have them served from there instead.

> and those SULibrary
> files are just the standard Scriptaculous distribution that comes
> bundled with Seaside.

I thought so too until I looked at them. The treePatch.js file isn't  
standard as far as I can tell. I haven't compared the other files to  
the distribution, but that's the point. I don't want to compare them.  
I want to push them up once at deployment and know that they're  
identical to the files I've been testing with.

Even if I use updateRoot: on my component to do this for  
Scriptaculous, I would still need a solution for WAStandardFiles. I  
think I will add a setter for WAApplication>>libraries: so that I can  
force them to an empty collection. Then I will use my app's  
#updateRoot: to set them to resource URLs.

Thanks everyone for the suggestions.

--
Ken Treis
Miriam Technologies, Inc.

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

My deployment script just gathers all files from all libraries and pushes them to s3 with the rest of it and then links to them from updateroot after removing those classes from libraries collection. I didn't need to hack anything to get it to work and it was reasonably easy to do.

Cheers!

-Boris
(Sent from a BlackBerry)

----- Original Message -----
From: [hidden email] <[hidden email]>
To: Seaside - general discussion <[hidden email]>
Sent: Thu Aug 02 15:05:59 2007
Subject: Re: [Seaside] WAFileLibrary / Resource Path

On Aug 2, 2007, at 1:46 PM, Patrick Collison wrote:

> Libraries are designed to be served from Seaside,

Sure, and it works great that way for development. They keep my small 
static assets in the same version-controlled bundle with my code. But 
when it comes time to deploy, I'd like to be able to squirt them up 
to S3, flip a switch, and have them served from there instead.

> and those SULibrary
> files are just the standard Scriptaculous distribution that comes
> bundled with Seaside.

I thought so too until I looked at them. The treePatch.js file isn't 
standard as far as I can tell. I haven't compared the other files to 
the distribution, but that's the point. I don't want to compare them. 
I want to push them up once at deployment and know that they're 
identical to the files I've been testing with.

Even if I use updateRoot: on my component to do this for 
Scriptaculous, I would still need a solution for WAStandardFiles. I 
think I will add a setter for WAApplication>>libraries: so that I can 
force them to an empty collection. Then I will use my app's 
#updateRoot: to set them to resource URLs.

Thanks everyone for the suggestions.

--
Ken Treis
Miriam Technologies, Inc.

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

> I thought so too until I looked at them. The treePatch.js file isn't
> standard as far as I can tell. I haven't compared the other files to
> the distribution, but that's the point. I don't want to compare them.
> I want to push them up once at deployment and know that they're
> identical to the files I've been testing with.

Yes, but this is only needed if you use SUSortable with trees enabled.
Unfortunately I had to patch here, to make it possible for Seaside to
gather the data. If you don't use SUSortable with trees enabled you
don't need this file.

Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Ken Treis wrote:
>  ...
>
>     http://s3.amazonaws.com/myBucket/SULibary/effects.js
>
> What would be the best way to refactor the code to accomplish this? If
> nobody's interested in this sort of feature, then I guess I can hack
> it however I want. But if other people are interested, I'd like to
> know how to make it as useful as possible.
>
Dear Ken,

If you have a look in the Jetsam28Seaside package in
http://www.squeaksource.com/Jetsam you will find a number of adaptations
to the standard seaside file serving capability.

I haven't used this myself do to what you are asking, but I did have it
in mind. I will outline the principles as to how you can achieve what
you want to do quite easily.

1. Define your file library 'YourAppLibrary' as a subclass of
WAFileLibrary. If you want to include the standard seaside stuff as well
then subclass WAStandardFiles. The version in Jetsam allows subclassing
and inheritance/overriding of file selector based resources.

2. In your rendering code, refer to your links like so

html image url: YourAppLibrary / 'example.jpg'  (This is another
enhancement)

or the standard

html image url: YourAppLibrary / exampleJpg

3a For in image serving, use your library in the standard way,
3b For serving from disk, set the #libraryName method to be the name of
the directory of the files you wish to serve, if it finds a file in
there that file will override the in image version.
3c. For serving from an external source (as requested...)

The std version hardwires the baseUrl provider to 'WAFileHander
default'. The Jetsam version gets this from #fileHandler. So all you
have to do is override #fileHander to return something which returns
#baseUrl.

So the easiest option is to change #fileHandler to return self, and you
set #baseUrl to return the url of the external resources. If should be
possible to wire this up to the current application's configuration
settings if you want to.

let me know how you get on

Keith







_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Ken,
>
> So the easiest option is to change #fileHandler to return self, and
> you set #baseUrl to return the url of the external resources. If
> should be possible to wire this up to the current application's
> configuration settings if you want to.
>
> let me know how you get on
>
> Keith
 Actually I will just refactor out the #filehandler step and use
#baseUrl directly. Then all you have to do is override
#baseUrl

Notice that it will still add the #libraryName to the #baseUrl you specify

best regards

Keith
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Hi Ken,

        maybe is too KISS but I've made this a couple of hours ago because I
didn't want to manage files manually too much. So with this an image is able
to write in disk all the libraries that a seaside app uses writed in a
consistent shape under your image dir.

        I'll use rewrite rules in apache so copying that file directory path
should be enough I think. Maybe you find a way to copy that to S3 and make
your apps work in deployment/development by making behave differently in
#updateRoot: ? Could be "the switch" you asked for?

        Be warned: I'm testing it just now myself so.. no guarantees ;)

Sebastian Sastre

 

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Ah, I forgot to mention that you use it by evaluating:

        (WADispatcher default entryPointAt: 'applicationName') deployFiles

        then search the created folder called 'files' and copy that tree
anywere deployment files is expected to be.

        No need comparisions.

        Enjoy

Sebastian Sastre

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

2007/8/14, Sebastian Sastre <[hidden email]>:
I added it to Seaside2.8a1-pmm.435 with the usual
SeasidePlatformsupportfication (two more methods :(, where is this
gonna end?). I didn't add #refreshResources. The main problem here is
that people might use it to deploy resources. This kinda sounds like a
joke but WAFileLibrary is not suited for this. Problem is that it
hideously bloats the source size for binary files. An other is that
serving big files over Seaside is not recommended and we can't tell
you where the limit is to the size of files Seaside can serve.
Deployment of resources is an issue that needs to be solved at the
Smalltalk or version control level.

Cheers
Philippe
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

On Aug 13, 2007, at 10:32 PM, Philippe Marschall wrote:

I think there's a typo in Seaside2.8a1-pmm.435.. in the method  
signature SeasidePlatformSupport>>wirte:toFile:inFolder:.

--
Ken Treis
Miriam Technologies, Inc.

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

I apologize for quoting Philippe out of context here, but we've been looking at the issue of uploading/dowloading large files via Seaside recently and my conclusion is that especially for downloading it's not really Seaside's problem to handle. It's for the associated HTTP server to worry about. To test that we can stream downloaded files directly into the socket, all I did was add a subclass of WAFileLibrary and overrode documentAt:ifAbsent: as follows:

WAExternalFileLibrary>>documentAt: aFilename ifAbsent: aBlock

        | fn |
        fn := '.' asFilename construct: aFilename.
        ^fn exists
                ifTrue: [ | mimeType |
                        mimeType := self mimetypeForFile: aFilename.
                        WAResponse new
                                contentType: (mimeType ifNil: ['application/octet-stream']);
                                headerAt: 'Expires' put: 'Thu, 01 Jan 2095 12:00:00 GMT';
                                stream: fn readStream;
                                yourself ]
                ifFalse: [ aBlock value ]

Ignore the hardwired directory for the moment. With this I could hit  http://localhost:7777/seaside/go/files/WAExternalFileLibrary/visual.im and the 11M image file was downloaded pretty much instantly with the browser and server on the same host.

The key is that we build a WAResponse here with an external stream in it, and the rest is handled by the HTTP server. Am I missing something from the bigger picture ?

Thanks,

Martin

Philippe Marschall wrote:
> An other is that serving big files over Seaside is not recommended and we can't tell
> you where the limit is to the size of files Seaside can serve.
> Deployment of resources is an issue that needs to be solved at the
> Smalltalk or version control level.
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

That sounds like a great idea, if you have enough hooks to pass streams
around without reading/writing them yourself, you can sidestep the issue
completely and let the HTTP front broker deal with it efficiently, I
like it.

-Boris

--
+1.604.689.0322
DeepCove Labs Ltd.
4th floor 595 Howe Street
Vancouver, Canada V6C 2T5
http://tinyurl.com/r7uw4

[hidden email]

CONFIDENTIALITY NOTICE

This email is intended only for the persons named in the message
header. Unless otherwise indicated, it contains information that is
private and confidential. If you have received it in error, please
notify the sender and delete the entire message including any
attachments.

Thank you.
server to
> worry about. To test that we can stream downloaded files directly into
the ['application/octet-
> stream']);
> headerAt: 'Expires' put: 'Thu, 01 Jan
2095
> 12:00:00 GMT';
> stream: fn readStream;
> yourself ]
> ifFalse: [ aBlock value ]
>
> Ignore the hardwired directory for the moment. With this I could hit
> http://localhost:7777/seaside/go/files/WAExternalFileLibrary/visual.im
and
> the 11M image file was downloaded pretty much instantly with the
browser
> and server on the same host.
>
> The key is that we build a WAResponse here with an external stream in
it,
> and the rest is handled by the HTTP server. Am I missing something
from
> the bigger picture ?
>
> Thanks,
>
> Martin
>
> Philippe Marschall wrote:
> > An other is that serving big files over Seaside is not recommended
and
> we can't tell
> > you where the limit is to the size of files Seaside can serve.
> > Deployment of resources is an issue that needs to be solved at the
> > Smalltalk or version control level.
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

On Aug 13, 2007, at 6:37 PM, Sebastian Sastre wrote:

URL rewriting is an interersting approach, but doesn't really work  
for S3. I need to use a completely different hostname in the URL for  
the files. It seems to fit much more nicely into the idea of using  
the resourceBaseUrl.

Since WAFileLibrary doesn't have a convenient way to generate  
resource-relative URLs -- without "/seaside/files" prepended -- I  
added a couple of methods to accomplish this. It's not strictly  
necessary by itself -- I can replicate the /seaside/files path in my  
S3 bucket, but I didn't want to have to mirror the entire path on S3.

I like the way Seaside 2.8 uses a binary selector #/ for the URL of  
the FileHandler, so I used #* for my more relative flavor. That way I  
can do something like this:

(html image)
        class: 'tl';
        resourceUrl: VisualAssets * #boxCornerTLPng. "returns 'VisualAssets/
boxCornerTL.png'"

In #updateRoot:, I didn't want to specify all of the individual  
javascripts and stylesheets, so I added a method to WAFileLibrary  
called #updateResourceRoot:. It works just like the regular  
#updateRoot: but it uses resourceUrl: and the stripped-down relative  
URLs instead. Then I remove all libraries and implement my  
application's #updateRoot as follows:

updateRoot: aRoot
        super updateRoot: aRoot.
        (Array
                with: Seaside.WAStandardFiles
                with: Seaside.SULibrary
                with: VisualAssets) do: [:each | each default updateResourceRoot:  
aRoot]

That way, in development, I can simply set resourceBaseUrl to /
seaside/files, and in deployment I set it to http://s3.amazonaws.com/ 
myBucket. Deploying to S3 can be done directly from within the image  
without using files, since it's all done over HTTP.

This is how I solved the problem, anyhow. It may be overkill, but it  
works well enough for me.

--
Ken Treis
Miriam Technologies, Inc.

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

2007/8/14, Martin Kobetic <[hidden email]>:
That's true. WAFileLibrary is a simple solution for simple problems.
It doesn't work. Handling "big" files in Seaside is generally
discouraged. We prefer to leave it to traditional file servers like
Apache or lighy which are optimized for this task. This also takes
load off your Smalltalk Image.

> The key is that we build a WAResponse here with an external stream in it, and the rest is handled by the HTTP server. Am I missing something from the bigger picture ?

Depends on whether you server adaptor supports output streaming.

Cheers
Philippe
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Philippe Marschall wrote:
> That's true. WAFileLibrary is a simple solution for simple problems.
> It doesn't work. Handling "big" files in Seaside is generally
> discouraged. We prefer to leave it to traditional file servers like
> Apache or lighy which are optimized for this task. This also takes
> load off your Smalltalk Image.

I understand. Actually, we've been recommending the same for VW based web servers for long time as well.
However there were several "streaming server" projects recently with quite encouraging results, suggesting that we may be able to push the limits of pure Smalltalk servers quite a bit higher than we would dare just a few years ago. I certainly don't expect to beat Apache and such in their game, but a fairly large class of web applications might be just fine with pure Smalltalk solution with all its benefits. So I feel that we shouldn't just dismiss the possibility of serving files from Smalltalk outright.

>> The key is that we build a WAResponse here with an external stream
>>  in it, and the rest is handled by the HTTP server. Am I missing
>> something from the bigger picture ?
>
> Depends on whether you server adaptor supports output streaming.

Well, I guess, for some definition of "streaming". AFAIK there are following alternatives for handling a large file at the HTTP level.

1) switch the connection to transient for the request ('Connection: close' or 'Keep-Alive: false' header), write the file into the socket and close the connection.

2) if you can determine the size of the file upfront, which is often fairly easy, and if you don't intend to use compressed transfer encoding, just use the size for the Content-Length header.

3) use chunked transfer encoding (I believe that's HTTP 1.1 only)

Any of these would allow you to handle the file reasonably efficiently. Of course the server can still just read the stream into memory and deal with it as with any other content, just less efficiently. That's what I meant by the "it's up to the HTTP server".

Cheers,

Martin

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

2007/8/14, Martin Kobetic <[hidden email]>:
Your web server accepts a response object who's contents are a stream
and sends them out over http without fully reading the stream upfront.

Philippe
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

But what would stop servers not capable of 'streaming' from reading the
contents of the stream and serving the same thing with Content-Length?
Surely you'd have a memory hit, but that's why the preference would be
to use a server that supports streaming. I think we might all be talking
about the same thing here and agreeing on it at the same time.

-Boris

--
+1.604.689.0322
DeepCove Labs Ltd.
4th floor 595 Howe Street
Vancouver, Canada V6C 2T5
http://tinyurl.com/r7uw4

[hidden email]

CONFIDENTIALITY NOTICE

This email is intended only for the persons named in the message
header. Unless otherwise indicated, it contains information that is
private and confidential. If you have received it in error, please
notify the sender and delete the entire message including any
attachments.

Thank you.
problems.
> > > It doesn't work. Handling "big" files in Seaside is generally
> > > discouraged. We prefer to leave it to traditional file servers
like
> > > Apache or lighy which are optimized for this task. This also takes
> > > load off your Smalltalk Image.
> >
> > I understand. Actually, we've been recommending the same for VW
based
> web servers for long time as well.
> > However there were several "streaming server" projects recently with
> quite encouraging results, suggesting that we may be able to push the
> limits of pure Smalltalk servers quite a bit higher than we would dare
> just a few years ago. I certainly don't expect to beat Apache and such
in
> their game, but a fairly large class of web applications might be just
> fine with pure Smalltalk solution with all its benefits. So I feel
that we
> shouldn't just dismiss the possibility of serving files from Smalltalk
> outright.
> >
> > >> The key is that we build a WAResponse here with an external
stream the
> HTTP level.
> >
> > 1) switch the connection to transient for the request ('Connection:
> close' or 'Keep-Alive: false' header), write the file into the socket
and
> close the connection.
> >
> > 2) if you can determine the size of the file upfront, which is often
> fairly easy, and if you don't intend to use compressed transfer
encoding,
> just use the size for the Content-Length header.
> >
> > 3) use chunked transfer encoding (I believe that's HTTP 1.1 only)
> >
> > Any of these would allow you to handle the file reasonably
efficiently.
> Of course the server can still just read the stream into memory and
deal
> with it as with any other content, just less efficiently. That's what
I _______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside