Gemtools in Seaside 3.0

Hi All, 

Do we have any progress on Gemtools for Seaside 3.0 (based on Pharo 1.2) and the Cog'ed VM? 

I have tried the one-click download from Gemstone (GemTools-1.0-beta.8-244x.app, apparently based on Pharo 1.1) and it runs a very strange memory image on my MacBook Air: 31MB real mem, 286MB virtual (but the current 1.1.1 one-click is similar with 37/286). This kills performance, fex: it takes 10 seconds to open a Metacello repository, and even bringing up the world menu takes a few seconds. 

I've tried loading the latest Gemtools - which I take to be the latest ConfigurationOfGemstools from the Metacello repository at squeaksource - into Seaside 3.0 but it fails with:

preloadDoitForPharo11 

        Preferences allowUnderscoreAssignmentSetting: true. 

If I remove this line and set the 'allow underscores' in the preferences to true and continue it fails further down the line with other errors (the OB was involved I seem to recall). 

I think it would really help to have Gemstools in Seaside 3.0 as 3.0 is faster, smaller and now the Seaside base for the foreseeable (at least according to the team).   

T. 

Ted Wrinch 

Ted,

Before heading too far down this path, please checkout the thread on running GemTools on a Mac[1]. The upshot is that there is an FFI bug in the vm that was introduced in all post 4.2.2 vms, so you have to use version 4.2.2 or earlier on the Mac. There is also a bug report on glassdb for this issue[2].

With that said, I have been regularly using Pharo1.1.1 with GemTools 1.0-beta.8.5 loaded and I have not seen any poor performance or odd things going on my mac. I have not tried running GemTools in later versions of Pharo, mainly because for the longest time I thought the issue was related to the version of Pharo that I was using, not the version of the vm...

Are you running the stone/gems on your Mac? The kind of performance you are describing sounds like you are running GemTools over the WAN. Unfortunately, GemTools is very chatty over the network and there can be long delays while the GemTools windows are updated.

I am working on tODE as a replacement for GemTools (tODE runs in a web browser and is written in Seaside30). I can do about 85% of my development work from within tODE right now, so it is very promising.

With _that_ said, GemTools and Seaside30 are orthogonal. GemTools is a development environment that runs inside client Smalltalk like Pharo/Squeak and connects to GemStone over the network. GemTools does not require Seaside (2.8 or 3.0) to run and Seaside does not require GemTools, so I am a little bit confused.

If you are developing in Pharo and deploying in GemStone I think that it is valid to run two separate VMs: one in which you do your Pharo-based Seaside3.0 development and one in which you connect to GemStone and do your GemStone-based development, especially if you want to use a version of Pharo for your Seaside3.0 development that is not compatible with GemTools...

With __that__ said, I am interested in getting to the bottom of you performance issue if the above comments have not lead to an alternate resolution.

Dale

[1] http://forum.world.st/GemTools-for-pharo-1-1-td2542305.html
[2] http://code.google.com/p/glassdb/issues/detail?id=258

----- Original Message -----
| From: "Ted Wrinch" <[hidden email]>
| To: [hidden email]
| Sent: Friday, July 22, 2011 12:07:09 PM
| Subject: [GS/SS Beta] Gemtools in Seaside 3.0
|
|
|
| Hi All,
|
| Do we have any progress on Gemtools for Seaside 3.0 (based on Pharo
| 1.2) and the Cog'ed VM?
|
| I have tried the one-click download from Gemstone
| (GemTools-1.0-beta.8-244x.app, apparently based on Pharo 1.1) and it
| runs a very strange memory image on my MacBook Air: 31MB real mem,
| 286MB virtual (but the current 1.1.1 one-click is similar with
| 37/286). This kills performance, fex: it takes 10 seconds to open a
| Metacello repository, and even bringing up the world menu takes a
| few seconds.
|
| I've tried loading the latest Gemtools - which I take to be the
| latest ConfigurationOfGemstools from the Metacello repository at
| squeaksource - into Seaside 3.0 but it fails with:
|
| preloadDoitForPharo11
|
| Preferences allowUnderscoreAssignmentSetting: true.
|
| If I remove this line and set the 'allow underscores' in the
| preferences to true and continue it fails further down the line with
| other errors (the OB was involved I seem to recall).
|
| I think it would really help to have Gemstools in Seaside 3.0 as 3.0
| is faster, smaller and now the Seaside base for the foreseeable (at
| least according to the team).
|
| T.
|
| Ted Wrinch
|
|
|
|
|

Hi Dale,

Thanks for responding.

Yes, I am running everything on my Mac. I was aware that there is a problem with the post 1.1 VM and had seen Norbert's fix, but hadn't realised this was still an issue.

I am developing in Seaside 3.0 against an image dbase. What I would like to do is use Gemstone to go to market. My image dbase been designed so that there should be no, or very little, extra work or changes to swap in a Gemstone dbase. I'm not sure that I understand what you mean when you say that Seaside 3.0 and Gemtools are orthogonal. As I understand it, the approach in the Glass appliance and the native image is to load Gemtools onto Pharo. Seaside is also a load of a few Monticello packages onto Pharo. What I had expected that I would need to do to use Gemstone is to load in some connection and proxying etc Gemstone tools into my Seaside image (maybe not the full Gemtools package, from what you suggest?).  

T.

Ted Wrinch

Hi Dale,

Thanks for responding.

Yes, I am running everything on my Mac. I was aware that there is a problem with the post 1.1 VM and had seen Norbert's fix, but hadn't realised this was still an issue.

I am developing in Seaside 3.0 against an image dbase. What I would like to do is use Gemstone to go to market. My image dbase been designed so that there should be no, or very little, extra work or changes to swap in a Gemstone dbase. I'm not sure that I understand what you mean when you say that Seaside 3.0 and Gemtools are orthogonal. As I understand it, the approach in the Glass appliance and the native image is to load Gemtools onto Pharo. Seaside is also a load of a few Monticello packages onto Pharo. What I had expected that I would need to do to use Gemstone is to load in some connection and proxying etc Gemstone tools into my Seaside image (maybe not the full Gemtools package, from what you suggest?).  

T.

Ted Wrinch

Hi Ted,

Use the one-click image + vm from the bottom of the page here:

http://seaside.gemstone.com/downloads.html

to load your packages and Seaside 3.0 into your Gemstone installation.

Gemtools is just a package that allows you to use Pharo as a GUI for
developing on/in Gemstone.  So any web framework you choose to run on/in
Gemstone wouldn't affect it in any way.  But Gemtools will not run in
versions of Pharo > 1.1 and also will not run if you use the cog vm.  So
use the one-click to load your stuff into Gemstone.

Generally you're best off developing the app in Pharo, running the code
critics tool and fixing all the Grease issues, saving the packages with
monticello, and then load those packages into Gemstone using Metacello
or manually.

If your image database has data you need to get into gemstone you'll
need to use SIXX or Fuel or ??? to serialize it to a file from Pharo and
then load it into gemstone.


Good luck.


Paul









On 07/23/2011 07:11 AM, Ted Wrinch wrote:

Thanks Paul,

I did try that image and it ran slow and had those strange memory characteristics I described in my first post.

From your post, I think I see what I may have incorrectly assumed as the way I should be using Gemstone. I had assumed that I should run Seaside images in production that talk to Gemstone (via the Gems I guess) to do the dbase work. If I understand you correctly, the preferred approach is to load Seaside, and one's application code, into the Gemstone server, via the Pharo Gemtools image or whatever, and run just the Gems. So instead of a Seaside VM responding to an HTTP request a Gem does. Is this right?

I was concerned that by not being able to run Seaside 3.0 we would miss the performance benefits of the Cog'ed VM. But as Gemstone itself, as I understand it, has a native code VM this may not be an issue. One thing that still does concern me is that I understand that running Seaside in Gemstone this way causes every HTTP request to be wrapped in a Gemstone transaction. This seems like overhead to me for requests that are not updating anything in the dbase (the majority I'd think). On Seaside 3.0 on my 1.86GHz MacBook Air SSD I can get 200 8k pages / second - nothing definitive but quite good I'd say. I think I saw it on one of the Gemstone blogs that Gemstone may reduce this to perhaps 50 /second.

As far as data upload goes: I already read it from a file and this approach could be continued from the Gemstone server I'd assume. But my data is stored in a class variable, which I could probably ship over to the server. When I last used Gemstone, back in '96 at IBM Hursley Park, we used to do a lot of executeOnServer: type calls (not sure now what the method was actually called).  

T.
   
Ted Wrinch

Ted,

Ah, that explains it... Yes, you want to run the Seaside application completely in GemStone, serving http requests directly from gems.

In the old days, you would typically run GemStone as a server with a GUI in a client Smalltalk (like VW or VA) and the client smalltalk would communicate with GemStone via GCI. The `executeOnServer:` sounds like a GBS (GemBuilder for Smalltalk)[4] call and would have been used to run server-side code from the client smalltalk.

Web apps are strictly server-side applications without any need for a GUI so the entire system can be deployed in GemStone with no need for client smalltalk vms to be involved.

Today, most of the folks using GemStone will do development in their Pharo/Squeak image and then deploy their application in GemStone. GemTools is provided strictly as a GUI for development purposes. Pharo/Squeak will always provide a superior set of development tools, while GemStone provides a stable, scalable deployment platform.

Regarding "a commit per request". It turns out to be real useful to have the transaction boundary correspond to the request boundary (there is no need to put any transaction logic into your application) ... it seems that every time we come up against a problem that suggests we should move the transaction boundaries around we find that we were mistaken:)

If you want to avoid persisting session state then you should look into using SeasideREST[1] for your requests ... with SeasideREST session state is not used, so if the request doesn't dirty any persistent data, the commit is treated as an abort.

Regarding "benchmarks"...I've benchmarked Seaside2.8 running on GemStone at 7000 requests/second ... sustained[2].

Finally, if you use SSD drives, you will see a significant speed boost for GemStone[3] as the limiting factor in all GemStone tests is disk i/o for commits...

I am still a bit curious about the things that lead to your odd memory/performance behavior because I just don't see that happen on my mac laptop...

Dale

[1] http://code.google.com/p/seaside/wiki/SeasideRest
[2] http://gemstonesoup.wordpress.com/2009/10/18/glass-a-share-everything-architecture-for-seaside-esug-2008/
[3] http://gemstonesoup.wordpress.com/2009/02/28/approaching-the-speed-of-light-ssd-drives-for-gemstones/
[4] http://seaside.gemstone.com/docs/GemBuilder_Smalltalk.pdf

----- Original Message -----
| From: "Ted Wrinch" <[hidden email]>
| To: [hidden email]
| Sent: Saturday, July 23, 2011 8:33:39 AM
| Subject: Re: [GS/SS Beta] Gemtools in Seaside 3.0
|
| Thanks Paul,
|
| I did try that image and it ran slow and had those strange memory
| characteristics I described in my first post.
|
| >From your post, I think I see what I may have incorrectly assumed as
| >the way
| I should be using Gemstone. I had assumed that I should run Seaside
| images
| in production that talk to Gemstone (via the Gems I guess) to do the
| dbase
| work. If I understand you correctly, the preferred approach is to
| load
| Seaside, and one's application code, into the Gemstone server, via
| the Pharo
| Gemtools image or whatever, and run just the Gems. So instead of a
| Seaside
| VM responding to an HTTP request a Gem does. Is this right?
|
| I was concerned that by not being able to run Seaside 3.0 we would
| miss the
| performance benefits of the Cog'ed VM. But as Gemstone itself, as I
| understand it, has a native code VM this may not be an issue. One
| thing that
| still does concern me is that I understand that running Seaside in
| Gemstone
| this way causes every HTTP request to be wrapped in a Gemstone
| transaction.
| This seems like overhead to me for requests that are not updating
| anything
| in the dbase (the majority I'd think). On Seaside 3.0 on my 1.86GHz
| MacBook
| Air SSD I can get 200 8k pages / second - nothing definitive but
| quite good
| I'd say. I think I saw it on one of the Gemstone blogs that Gemstone
| may
| reduce this to perhaps 50 /second.
|
| As far as data upload goes: I already read it from a file and this
| approach
| could be continued from the Gemstone server I'd assume. But my data
| is
| stored in a class variable, which I could probably ship over to the
| server.
| When I last used Gemstone, back in '96 at IBM Hursley Park, we used
| to do a
| lot of executeOnServer: type calls (not sure now what the method was
| actually called).
|
| T.
|    
| Ted Wrinch
|
| --
| View this message in context:
| http://forum.world.st/Gemtools-in-Seaside-3-0-tp3687549p3689009.html
| Sent from the GLASS mailing list archive at Nabble.com.
|

Dale,

Thanks for your reply.

The old days, including GemBuilder (I'd forgotten that name!), were exactly as you describe. This is what had initially guided my thinking and is what I am having to move away from. We used to avoid putting code on the server, partly due to client-server Smalltalk differences, and partly because the Envy code repository we used was not able to upload code to the server. Your consultancy arm of those days eventually supplied our team with a somewhat hacked enhancement to Envy to achieve this. Your use of Monticello today to do this looks to be a significant step forward.

But I am still somewhat concerned about reliably and safely bridging the client/server Smalltalk differences. I understand that you use Grease to facilitate this but you do seem to have a few missing/overwritten method gotchas that come up from time to time, as well as cases where code upload seems to slow down. In general bridging this gap seems to me to be a possible point of risk during the move to production - but I hope that I'm wrong.

That's an impressive benchmark! I will take a look at your video. Sounds like commits are not a problem and I take your point about the ease of use you get this way. SSDs have seemed the way to go for dbases for a while to me, but I wasn't able to get much traction with that idea at the last company I worked for (but it will come..).  

I don't really know what to say about my slow Pharo 1.1 laptop experience. But there is something peculiar about that image's virtual memory usage being so high, at over 200MB. It's only a few 10s on Seaside 3.0 (Pharo 1.2).

T.

Ted Wrinch

2011/7/23 Ted Wrinch <[hidden email]>:
There should be some Oracle keynotes about how it's the way of the
future. If that doesn't help then I don't know.

Cheers
Philippe

Ted,

Monticello and Metacello makes it relatively straightforward to maintain a code base that runs on multiple smalltalk platforms.

As for platform code differences. If you are intentionally writing for multiple platforms, you can usually bridge the gaps pretty easily ... In 2.4 the Stream hierarchy is probably the biggest barrier to easy portability because the basic implementation of PositionableStream in GemStone is so different from the implementation in other Smalltalks. In 3.0 we've implemented a "portable" stream hierarchy that is compatible with the majority implementation (note that since the current GLASS code base is dependent upon the "legacy" stream hierarchy, the "portable" hierarchy is not the default).

The Seaside team has been writing for multiple platforms and they have created a set of tools that analyzes your code looking for known non-portable constructs the "code critics tool" that Paul mentioned in his mail.

The Mock GemStone project[1] was created to provide implementations of some GemStone methods/classes that you may find convenient to have in Pharo.

Since Metacello allows you to manage platform-specific or even project-specific source files you've got a bit of flexibility for customizing things in sae and manageable manner.

Dale

[1] http://seaside.gemstone.com/ss/MockGemStone.html

----- Original Message -----
| From: "Philippe Marschall" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Sunday, July 24, 2011 2:41:13 AM
| Subject: Re: [GS/SS Beta] Gemtools in Seaside 3.0
|
| 2011/7/23 Ted Wrinch <[hidden email]>:
| > Dale,
| >
| > Thanks for your reply.
| >
| > The old days, including GemBuilder (I'd forgotten that name!), were
| > exactly
| > as you describe. This is what had initially guided my thinking and
| > is what I
| > am having to move away from. We used to avoid putting code on the
| > server,
| > partly due to client-server Smalltalk differences, and partly
| > because the
| > Envy code repository we used was not able to upload code to the
| > server. Your
| > consultancy arm of those days eventually supplied our team with a
| > somewhat
| > hacked enhancement to Envy to achieve this. Your use of Monticello
| > today to
| > do this looks to be a significant step forward.
| >
| > But I am still somewhat concerned about reliably and safely
| > bridging the
| > client/server Smalltalk differences. I understand that you use
| > Grease to
| > facilitate this but you do seem to have a few missing/overwritten
| > method
| > gotchas that come up from time to time, as well as cases where code
| > upload
| > seems to slow down. In general bridging this gap seems to me to be
| > a
| > possible point of risk during the move to production - but I hope
| > that I'm
| > wrong.
| >
| > That's an impressive benchmark! I will take a look at your video.
| > Sounds
| > like commits are not a problem and I take your point about the ease
| > of use
| > you get this way. SSDs have seemed the way to go for dbases for a
| > while to
| > me, but I wasn't able to get much traction with that idea at the
| > last
| > company I worked for (but it will come..).
|
| There should be some Oracle keynotes about how it's the way of the
| future. If that doesn't help then I don't know.
|
| Cheers
| Philippe
|

Dale,

It seems like I should try and develop against 3.0. You've had the beta out for some while but when I ran GemTools against it it complained that the client-server versions did not match. I'd updated client and server using the launcher but not gci, so maybe that was it.

I have tried the 'code critics' tool briefly. Thanks for the pointer to its usefulness in helping write code that should be more portable to Gemstone. We are becoming cosseted in the availability of helpful tools in the Smalltalk environment (and I love eCompletion)!

Thanks for the pointer to the Mock project - I think that it will come in v. useful.

T.

Ted Wrinch

Ted,

Sounds good ... every once in a while we have to tweak the gci protocol and we bump the compatibility version when that happens ... then you need to refresh your client libraries ...

Dale

----- Original Message -----
| From: "Ted Wrinch" <[hidden email]>
| To: [hidden email]
| Sent: Sunday, July 24, 2011 12:29:57 PM
| Subject: Re: [GS/SS Beta] Gemtools in Seaside 3.0
|
| Dale,
|
| It seems like I should try and develop against 3.0. You've had the
| beta out
| for some while but when I ran GemTools against it it complained that
| the
| client-server versions did not match. I'd updated client and server
| using
| the launcher but not gci, so maybe that was it.
|
| I have tried the 'code critics' tool briefly. Thanks for the pointer
| to its
| usefulness in helping write code that should be more portable to
| Gemstone.
| We are becoming cosseted in the availability of helpful tools in the
| Smalltalk environment (and I love eCompletion)!
|
| Thanks for the pointer to the Mock project - I think that it will
| come in v.
| useful.
|
| T.
|
| Ted Wrinch
|
| --
| View this message in context:
| http://forum.world.st/Gemtools-in-Seaside-3-0-tp3687549p3690773.html
| Sent from the GLASS mailing list archive at Nabble.com.
|