6.2 news: Image based persistency

Dear Aiders,

In newest 6.2 one very important feature is enabled by default and it is
important to understand the consequences of that: Image based persistency.

Image based persistency basically means a periodic snapshoot of the
image to the disk. This is the simplest but very effective persistency
solution which I'm successfully using on my production servers for
years. It is suitable for low to medium sized projects, in my experience
near 99% of all. And you have a Gemstone as a solution to "escape" for
really big projects.

Aida 6.2 has a hourly snapshot enabled by default and active immediately
after the installation. It is active on OneClick image as well.

Snapshot is implemented in method:

        AIDASite class>>imageSnapshot.

Snapshot is periodically scheduled in Aida's scheduler, see:

        AIDASite default scheduler

Another feature enabled by default is a nightly cleanup, which removes
all nonactive guest sessions and release application state on all
session. This feautre is also scheduled in above mentioned scheduler.

Best regards
Janko

--
Janko Mivšek
Aida/Web
Smalltalk Web Application Server
http://www.aidaweb.si
_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida

You forgot to mention, there is also Magma to "escape" for really big
projects.  BTW, does Aida run on GemStone?

I don't want to start a debate, but I think it's good to be
conservative about advocacy for image-persistence.  IMO, it's fine for
"small prototypes" (< 1GB model) where it's ok to lose data, but not
"medium" ones (2 - 10GB) and not any project where it's not ok to lose
data.  If the image is somehow corrupted and being saved in that state
so it won't relaunch, you would not discover this until the _next_time
you tried to restart your server, which could be months later!  Sure,
that scenario is unlikely, but certainly possible, and it would be a
total loss if it happened which, to me, is unacceptable.

What happens if the model grows faster than expected beyond memory
capacity?  How do you get your app ported over to GemStone w/
licensing, etc. and back up and running quickly?  How does one
"upgrade their image" if the model is stuck in an old image?  Does the
image save start to take longer and longer as the image grows?

Magma is simple and fast enough that, if I ever had any qualms about
any of the above issues, then I would just start with that in the
first place, and be comforted in knowing I'm prepared for growth and
cannot lose data.  No object-persistence mechanism for Squeak or Pharo
comes close to Magma in terms of the _safety_ of the domain model.

 - Chris


2011/2/22 Janko Mivšek <[hidden email]>: _______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida

No doubt you would declare an interest here, Chris, having done stirling work in the creation of Magma. :-)

I toyed with the idea of using Magma for my blog, which has been live since 2007. In the end, for my size site it was just not worth the effort. I write out all new blogs to a file, and the hourly save that comes with Aida 6.2 is perfectly adequate for that and the other sites I maintain. I run two headless images for my six sites and they are very stable. I have never yet experienced any image corruption, so am not motivated to find solutions for such absent corruption.

As for upgrading an image, I have just upgraded from Aida 6.1 to 6.2 without any model issues.

Of course, my sites are each small, but then I suspect that many Aida implementations will fall into that category, and for me, image based persistence works fine. Indeed I have always been attracted to the notion that a Smalltalk image is effectively and object-oriented database in ram, and would like to see that notion developed rather more than it has been.

Having said that, I like Magma and if I needed more traditional persistence would almost certainly go down the Magma path, except that I use VW on which Magma is not available.

Regards,

Nicholas

On Tuesday 22 Feb 2011 17:10:21 Chris Muller wrote:

> You forgot to mention, there is also Magma to "escape" for really big

> projects. BTW, does Aida run on GemStone?

>

> I don't want to start a debate, but I think it's good to be

> conservative about advocacy for image-persistence. IMO, it's fine for

> "small prototypes" (< 1GB model) where it's ok to lose data, but not

> "medium" ones (2 - 10GB) and not any project where it's not ok to lose

> data. If the image is somehow corrupted and being saved in that state

> so it won't relaunch, you would not discover this until the _next_time

> you tried to restart your server, which could be months later! Sure,

> that scenario is unlikely, but certainly possible, and it would be a

> total loss if it happened which, to me, is unacceptable.

>

> What happens if the model grows faster than expected beyond memory

> capacity? How do you get your app ported over to GemStone w/

> licensing, etc. and back up and running quickly? How does one

> "upgrade their image" if the model is stuck in an old image? Does the

> image save start to take longer and longer as the image grows?

>

> Magma is simple and fast enough that, if I ever had any qualms about

> any of the above issues, then I would just start with that in the

> first place, and be comforted in knowing I'm prepared for growth and

> cannot lose data. No object-persistence mechanism for Squeak or Pharo

> comes close to Magma in terms of the _safety_ of the domain model.

>

> - Chris

>

>

> 2011/2/22 Janko Mivšek <[hidden email]>:

> > Dear Aiders,

> >

> > In newest 6.2 one very important feature is enabled by default and it is

> > important to understand the consequences of that: Image based persistency.

> >

> > Image based persistency basically means a periodic snapshoot of the

> > image to the disk. This is the simplest but very effective persistency

> > solution which I'm successfully using on my production servers for

> > years. It is suitable for low to medium sized projects, in my experience

> > near 99% of all. And you have a Gemstone as a solution to "escape" for

> > really big projects.

> >

> > Aida 6.2 has a hourly snapshot enabled by default and active immediately

> > after the installation. It is active on OneClick image as well.

> >

> > Snapshot is implemented in method:

> >

> >        AIDASite class>>imageSnapshot.

> >

> > Snapshot is periodically scheduled in Aida's scheduler, see:

> >

> >        AIDASite default scheduler

> >

> > Another feature enabled by default is a nightly cleanup, which removes

> > all nonactive guest sessions and release application state on all

> > session. This feautre is also scheduled in above mentioned scheduler.

> >

> > Best regards

> > Janko

> >

> > --

> > Janko Mivšek

> > Aida/Web

> > Smalltalk Web Application Server

> > http://www.aidaweb.si

> > _______________________________________________

> > Aida mailing list

> > [hidden email]

> > http://lists.aidaweb.si/mailman/listinfo/aida

> >

> _______________________________________________

> Aida mailing list

> [hidden email]

> http://lists.aidaweb.si/mailman/listinfo/aida

>

--

Nicholas Moore

_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida

Chris Muller <[hidden email]> writes:

>
> Magma is simple and fast enough that, if I ever had any qualms about
> any of the above issues, then I would just start with that in the
> first place, and be comforted in knowing I'm prepared for growth and
> cannot lose data.  No object-persistence mechanism for Squeak or Pharo
> comes close to Magma in terms of the _safety_ of the domain model.
This may all be, but you still have to get your classes into the RDB and
this is quite a challenge. I don't know of any ORM which is
unproblematic. The more objects and the  more oo-ish you work the worse.

Yes there are times where data corruption could not be tolerated, but
then there is Gemstone which is a kind of persistent storage and I do
not remember having read that Gemstone stuff was corrupted  beyond any
repair. I think for most of Aida uses, the image as storage is "more
than enough"

Regards
Friedrich

--
Q-Software Solutions GmbH; Sitz: Bruchsal; Registergericht: Mannheim
Registriernummer: HRB232138; Geschaeftsfuehrer: Friedrich Dominicus
_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida

Hi Chris,

On 22. 02. 2011 17:10, Chris Muller wrote:
> You forgot to mention, there is also Magma to "escape" for really big
> projects.  BTW, does Aida run on GemStone?

Yep, just much older 5.4, we need to upgrade it. In next months I hope.

> I don't want to start a debate, but I think it's good to be
> conservative about advocacy for image-persistence.  IMO, it's fine for
> "small prototypes" (< 1GB model) where it's ok to lose data, but not
> "medium" ones (2 - 10GB) and not any project where it's not ok to lose
> data.  If the image is somehow corrupted and being saved in that state
> so it won't relaunch, you would not discover this until the _next_time
> you tried to restart your server, which could be months later!  Sure,
> that scenario is unlikely, but certainly possible, and it would be a
> total loss if it happened which, to me, is unacceptable.

First, there is a difference between VW and Squeak images and specially
VMs. While VW is rock solid, Squeak/Pharo is/was not and from here I see
the source of perception like yours.

I namely run my hosting business for 7 years now on image based
persistence, on VW image currently around 600MB in size. Without any
loss of data in whole 7 years!

With Eliot as VW VM designer now working on Squeak/Pharo VMs I'm sure we
will soon achieve similar stability and therefore suitability for
Squeak/Pharo image based persistency as well.

> What happens if the model grows faster than expected beyond memory
> capacity?  How do you get your app ported over to GemStone w/
> licensing, etc. and back up and running quickly?  How does one
> "upgrade their image" if the model is stuck in an old image?  Does the
> image save start to take longer and longer as the image grows?

Those if not all reasons goes int that 1% of special cases I wrote. At
least on VW I upgrade an image by "filling-out" all objects and file in
on the fresh one. This is called BOSS-out/in on VW, while on
Squeak/Pharo I still need to check the right way doing it.

> Magma is simple and fast enough that, if I ever had any qualms about
> any of the above issues, then I would just start with that in the
> first place, and be comforted in knowing I'm prepared for growth and
> cannot lose data.  No object-persistence mechanism for Squeak or Pharo
> comes close to Magma in terms of the _safety_ of the domain model.

I came from VW world and start with Gemstone back in 1998, while coming
to Squeak and heard about Magma about 4 years ago. This is one of
reasons why I don't know it much, the other is that Gemstone actually
provides near 100% transparency (you don't need to know any Gemstone
specifics) while on Magma you need to program the Magma presistency,
AFAIK. Which means that you became dependent on Magma, therefore your
code is not portable anymore. While on Gemstone because of so strongy
transparency it is. Magma is also working only on Squeak (Pharo?), while
Aida apps are working and directly transferable to 6 dialects now.

One big Magma advantage is open sourcelness of course and this is why I
warmly support your work now and in the future. And I'm looking forward
to support in Aida too, if someone will volunteer to add Magma support.

Best regards
Janko

--
Janko Mivšek
Aida/Web
Smalltalk Web Application Server
http://www.aidaweb.si
_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida

Janko Mivšek <[hidden email]> writes:

> Hi Chris,
>
> On 22. 02. 2011 17:10, Chris Muller wrote:
>> You forgot to mention, there is also Magma to "escape" for really big
>> projects.  BTW, does Aida run on GemStone?
>
> Yep, just much older 5.4, we need to upgrade it. In next months I
> hope.
Well I will profit from it. Any hints on how you have "ported" the
previous versions?

Regards
Friedrich

--
Q-Software Solutions GmbH; Sitz: Bruchsal; Registergericht: Mannheim
Registriernummer: HRB232138; Geschaeftsfuehrer: Friedrich Dominicus
_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida

Hi,

> I namely run my hosting business for 7 years now on image based
> persistence, on VW image currently around 600MB in size. Without any
> loss of data in whole 7 years!

Of these two prongs of risk-management; 1) the chance of something
going wrong, and 2) the penalty if it does go wrong; I already
expressed my agreement in my original post that a loss is unlikely.  I
was criticizing the other prong, which is the high penalty IF
something went wrong with the image.  I see no way to recover it other
than trying to dig the domain objects out of the binary image file.

The possibility of havoc could be wrought not only by image / VM
instability.  It could be the Aida end-users application logic.  Maybe
they deploy some new feature and it has a bug that slightly corrupts
the domain model over time.  This is a situation where a database with
transaction logs can help, since you can recover the model to any
point in time.

I am not against image-based persistency, I just was saying that I
like to be conservative about _advocating_ it, particularly in this
manner that ignores the 2nd RM prong..

> With Eliot as VW VM designer now working on Squeak/Pharo VMs I'm sure we
> will soon achieve similar stability and therefore suitability for
> Squeak/Pharo image based persistency as well.

I'm not sure what VM stability problems you refer to.  We've had
non-Cog VM's running in production for years; zero downtime due to the
VM.

I know of BOSS, I'm curious why to advocate image-persistency"
strategy over using that?

Incidentally, the Squeak equivalent is SmartRefStream or MaObjectSerializer.

> I came from VW world and start with Gemstone back in 1998, while coming
> to Squeak and heard about Magma about 4 years ago. This is one of
> reasons why I don't know it much, the other is that Gemstone actually
> provides near 100% transparency (you don't need to know any Gemstone
> specifics)
> while on Magma you need to program the Magma presistency,
> AFAIK. Which means that you became dependent on Magma, therefore your
> code is not portable anymore. While on Gemstone because of so strongy

Hmm, I'd like to know the basis of these assertions.  As far as I can
tell from my working with GemStone (since 1999), Magma has essentially
the same level of transparency.  You operate in a session, and at some
point you have to tell it to begin and commit transactions.  That's
essentially the only intrusion into "100% transparency," but this is a
_good_ intrusion, since you need to be able to define proper atomic
boundaries for altering the domain consistently.  In GemStone,
thankfully, you have the same requirement:  You have to open a Gem and
tell it to commit at some point in the code.

Friedrich was also wrong when he referred to Magma as a relational
mapper.  Rest assured, it is an ODBMS 100% with transparency
comparable to GemStone.

> transparency it is. Magma is also working only on Squeak (Pharo?), while
> Aida apps are working and directly transferable to 6 dialects now.

By "directly transferable" do you mean with zero code changes?  If so,
that's really impressive!

Regards,
  Chris
_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida

Chris Muller <[hidden email]> writes:

> I am not against image-based persistency, I just was saying that I
> like to be conservative about _advocating_ it, particularly in this
> manner that ignores the 2nd RM prong..
I for my part have understood it that way. I just can see wha Janko has
used it in Aida. It really IS a solution for "hobbyists" and not so
large projects. And well as I  understood he does make snapshots every
once in a while. So the data which are  lost are thos which were changed
after the last snapshot. Or am I missing something?

>
>> With Eliot as VW VM designer now working on Squeak/Pharo VMs I'm sure we
>> will soon achieve similar stability and therefore suitability for
>> Squeak/Pharo image based persistency as well.
>
> I'm not sure what VM stability problems you refer to.  We've had
> non-Cog VM's running in production for years; zero downtime due to the
> VM.
I has experimented with Aida in January and Pharo simply has frozen. But
maybe it still was "alive".... You are right. I have meant GLORP. I'm sorry that I messed this up.
But then please let us learn more about Magma. I had a "hasty" look over
it and yes I had it installed once...

>
> By "directly transferable" do you mean with zero code changes?  If so,
> that's really impressive!
I don't think that this is a bit asked too much. But it seems the effort
of the "ports" is not that demanding. But I may be wrong about it. I'm
just using it with Pharo

Regards
Friedrich
_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida

FDominicus wrote

...
But then please let us learn more about Magma. I had a "hasty" look over it and yes I had it installed once...

I think Magma needs more visibility because I have a feeling that once more people know about Magma, more people will want to learn more about it.  Maybe a nice website with tutorials and faq's?