Simple Image Based Persistence in Squeak

In the case you didn' yet read Ramon Leon post on his blog:
http://onsmalltalk.com/programming/smalltalk/simple-image-based-persistence-in-squeak/

 From the post:

     "It’s a mistake to assume every application needs a relational
database, or even a proper database at all. It’s all too common for
developers to wield a relational database as a golden hammer that solves
all problems, but for many applications they introduce a level of
complexity that can making development feel like wading through a pond
full of molasses where you spend much of your time trying to keep the
database schema and the object schema in sync. It kills both
productivity and fun, and god dammit, programming should be fun!

...
It shouldn’t come as a surprise that one can take a single step up the
complexity ladder, and build simple applications that aren’t ACID
compliant, don’t support transactions, and manage to successfully run
most small businesses BETTER than Word and Excel while purposely not
taking a further step and moving up to a real database which would
introduce a level of complexity that might blow the budget and make the
app infeasible.

No object relational mapping layer (not even Rails and ActiveRecord) can
match the simplicity, performance, and speed of development one can get
just using plain old objects that are kept in memory all the time"




Image as a database is my way of running my web apps for four years
already, and I make my living of them, so I can just agree and confirm
Ramon here!

Janko


--
Janko Mivšek
AIDA/Web
Smalltalk Web Application Server
http://www.aidaweb.si

2008/1/15, Janko Mivšek <[hidden email]>: We regularly get flamed big time for using the image as a database in SqS.

Cheers
Philippe

Philippe Marschall wrote:

>> In the case you didn' yet read Ramon Leon post on his blog:
>> http://onsmalltalk.com/programming/smalltalk/simple-image-based-persistence-in-squeak/

>> No object relational mapping layer (not even Rails and ActiveRecord) can
>> match the simplicity, performance, and speed of development one can get
>> just using plain old objects that are kept in memory all the time"

>> Image as a database is my way of running my web apps for four years
>> already, and I make my living of them, so I can just agree and confirm
>> Ramon here!

> We regularly get flamed big time for using the image as a database in SqS.

Well, a database lobby (let me name them such) succeeded to plant deeply
in our minds a feeling of fear of anything else than a relational
database. A fear of loosing your valuable data, that is. Fear, FUD,
always a strong weapon...

Janko



--
Janko Mivšek
AIDA/Web
Smalltalk Web Application Server
http://www.aidaweb.si

On Tue, 15 Jan 2008 13:29:21 +0100, Janko Mivšek <[hidden email]>
wrote:

> Well, a database lobby (let me name them such) succeeded to plant deeply
> in our minds a feeling of fear of anything else than a relational
> database. A fear of loosing your valuable data, that is. Fear, FUD,
> always a strong weapon...

Personally, I think you'd be better off using Magma rather than the image.
I've played a bit with it, and have big plans for the future with it.

Have you tried it, and decided it wasn't usable for your application?

Later,
Jon

--------------------------------------------------------------
   Jon Hylands      [hidden email]      http://www.huv.com/jon

  Project: Micro Raptor (Small Biped Velociraptor Robot)
           http://www.huv.com/blog

Jon Hylands wrote:

>> Well, a database lobby (let me name them such) succeeded to plant deeply
>> in our minds a feeling of fear of anything else than a relational
>> database. A fear of loosing your valuable data, that is. Fear, FUD,
>> always a strong weapon...
>
> Personally, I think you'd be better off using Magma rather than the image.
> I've played a bit with it, and have big plans for the future with it.
>
> Have you tried it, and decided it wasn't usable for your application?

I admit I didn't look at Magma yet.

My comment was directed to relational database vendors, who control our
database mentality and perceptions, and consequently a fear against
databases like Magma (and Gemstone etc) too. But definitively against
image as a database.

Why I like a concept of image as a database? Because it simply works,
and that is my concrete experience. Also a fear of losing data is mostly
baseless - a chance of losing data is very small, again from experience.
So small that it is close to a chance of loosing of data on relational
databases, I would say. That is a strong wording I know, but let we at
least start thinking about this probability.

Janko

--
Janko Mivšek
AIDA/Web
Smalltalk Web Application Server
http://www.aidaweb.si

On Tue, 15 Jan 2008 14:02:14 +0100, Janko Mivšek <[hidden email]>
wrote:

> Why I like a concept of image as a database? Because it simply works,
> and that is my concrete experience. Also a fear of losing data is mostly
> baseless - a chance of losing data is very small, again from experience.
> So small that it is close to a chance of loosing of data on relational
> databases, I would say. That is a strong wording I know, but let we at
> least start thinking about this probability.

One issue I have with image-based is it is all or nothing. I want to be
able to make small incremental changes, and persist those changes.

The other issue for me, and the real deciding factor, is you can't fit the
amount of data I need persisted in an image. I expect its going to be
measured in hundreds of GB, and the image is just going to be a small cache
of recently used/accessed data.

Later,
Jon

--------------------------------------------------------------
   Jon Hylands      [hidden email]      http://www.huv.com/jon

  Project: Micro Raptor (Small Biped Velociraptor Robot)
           http://www.huv.com/blog

We have likewise used "image as a database" for bountifulbaby.com for years.

In the beginning we used PostgreSQL, but eventually yanked it.  We've
never had a problem with just using the image.  However, we have a
couple of other processes in place that help reduce the risk:

1. the production image is saved and a copy is brought over to the
development system for all developmental coding updates.  This results
in a production image save about 3 to 5 times per week.  The production
image is saved into it's own separate archive area on the development
system(s), so we can revert to the image that existed at virtually any
point in time if we choose to, simply by grabbing any old production
image that we wish to.

2. Several key domain objects use FileStream>>fileOutClass:andObject: to
hierarchically write their contents to disk.  This literally spools out
all of the important domain object's data to disk, where they can be
read back in by any other production and/or development image.  This
"save" snapshot happens automatically once a day, in the early hours of
the morning.  Any image that reads this data back in from disk (via
FileStream>>fileInObjectAndCode) will be restored to the state of the
image at the time of the snapshot.

Consequently, if the production image ever hoses up, we at most risk the
loss of a few hours data.  Furthermore, our normal order processing
produces a paper trail from which we can manually update to restore even
that, if we needed to.

But we've never needed to.

The "image as database" approach has been very reliable for us.  And a
whole lot easier than any other approach.

Nevin

Jon Hylands wrote:
Fantastic!

Is Magma currently ready for 10s or 100s of GB of data?
Or maybe soon?

Outside of moving from the all or nothing of the image, a what point
(size) does the image start losing advantages as a data store?

In that vein, is there a point where an image is larger than desired
from a caching point of view?

I know there may not be any hard data, or even honest opinions regarding
such. But if you have any opinions or data for some possible best
practices regarding such, it would be greatly appreciated.

Thanks.

Jimmie

>>>>> "Jimmie" == Jimmie Houchin <[hidden email]> writes:

Jimmie> Is Magma currently ready for 10s or 100s of GB of data?
Jimmie> Or maybe soon?

There's always the Gemstone solution.  Free for small e-commerce,
and scales well once you get your Angel partners. :)

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<[hidden email]> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

On Tue, 15 Jan 2008 12:38:15 -0600, Jimmie Houchin <[hidden email]>
wrote:

> Is Magma currently ready for 10s or 100s of GB of data?
> Or maybe soon?

I have no idea :-) I know that they guy who wrote it is pretty smart, and
if he has a "customer" with a large application, hopefully we can work
together to make it happen.

> Outside of moving from the all or nothing of the image, a what point
> (size) does the image start losing advantages as a data store?

I have no idea where that point is. I know that my application will be way
past that point, so I haven't really worried/though much about it.

> In that vein, is there a point where an image is larger than desired
> from a caching point of view?

Well, presumably when you run out of RAM on loading it...

Later,
Jon

--------------------------------------------------------------
   Jon Hylands      [hidden email]      http://www.huv.com/jon

  Project: Micro Raptor (Small Biped Velociraptor Robot)
           http://www.huv.com/blog

On Tue, 15 Jan 2008 01:57:45 -0800, Janko Mivšek <[hidden email]>  
wrote:

How do you handle reporting in that situation? Particularly reporting  
using 3rd party tools?

        ===Blake===

I agree with Ramon and Janko.

My stop condition to start using Magma was that.......the image ever
worked fine to me!

I admit I'm not managing TBs (nor GBs) of data in my current
production systems but, as Ramon comment on his blog, the image works
ok to a wide majority of projects (at least in my own experience
developing systems to small and medium companies).

Ever I found (with few lines of code) the way of ensure integrity,
backups, changes, etc.

Just my 0,02.

Cheers.
gsa.


2008/1/15, Blake <[hidden email]>:

Blake wrote:
> How do you handle reporting in that situation? Particularly  
> reporting using 3rd party tools?

I'm happy with a workspace, a debugger and a browser. Just write some
custom code to walk the objects. Haven't found a 3rd party tool with
enough power & usability yet.

Stephan

[hidden email] wrote: Yes, my sentiment exactly.

Then put the "reporting" code in some methods somewhere.

Then wrap it in Seaside.

Nevin

On Tue, 15 Jan 2008 17:49:18 -0800, <[hidden email]> wrote:

> Blake wrote:
>> How do you handle reporting in that situation? Particularly reporting  
>> using 3rd party tools?
>
> I'm happy with a workspace, a debugger and a browser. Just write some
> custom code to walk the objects. Haven't found a 3rd party tool with
> enough power & usability yet.
>
> Stephan

But then do what with it? It's not like reporting is (necessarily) a  
trivial thing.

On Tue, Jan 15, 2008 at 11:13:58AM -0800, Blake wrote:
> On Tue, 15 Jan 2008 01:57:45 -0800, Janko Miv?ek <[hidden email]>  
> wrote:
>
> >Image as a database is my way of running my web apps for four years  
> >already, and I make my living of them, so I can just agree and confirm  
> >Ramon here!
>
> How do you handle reporting in that situation? Particularly reporting  
> using 3rd party tools?

Pump the interesting data into a relational database and let people query
it there.

Dave

On Tue, 15 Jan 2008 19:25:24 -0800, David T. Lewis <[hidden email]>  
wrote:

> Pump the interesting data into a relational database and let people query
> it there.

I thought of this. Anyone actually doing it? It'd be slicker to have a  
package that allowed Squeak to act as an ODBC engine. (Or slicker still to  
have a good reporting tool. Also, ponies!)

Confine Squeak objects within walls of a relational db is possible
using the ODBC package from Diego GD.



2008/1/16, Blake <[hidden email]>:

Jon Hylands ha scritto:
Not to mention that if you're going to have many concurrent
access/modifications to the same data stored in the image, things will
get messy very, very quickly.

        Giovanni