Nearly limitless Image: revisited.

> it's about image size.
>
> Currently,  Pharo 1.3-13315 at my 4 Gigabyte Debian amd64 Linux machine reports
>
> memory            75,392,160 bytes
>    old            61,370,144 bytes (81.4%)
>    young        2,169,692 bytes (2.9000000000000004%)
>    used        63,539,836 bytes (84.30000000000001%)
>    free        11,852,324 bytes (15.700000000000001%)
>
> (nothing peculiar added, just the Seaside package and a few classes of my own. )
>
> So, if i understand this well I have only 11,8 megabytes to play with?
>
> To begin with: why is only 75 megabyte available in Pharo (or Squeak as well)
> when there is 4 gigabyte physical memory available on my machine??

> "I first used image persistence but the image grow too large.."
> thanks for reminding, Norbert.
>
>
> Hi all,
>
> perhaps you'll find me an idiot, I don't care.
> (i know that I am: (class SelfReflection :o) )
>
> Nevertheless, some brainstorming here.
>
> I've brought this up before, but it seems nobody is interested in this
> topic.. why?
>
> it's about image size.
>
> Currently,  Pharo 1.3-13315 at my 4 Gigabyte Debian amd64 Linux machine
> reports
>
> memory            75,392,160 bytes
>     old            61,370,144 bytes (81.4%)
>     young        2,169,692 bytes (2.9000000000000004%)
>     used        63,539,836 bytes (84.30000000000001%)
>     free        11,852,324 bytes (15.700000000000001%)
>
> (nothing peculiar added, just the Seaside package and a few classes of
> my own. )
>
> So, if i understand this well I have only 11,8 megabytes to play with?
>
> To begin with: why is only 75 megabyte available in Pharo (or Squeak as
> well)
> when there is 4 gigabyte physical memory available on my machine??
>
> Anyway, IMHO this very limited storage violates the Smalltalk-all-in-one
> image ideal,
> because with this limited storage one cannot avoid the use of external
> data stores
> like databases  (of course, exception: there always will be import an
> export of objects/data)
>
> External data storage is complex, unreliable and should be something of
> the past.
> I'd suggest: don't waste time with this. rather improve the Smalltalk
> internal data handling.
>
> I wish to have a very large image so  e.q. I can have an entire
> company's  administration
> (client and accounting data etc.) alive in it.
> I want to use Collections and descendants, not external databases.
> So, everything within the image.
>
> The beautiful (although opinions may vary) principle
> of a Smalltalk environment is Image Persistence, isn't it?
>
> IMHO this requires:
> - virtually unlimited image memory size.
> -no external databases like e.g. Mongo or DB2 whatever.
> -no persistent external storage at all.
>
> -You should see the (currently still) faster database performance as a
> challenge
> to improve the Collection and other related data handling classes
>
> -You would have to completely restructure the architecture.
> or just extend the address width: 64 bits?
>
> what about an image of virtually unlimited size > gigabytes so
>
> (for now, if it goes beyond the physical memory size (say 32 GB?)
> one could use a VM that works with virtual storage based VM
> Virtual storage was first in use successfully with IBM mainframes
> in the 1970s (and still is) )
>
> on the other hand, I estimate holographic? physical memory
> going in Terabytes is about to be available in 5-10 years from now.
> blindingly fast, compared with today's standards.
>
> So, one has about 5 years to come up with such a system :o)
> if one does, Smalltalk is ahead of everything then..
>
> Don't laugh. One didn't even dream of 4 gigabytes of memory
> during the Commodore 64 era. 1982-1994.
>
> In short: what would have to be changed to enable this?
>
> Some thoughts and speculations?
> Anyone for tennis?
>
> Thanks
> Ted
>
>
>
>
>
>

> Hi Ted,
>
> I also support the image as a sole persistence idea and I'm actually
> using it for years on VisualWorks.
>
> For Pharo there seems everyone is avoiding this idea, mostly because of
> image corruption fear. Well, we need to improve the robustness of Pharo
> image and VM then this fear will vanish. It is obviously doable, if VW
> is reliable enough, why not once Pharo?
>

> Incidentally, we had a little chat with Marcus yesterday about that.
>
> No, i don't think it is feasible to use single image to store everything.
> It is convenient, cheap and of course it is way better than dealing
> with communicating with external DB/servers whatever.
>
> But there's one thing you should know already: the days of vertical
> growth is over.
>
> Running a service (under VM or not) on a single machine is asking for troubles:
>  - limits on load
>  - susceptible to power outage and other reliability problems
>  etc
>
> Also, think that the amount of data you need to process correlates
> with CPU horsepower available.
> Which means that yes, you can run a huge image with 64Gb data in it..
> but that means that responsiveness
> of your service will quite often fall beyond any usability limits.
>
> If we look in terms of VM and pick only one thing - garbage collection,
> you will see that there is certain limits beyond which a performance
> will drop too much, so you naturally
> will start thinking about ways to split data to separate chunks and
> run them on different machines/VMs.
>
> It is because GC's mark algorithm is O(n) bound, when n is total
> number of references between objects,
> and GC's scavenge algorithm is at best O(n) bound where n is total
> number of objects in object memory,
> and at worst is where n is total memory used by objects.
> No matter how you turn it, i just wanted to indicate that time to run
> GC is in linear dependency from the amount of data.
>
> Yes, we might invest a lot of effort in making GC more clever, more
> complex and more robust.. but no matter what you do,
> you cannot change the above facts. It means, that any improvements
> will be about diminishing returns, but won't change the picture
> radically.
>
> That means that sooner or later you will have to deal with it: a
> problem of splitting data on multiple independent chunks,
> and making your service to run on multiple machines , in order to use
> more CPU power, more memory and be more reliable etc.
> At this point, your main dilemma is to invent a fast and robust
> interfaces to communicate between images or between image(s)/ database
> etc.
>
> We should concentrate on things which dealing with inter-image
> communication and image-database communication,
> because it is the only way to ensure that we will answer upcoming
> future problems. Relying on using a single huge image is way to
> nowhere.
>