Documentation about Gemstone internal info :)

Hi Mariano,

Last year I did a half-day tutorial in Buenos Aires that covered many of these questions. You can watch the videos by following links at http://programminggems.wordpress.com/2010/02/05/scaling-objects-videos/. In particular, see the second video on "Object format and pointers" that discusses exactly these issues. Other comments follow...

On Jan 6, 2011, at 3:34 AM, Mariano Martinez Peck wrote:

> Hi. I am just curious about some details of the Gemstone implementation. Please forgive me if the questions are not appropriate. I read the Programmer Guide but I didn't find the answers.
>
> So, basically I would like to know some details of the VM and Object Memory, for example:
>
> 1) I know that Gemstone uses ObjectTables. With them, I can think different pros:
>    - a fast #become since it is just changing one pointer and not a full scan
>    - when using proxies/stubs you probably may need a #become just after
>    - gemstone uses stubs a lot (I guess)
>    - Having an OT may help when trying to implement kind of virtual memory/swapping/pagging system (like Gemstone).
>    - split the OT in "regions" and place them there?
>
> Of course not everything is pink, and you can have a big impact in performance since you have for example the cost of accessing once more (the indirection of the OT), and problable much more things.
>
> So, I wonder, which are the reasons of Gemstone behind this decision of using ObjectTable?

I believe that there are two main reasons. The first is to support multiple database views (discussed below in #2). The second is to facilitate repository-wide garbage-collection (GC). With an in-memory GC in a traditional Smalltalk VM (i.e., Squeak), when the VM moves an object it has to scan memory and fix all the pointers. With a disk-based object space of hundreds of GB, this could take hours and would be impractical.

> 2) There is one ObjectTable per Gem or per Stone?

The is actually one (virtual) Object Table (OT) per committed transaction! (A number of optimizations exist to reduce the space taken by each additional OT.)

This is necessary because two views of the database might need to see different versions of the same object. Since the object is the identical object in each view, it must have the same identifier, but if it was changed in a transaction, it must have different data. If it has different data, it must have different storage.

An OT needs to be preserved as long as there is a transaction view relying on the information in the OT. An old OT can only be released when there is no longer a session relying on its view (or an older view). This is the basis of the notorious "Commit Record Backlog" (CRB) problem that can plague a GemStone system in which a logged-in session sits on one view for a long time while other sessions are doing many commits.

> 3) Memory addresses.  in LOOM for example, they used different memory address for disk (32) and ram (16). Now, I guess you can do the same with 32/64 or directly use 64bits in both. how do you do?  if it is not the same, how do you map from one to the other?

Each Gem (VM) that references an object copies the object from shared memory (and into shared memory from disk if needed) into the local VM memory. Each Gem maintains a map of an object's ID (entry in the OT) to its local, in-memory address. When an in-memory object references another object, the Gem checks first to see if the object is already loaded and if so, it updates the reference from the ID to the pointer so subsequent lookups do not pay the cost of the indirection. This keeps in-memory references inexpensive. When a modified object is committed, its instance variables are converted from in-memory pointers back to Object Table Identifiers before being written to the shared memory (and then to disk) where other Gems can see the object's instance variables.

An Object Table Identifier (OOP) always has 001 as the low-order three bits, while in-memory pointers always have 000 as the low-order three bits. (Other values for the low-order three bits indicate special or immediate objects such as instances of SmallInteger, Character, Boolean, UndefinedObject, and SmallDouble).

> 4) I guess gemstone uses stubs a lot when moving objects from primary memory to secondary memory. Now, you don't create a stub instance (with object header and blah) because of the size, so you can mark them directly in the address/reference. So....you can mark an address as stub?  I guess yes. Then, how do you know where on disk is that stub for example?  you can encode such info also in the same reference ? (if you use 64bits in both I guess that yes)

I'm not sure what you mean by 'stubs' in this context. Every object takes up space in-memory (if a Gem is actively referencing it) and on disk. If a Gem gets constrained on space, it can remove unmodified persistent objects from RAM by replacing in-memory references with the OOP.

> 5) How do you trace object usage? I mean the shared object cache can move objects from disk to primary memory and the other way around. How it selects which objects to move?  how do you know which objects has been used frequently ?

Every object is associated with a "page" and the Shared Page Cache keeps a record of the requests that Gems make for a page. Recently-referenced pages stay in the cache while not-recently referenced pages are eligible for being flushed from the cache.

> Thanks in advance,
>
> Mariano
>
> ps: don't worry, I won't create my own gemstone hahahahha I just want to learn :)
> ahh if the info is private I can just undertand that or you can send me a private email.

I hope this helps,

On Jan 11, 2011, at 2:36 AM, Mariano Martinez Peck wrote:

On Thu, Jan 6, 2011 at 3:12 PM, James Foster <[hidden email]<mailto:[hidden email]>> wrote:
Hi Mariano,

Last year I did a half-day tutorial in Buenos Aires that covered many of these questions. You can watch the videos by following links at http://programminggems.wordpress.com/2010/02/05/scaling-objects-videos/. In particular, see the second video on "Object format and pointers" that discusses exactly these issues. Other comments follow...

Hi James. Sorry for the late answer. But I had several videos to watch ;)
I wonder which talks were at the same time that I didn't go to your Gemstone lesson. I am regrated hahahaha...fortunatly there are videos :)

Actually, the videos were from the 2009 conference. I still haven't posted the videos from the 2010 conference.

  > 2) There is one ObjectTable per Gem or per Stone?

The is actually one (virtual) Object Table (OT) per committed transaction! (A number of optimizations exist to reduce the space taken by each additional OT.)

WOW. I wonder how you can do those optimizations and not blowup in terabytes of memory used by OTs hahhahahaha

Of the many impressive things about GemStone, this is indeed one of the more striking ones!

> 5) How do you trace object usage? I mean the shared object cache can move objects from disk to primary memory and the other way around. How it selects which objects to move?  how do you know which objects has been used frequently ?

Every object is associated with a "page" and the Shared Page Cache keeps a record of the requests that Gems make for a page. Recently-referenced pages stay in the cache while not-recently referenced pages are eligible for being flushed from the cache.

Clear. Now, I wonder HOW an object is associated with a page. Because of GC and maybe because of object usage (when an object is used, maybe its instance varaibles are used too), it would make sense to group an object and its instVar in the same page. But at the same time, you have pages that you need to complete. So...is there a strategy in which page to choose for a certain object?

By default, a modified object is simply placed on the next available (otherwise empty) page. A GC will, likewise, simply move live objects from partially-filled pages to a new page in attempt to increase the packing. To address the issue of having a objects used together close together, GemStone provides "Clustering" in which you can specify a virtual "Cluster Bucket" and associate an object with a bucket. In this case, the object will be written to a page containing only objects also associated with that bucket. The GC process preserves this property by identifying a page with a bucket and when a page is partially filled, live objects will be moved to a new page with other objects from the same bucket. (Details are in the documentation and in the comments to the #'cluster' method and friends.)

Last question, totally offtopic, (sorry, I don't have a gemstone handly), for hashed collections, the #identityHash   is the Object ID ?

I'm not sure how it is implemented at the C level (in Smalltalk it just calls a primitive), but I'll ask around (or maybe someone else can jump in).

> Thanks in advance,
>
> Mariano
>
> ps: don't worry, I won't create my own gemstone hahahahha I just want to learn :)
> ahh if the info is private I can just undertand that or you can send me a private email.

I hope this helps,

yes, a lot!

Thanks James for all your answers.

Glad to help! GemStone is cool technology and I enjoy having it better understood.

On Jan 11, 2011, at 12:29 PM, Mariano Martinez Peck wrote:

> Thanks James, this [discussion of clustering] was interesting. I will also check in the documentation. What it is important here is that I guess that the decision is up to the user, to say that a certain object should be associated to a certain bucket.

Yes, it is up to the programmer to determine which objects go together, and it depends completely on the object model. For example, consider a Customer object with instance variables for address fields such as city. If you had a unique String for each Customer, then you would probably want the city (a String) clustered with the Customer. On the other hand, if you had a canonical list of City objects (to avoid having 4 million copies of the string 'Buenos Aires', then you would _not_ want the City object clustered with the Customer. Instead, you would cluster all the City objects together so that if one was present the others would likely be available as well.