Memory Consumption for Complex Object Graphs

>
> I was wondering if someone could explain how memory is allocated and consumed
> in Gemstone for complex object graphs. I am trying to decide if I should use
> GLASS for an upcoming project, but I have some concerns about if it is
> appropriate and how I can solve some architectural issues.
>
> Namely, I have very deep or interconnected object graphs. I am concerned
> that when I bring these objects back, even parts of the graph I am not
> currently using will be dereferenced or at least retrieved from the Stone in
> some situations.
>
> For example, if I simplify - Customer ->  Orders ->  Line Items. If I have an
> object in Smalltalk that follows this structure and I retrieve the customer
> but do not send any messages to the orders or line items collection, will
> they be allocated in memory or retrieved? I guess what I am getting at is
> does Gemstone allocate memory actively or lazily? If the memory is always
> allocated, I see this as a problem in my architecture as there can be very
> deep object graphs, recursive relationships, and so on. Imagine an actual
> graph for example (Vertexes and Edges) - I don't want the entire graph
> retrieved by accident and allocated into memory as it could include gigs of
> objects.
>
> Is there a good way to solve this potentially without sacrificing
> transparent persistence? Maybe some kind of light weight container or manual
> lazy approach? Up until now, I've been building simple models, so I haven't
> been as concerned. Thanks.

>
> Thanks Dale!
>
> My main concern was graph traversals. I planned to use indexes and streams
> as appropriate. Have any GemStone customers every used GemStone as a
> database with some graph like semantics or graph traversals? My thought was
> perhaps using a stream or some sort of other data structure to ensure that
> as I traverse, unneeded parts of the result set drop out of memory
> immediately. I have some concern here about overloading the SPC if I don't
> do traversals correctly.
>
> I hope I am not using a bad word for you guys here, but I noticed that
> Objectivity was building a product on top of their offering called
> InfiniteGraph, so I got the idea that I could build something similar to
> suit my needs on top of GLASS. They only support Java for now, and I am not
> very happy with the Smalltalk support in their core product.
>
> I'd be happy to discuss some of those ideas further with you if interested
> over private channels. Thanks again for your quick feedback, always
> appreciated along with your great work.

>
> Thanks Dale. That sounds reasonable.  What would be the best way to go about
> checking during a traversal? Is it reasonable to check the amount of room in
> the SPC, or are any of those operations expensive? I can imagine this
> strategy may not make a lot of sense as well if more than one machine is
> adding to the memory at the same time. Same thing goes for the VM if more
> than on thread is allocating memory.
>
> I would prefer to avoid exceptions, but if they do happen, I am assuming
> just catch the out of memory exception and go from there.  I suppose I could
> as a rule of thumb just commit under some criteria, primitive like every x
> iterations or more advanced by doing it some other way. I'd of course try to
> just use streams and indexes first, but I don't think this is always 100%
> possible, especially since we have some ad-hoc field creation in the mix.
> Thanks.