STS repository D5/D6 incompatablity

Is there a nasty incompatibility lurking in trying to connect D6 and D5
to the same STS repository?

I've been using STS with D5 for a number of years. When I connected D6
to the same repository, things seemed to be working okay. I could browse
package editions, compare them, etc. But, after importing a single
package, I discovered that omnibase.odl had blown up from 1,789 KB to
1,376,837 KB!

I managed to dig into it and find out what the problem is.
ODBClassInfoFile in the Omnibase that came with STS for D5 initialized
keyLength to 50, while in D6 it's initialized to 128. When it looks for
a particular class in the repository, the BTree key probes end up
looking at garbage, so it decides to insert (StsMethodEdition in my
case) off in never-never land. And the position dutifully extends the
file to accommodate.

A cursory check after hacking ODBClassInfoFile>>initialize to set
keyLength=50 seems to allow connecting D6 to the old repository without
the aforementioned problem. But...

A few questions about all this.

Anyone else trying to connect D6 to an old STS repository?
Is it safe to run with keyLength=50 in D6?
Are there any other incompatibilities?
Is there a way to "migrate" an old repository to the newer format? (Of
course the answer is yes, but does anyone have the script that would do it?)
Does Omnibase not store its parameters or a version number somewhere in
the files? Such that it could use the existing format when connecting?
Or at least report that there's a problem with the format?

thanks,
-Bill

--
Bill Dargel            [hidden email]
Shoshana Technologies
100 West Joy Road, Ann Arbor, MI 48105  USA

Bill,

> Anyone else trying to connect D6 to an old STS repository?

I had problems with this during the D6 beta and have decided not even to
attempt it.  In part it may be a question of what version of OmniBase you are
migrating from.


> Is it safe to run with keyLength=50 in D6?
> Are there any other incompatibilities?

Worth checking that ODBDeserializer class>>init builds the same array in both
versions (that was the first problem I hit when I attempted this).


> Is there a way to "migrate" an old repository to the newer format? (Of
> course the answer is yes, but does anyone have the script that would do
> it?) Does Omnibase not store its parameters or a version number somewhere
> in
> the files? Such that it could use the existing format when connecting?
> Or at least report that there's a problem with the format?

I would like some way of copying the state of a D5 repository into new D6 one
too.

What I am /not/ going to try again is connecting a D6 Dolphin to (even a copy
of) a D5 repository.  I found that leads to corruption of the repository at a
later, apparently indeterminate, time.  I did supply David with a reproducible
case but as far as I know the problem is still unresolved.  (That D5 repository
is still working happily, btw, nearly a year later)   Maybe I was seeing a
different manifestation of the same keyLength problem -- I can't check now :-(

Ideally I would like a way to "resynch" two repositories so that I can continue
D5 and D6 development and have changes in one replicated into the other.  I
suspect that's not feasible, and so the only option is to snapshot the D5
repository and use that (somehow) as the base for a new D6 one.

    -- chris

Thanks for your input Chris.

Chris Uppal wrote:
> I had problems with this during the D6 beta and have decided not even to
> attempt it.  In part it may be a question of what version of OmniBase you are
> migrating from.

Indeed. I had been using 'OmniBase 2.0 STS Edition' circa 06/2002.
Frankly, I'm somewhat shocked that OmniBase doesn't appear to do any
format/version checking when it connects to a repository.

> Worth checking that ODBDeserializer class>>init builds the same array in both
> versions (that was the first problem I hit when I attempted this).

In checking it out, I see that the new one is a superset of the old.  So
the old version would probably have trouble deserializing things written
by the new version. (If any of the newly added classes appeared). I'm
guessing that the new shouldn't have any trouble deserializing the old.
Seems that gives two options. Use the old ODBDeserializer setup in D6.
Or treat the whole thing as a one-way migration of the repository.

> What I am /not/ going to try again is connecting a D6 Dolphin to (even a copy
> of) a D5 repository.  I found that leads to corruption of the repository at a
> later, apparently indeterminate, time.  I did supply David with a reproducible
> case but as far as I know the problem is still unresolved.  (That D5 repository
> is still working happily, btw, nearly a year later)   Maybe I was seeing a
> different manifestation of the same keyLength problem -- I can't check now :-(

That's troubling. Do you recall what sort of corruption you saw?

> Ideally I would like a way to "resynch" two repositories so that I can continue
> D5 and D6 development and have changes in one replicated into the other.  I
> suspect that's not feasible, and so the only option is to snapshot the D5
> repository and use that (somehow) as the base for a new D6 one.

That would work for me too. :-)

Actually, I've only had a chance to dabble in D6, still doing all my
work in D5. So when I wanted to take a look at OmniBase, I went ahead
and loaded the latest (2.1.69) from http://www.gorisek.com into my D5
image, along with STS 3.2.1. I rather naively (in hindsight) connected
to my working repository. It didn't complain. And for the most part
seemed to be working okay. I did run into a few issues in STS, mainly
revolving around imported packages not having developer/timestamp
attributes, which the new change browsers wanted to display in their
lists. Those were fairly easily fixed, and I've since seen that those
STS methods in D6 already have the same, or similar, corrections.

But then the keyLength problem showed itself.

*If* that's the only issue, then I think I have a solution that would
work for me. Being able to continue to work in D5 with my current
repository (using the latest OmniBase, with STS keyLength 50), and also
being able to connect to it from D6 (with keyLength 50), since they'd
both be speaking the same format.

But testing to see if that's the only issue is a catch-22. I don't want
to put a bunch of my work into a repository if it's going to die. But I
can't really feel confident about it (lacking any other information)
without actually using it for a while.

So if there were any official ways to cross-connect, migrate and/or
synchronize repositories, I'd be happy.

--
Bill Dargel            [hidden email]
Shoshana Technologies
100 West Joy Road, Ann Arbor, MI 48105  USA

Bill,

> Frankly, I'm somewhat shocked that OmniBase doesn't appear to do any
> format/version checking when it connects to a repository.

Yes.


> > What I am /not/ going to try again is connecting a D6 Dolphin to (even
> > a copy of) a D5 repository.  I found that leads to corruption of the
> > repository at a later, apparently indeterminate, time.[...]
>
> That's troubling. Do you recall what sort of corruption you saw?

>From my original bug report:

==========
If I point D6 STS at a copy of my D5 repository, then everything starts OK.  It
can read the data just fine.  However, if I start versioning things then sooner
or later I end up with a corrupt repository.
[...]
I start with the copy and connect to it.  Everything's OK.  Now I try to
version all the existing code in the D6 image.  The results are
unpredictable -- sometimes it fails almost immediately, sometimes it takes a
long time before it fails, sometimes it seems to work OK.  If it has seemed to
work OK, then the STS browsing tools also seem to work OK.  Now I disconnect
and reconnect.  From here on nothing works.  It seems that the db connection's
ODBClassManager's list of #classes contains ODBClassDescription-s that contain
trash data. For instance the entry for 'StsPackageEdition' thinks that its
#instVarNames is:
    #( 'timestamp' 'IID' '' '' '' ''  ... '' )
and so its #varOrder is:
    #(1 18 18 18 18 ... 18)
which is nonsense.
==========

When I get a chance, I'll try to see what effect your change has on this.  But,
as you say, even if it does seem to cure that problem, that's only of academic
interest without some reason to believe that it fixes /all/ the problems of
this kind.

    -- chris

Chris Uppal wrote:
> [...]  It seems that the db connection's
> ODBClassManager's list of #classes contains ODBClassDescription-s that contain
> trash data. For instance the entry for 'StsPackageEdition' thinks that its
> #instVarNames is: [...]

Haven't really thought through the details, but this does sound like
something that might be caused by using the wrong keyLength on the
existing file.

> When I get a chance, I'll try to see what effect your change has on this.  But,
> as you say, even if it does seem to cure that problem, that's only of academic
> interest without some reason to believe that it fixes /all/ the problems of
> this kind.

Yes. I'm hoping that David will weigh in at some point.

--
Bill Dargel            [hidden email]
Shoshana Technologies
100 West Joy Road, Ann Arbor, MI 48105  USA

Bill,

I wasn't aware of this problem/bug but I know where it comes from.

In D5 and all previous versions STS came with special (trial) version of
OmniBase, and since I was always using the full version of OmniBase
which now comes with X6 I simple didn't have this problem.

The easiest thing to do if you want to work with the same repository
from X6 and D5 you have to do the following:

1. Convert class dictionary to 128 bytes long keys. You can do this by
subclassing ODBClassInfo with e.g. ODBClassInfo128, change the
initialize method to 128 bytes and then evaluate something like (not
tested!):

| f1 f2 a|
f1 := ODBClassInfo openOn: 'c:\db\omnibase.odl'.
f2 := ODBClassInfo createOn: 'c:\db\omnibase128.odl'.
a := f1 getFirst.
[a isNil] whileFalse: [
        f2 goTo: a key put: a value.
        a := f1 getNext].
f1 close.
f2 close.
File delete: 'omnibase.odl'.
File rename: 'omnibase128.odl' to: 'omnibase.odl'.


2. After the conversion is done, change the method
ODBClassInfo>>#initialize to use keys of size 128 bytes.


Best regards,

David Gorisek


Bill Dargel wrote:

David Gorisek wrote:

> I wasn't aware of this problem/bug but I know where it comes from.

Why didn't you look into the problems I had during D6 beta ?  Did you even look
at the reproducible test case I created for you ?


> In D5 and all previous versions STS came with special (trial) version of
> OmniBase, and since I was always using the full version of OmniBase
> which now comes with X6 I simple didn't have this problem.

My D5 installation uses the full OmniBase -- I.e. I /paid/ for it (at the same
time as STS).  It uses the 50 byte keys.


What in Hell do you mean "not tested" -- you are talking about our /data/ here.
You've made some kind of serious error which is endangering the integrity of
other people's repositories (people, I remind you, who have /paid/ for your
software), and you don't even test your "fix" ?

I'm bloody furious.

    -- chris

Chris,

sorry you lost your data when switching to X6. The key length of classes
dictionary in OmniBase was changed for VisualWorks and then synchronized
across dialects. Probably this was done somewhere in the lifetime of D5
but frankly I don't know when. Looking at the code I see that the b-tree
is storing its key length but its subclass used for storing class info
is overriding the default behavior. Bad mistake, but what do you expect
me to do now? Even I wonder how come I didn't notice any problems with
my repository database while I was working with D5 (and my repository
goes very very far back to 2002). The interesting thing is that this
error can never show up if on the start of every development session you
  load the code from the repository and then start developing. By
loading objects, the database classes dictionary is initialized via its
ID table and so b-tree is never used. If STS objects don't change their
definition then the database will never query/update the b-tree and the
problem will go unnoticed. The same goes for using D5 repository with
X6. If you first load a piece of code which will instantiate instances
of all possible classes, then you can later safely update D5 repository
without corrupting it. The database corruption can only happen if you
are storing an object for which the schema definition is not already
cached in the OmniBase class manager. Then it will go and query the
b-tree file, look for an existing definition (of course it wont find
any) and it will then corrupt the b-tree together with stored definitions.

Now, I understand your reaction to this problem, but on the other hand
you have to understand that the STS "budget" does not allow me to
dedicate much time to STS related work due to other on-going projects
which are in fact making development of things like STS and OmniBase
even possible. We can support and further develop the current version,
but support for D5 is now way over. OmniBase and STS prices were
symbolic and sales were low so you can calculate for yourself what that
means. So please don't expect a hand if we are offering only the finger.
The alternatives we have are either this level of support or nothing.

That is all I can say regarding this topic and wont go into further
discussion because everything has already been told on the subject of
Smalltalk development tools business.

Best regards,

David Gorisek



Chris Uppal wrote:

David Gorisek wrote:
> The key length of classes
> dictionary in OmniBase was changed for VisualWorks and then synchronized
> across dialects.

You definitely missed a trick at that point. Should have changed
ODBClassInfoFile>>descriptionString to reflect the incompatible format
change. BTW, it still has:
        Class Info File
        OmniBase 1.0, David Gorisek (c) 1998
Since the headers on all files are compared when opening a repository,
OmniBase could have reported: "File has bad header".

[Though it would also be good if STSManager>>install passed on that
error description to the user. As it stands, the install only says
"Repository does not exist. Do you want to create new repository?". No
matter what the error was in trying to open the repository. (Bad header,
a single file read-only, some transient error, etc.). Haven't tried it,
but it looks like a moment's lapse in thought, and it becomes a self
fulfilling prophesy of the repository "Not Existing", as it gets
overwritten by a newly created one.]

> Probably this was done somewhere in the lifetime of D5
> but frankly I don't know when.

But isn't every change to the method stored in your repository? :-/

> Looking at the code I see that the b-tree
> is storing its key length but its subclass used for storing class info
> is overriding the default behavior. Bad mistake, but what do you expect
> me to do now?

Okay, I see that now. Both the #create and #open are overridden, so the
keyLength was never written to the file and just assumed to be a
constant 50, ah, er, 128. Could have used a change to the version number
stored in the header to differentiate, even allowing seamless
compatibility with an older repository. But that ship too has sailed. At
this point, one could put together some sort of heuristic: look at the
first few keys assuming 50/128 and deciding which it was. Then use that
as the keyLength for ODBClassInfoFile.

This is somewhat assuming that one could continue to operate on an older
repository using a keyLength of 50. Seems to be working for me. (Though
it's not getting much exercize, as I've been busy with other things). I
do plan on giving your script a try, to convert the file to 128 byte
keys, as it seems the better long-term solution.

> The interesting thing is that this
> error can never show up if on the start of every development session you
> load the code from the repository and then start developing.

Hmm. For me, starting with a fresh Dolphin image, installing a few
packages, but then mainly just loading my code from the repository takes
the better part of an hour. Keep meaning to profile part of it...

David, what exactly is the state of the OmniBase license? The web site
says "OmniBase is available for free for commercial and educational
purposes. Free version of OmniBase is prohibited to be used in the oil
or military industries." But, on the other hand both the readme.txt and
package file comment have "OmniBase object database is licenced and sold
under a 'per developer sit' licence model". The latter seem dated, but
it's unclear at best.

My sense is that you're moving to make OmniBase more open/free, but it's
not really clear. I would think that being clear about that up front
could aid in broader uptake by the community, contributions of code,
etc. Make it the real "go-to" whenever GemStone is too much.

regards,
-Bill

--
Bill Dargel            [hidden email]
Shoshana Technologies
100 West Joy Road, Ann Arbor, MI 48105  USA

Chris Uppal wrote:
> David Gorisek wrote:
>>I wasn't aware of this problem/bug but I know where it comes from.

HA!

> Why didn't you look into the problems I had during D6 beta ?  Did you even look
> at the reproducible test case I created for you ?
  ...
> What in Hell do you mean "not tested" -- you are talking about our /data/ here.
> You've made some kind of serious error which is endangering the integrity of
> other people's repositories (people, I remind you, who have /paid/ for your
> software), and you don't even test your "fix" ?
>
> I'm bloody furious.

Chris, I really concur with what you're saying. I went back in the
newsgroup and found
http://groups.google.com/group/comp.lang.smalltalk.dolphin/msg/691bc83f646afc5c 
  where you repeated what had gone into the beta newsgroup. It looks
like you put a lot of effort into it. Only to have it ignored. Twice.
I'd be pissed too.

On the other hand, I empathize with David's constraints.

As suits my personality, I'm torn. :-) / :-(

Hopefully, we can all move forward from this point.

-Bill

--
Bill Dargel            [hidden email]
Shoshana Technologies
100 West Joy Road, Ann Arbor, MI 48105  USA

Bill, David,

> Hopefully, we can all move forward from this point.

Well, I have no intention of trying to start a war with David, nor of making
this newsgroup an uncomfortable place for him to visit.

I'm going to leave it at that.

    -- chris