D6 corrupting STS repository

I'm trying to base a new STS repository on a copy of my old D5 repository, but
I can't make it work -- I end up with a corrupted OmniBase db.

If I point D6 STS at a copy of my D5 repository, then everything starts OK.  It
can read the data just fine.  However, if I start versioning things then sooner
or later I end up with a corrupt repository.

Some details:

I start with the copy and connect to it.  Everything's OK.  Now I try to
version all the existing code in the D6 image.  The results are
unpredictable -- sometimes it fails almost immediately, sometimes it takes a
long time before it fails, sometimes it seems to work OK.  If it has seemed to
work OK, then the STS browsing tools also seem to work OK.  Now I disconnect
and reconnect.  From here on nothing works.  It seems that the db connection's
ODBClassManager's list of #classes contains ODBClassDescription-s that contain
trash data. For instance the entry for 'StsPackageEdition' thinks that its
#instVarNames is:
    #( 'timestamp' 'IID' '' '' '' ''  ... '' )
and so its #varOrder is:
    #(1 18 18 18 18 ... 18)
which is nonsense.

I believe that the original repository is OK.  It works find in D5, including
if I version all the packages.  It also #garbageCollect-s without problems
(both from D5 and D6), though I don't know how significant that is as an
integrity test.

The same thing happens on two different machines (WinXP and Win2K) so I doubt
if it's a disk problem (my first -- worried -- thought given that it seems to
occur rather at random).

I haven't seen any problems using a fresh repository from D6.  I have done a
version-everything under D6 (fresh repository) but haven't (yet) tried doing a
second version-everything on top of that.

I suspect the problem (whatever it is) is in OmniBase rather than STS, but I
haven't yet been able to track down exactly /when/ it's first going wrong.
David, do you have any suggestions to try to pin this down ?  Does anyone else
see a similar problem -- or alternatively has anyone else done the same thing
and had it work ?

    -- chris

This is definitely OmniBase and not STS problem. Your persistent class
dictionary seems corrupt. The GC wont detect this because during GC no
serialization/deserialization is done.

I can not give you any more information about why this happened or how
to solve it without looking at is first. I never had such problems and I
am still using the same repository database which I started using with
Dolphin version 3.

If your repository is corrupt then the easiest solution is to export
your projects from D5 repository and import them into a newly created X6
repository.

Best regards,

David Gorisek


Chris Uppal wrote:

David,

> This is definitely OmniBase and not STS problem. Your persistent class
> dictionary seems corrupt. The GC wont detect this because during GC no
> serialization/deserialization is done.

Right, thanks.

So I wrote a small validation tool, and as far as that can tell the DB is fine
before the operation begins.  All the tool does is traverse the tree of
references in the STS repository, attempting to load each
package/method/class/etc edition (and subsidiary data), and recording any
failures. Not very clever, but I think it should be sufficient to find any
problems with an initially corrupt OmniBase DB (the data might still not make
sense in STS terms, but it wouldn't be /corrupt/).


> I can not give you any more information about why this happened or how
> to solve it without looking at is first. I never had such problems and I
> am still using the same repository database which I started using with
> Dolphin version 3.

I can probably send the repository to you if you like, but at > 100M I'm
assuming that you'd rather I narrowed the problem down a bit first ;-)  (By
"probably" I mean that I'll have to strip some things out first, and there's a
chance that that would remove the source of the problem.)

But that's the question -- how to narrow it down ?  Do you have any suggestions
?

Anyway, I've attached the little validator tool. Feel free to point out its
shortcomings.  Also feel free to add it, or a derivative, to STS if you think
it worthwhile (don't bother preserving the attribution and copyright notices).

    -- chris


begin 666 CU STS Scanner.zip
M4$L#!!0````(`("24S,,6#_Y_A ``,]%```2````0U4@4U13(%-C86YN97(N
M<&%CW5SK;MPV%OX]`?P.M/UC[%U;<;))BDYV@_4E*0PT<1 [+1:)"W DS@QK
MC:@5-;9GU_M ^Y9[SN%%E$:R9R9N@6W1HA)%'AY^/'=R?,=R'E_QL6!W&T_<
MX^!O[*-]S/A4#%C_^#,[OSAGYS'/,E'THZIOSF]_$H66*ANP9Z\WGO2&7,OX
M6$VG(BMQI,KGA1Q/2O9?=CPII&:?\YRG>^SYP<%+H!-C6S3#MK]/1<G'F=*E
MC"-5C#>>;#PYE],\%4R;B5DYX24K9IEFZAI>>9I"DV"QRDJ8#EI'C&?$:B%R
MI66IBGG$V 7TD8G@#*8G$CR;PZ"BF.4E<,YN)!#2$W7#9CGCFHFB4(5F-Q,9
M3Y@LB5A1Z@@Y0EJE**::S;($>#"=YFK&IGS.9AJ^3F :K4;E#2\$@_\&`,L_
ML,-,ERQ3)8M3+J>&$QQX4Z@2&"RC!_JIFRSHA5,!;^9;01\!RBMB\K3?3QC/
M\T+$DA-M)D=$(Y%)UN^7\']=%G(X@X]3E<B1% F[-AL)"Y?E1,V(=BHX,%,6
M<YF-6:E@Z0#5%)8UQ X$_H1G8Z&).@[#O;X!N": 3"J(G=[^/J.-QI<^-3GY
M(7&QTN8%J7\0'1P<O& [LRR?#5.I)R+9->.JD0".UA] /H%JCR?)@&W#QEL1
M14E$6+1(1_7Y9,:+^0^I&O*4!@_8SKD`N,5-,&2W-F9,O0]3R?52_;4H/Q:B
M$/^<21!!&G*:@'S*<EX-)8[[4?35_/NQ4.."3]D[P$Q_/5%I/I$@R5,0\9*G
M5^Q5=/#U;/BKB$MV"*+H>GP]`I[<2__UX]%]_]/'KR>2IVJL#0VAJ^&N@9D>
MZ\Y[>G',?E @%N+JZSD@&0MV40"&*&KG<UV*:?]U%\B;^+QUC#+ 3L1(9A(U
M66]1NUV0G@U)2&J2`01EIDN>Q>(G7D@^3(65@SY(]'E9". ZMPNTKV0/[+,Q
M10D(;G)V>@**#F8GGA4%[._%;>8>K40CY4:3%?(^\$',-9G #[E2Z8F,<47P
MU3=3_],.YOL&$2/9S JK@0.:?U0*S,5[`0J:V-:MMUF"!M-);>)1M-_MEAB*
M;A A[@A7J++Q3,+^[_SP^?2$C0HU!;1@&X&O?[\Z>GOT_.7S[_>_^_[H9/_%
MBQ<O][\_?OYL_R^'KUZ\>OON\."[%R__T]_=K)&+_Y@.!"D!B5C-T@2%; @[
M.&=#P<0MS)^ 6(&)313QI7$$,J3%E(/QB,&$BOA*LQUR,-98(T>)0HWAZ509
MOS'!UZDJT!7!:F@8F6_D')=+GBLEDWXVS23:$);PD@_Q`9U7B>L!(X[#C-JA
MQP$,T;81"6VT`=CUBT1;X35G#R!*;,_6C@8B0.0PU8J-!>P.>"J-_(,WBCFI
M,TW]-AH;+^(V$>*30%3 G$;!1_S_S^"_G(VBH4=S%&\^2TNW->B:;PI9`NSD
MUB9D>S(=%S(O'5<%R!,8"\9AQ])4D$+BCF!OT']&!@!\&[ZK3%C9B#FX9K-V
M.]F.1+&:(Y*HIINAH,.BQPJ5_)TJ2+DV/V>DZ.23-UFC_]2H,'0F#;16"-:_
M!<S?H$@W>'4B3KP2?R,.-IDD+56PB&2&FLHPAD#LM@CJ7[Q]X_JP*/@\HNE(
ME(ZJ.2<BS9U6_8I1"P= ISG(;8GH<323\56+BLT]?GM IVRJU]Q*FX"-!/J)
M&,[&3&9 5)9[3,&,!<^)+&PQ["HN0-S&(C=.H+$$"\B ?1D(#B)XQ]"=&+V
M#@.&K9?!"JF19V<G1Y_$"!PYF-MO6"V^C^4U"!K0=;P1!S(CF>.T62<*& P\
MB2(G=E@V.7&,+CB:0 AP,VT';]\QE'>S+PX&T6PA:YW5`Y2M*6HG;DDX^F#E
M4)218E[(:PQ.48%D,B?3.2I)A%$0$3J<K6Z,S22$'EF*^3MP6A@?@JLX`1WL
M&W2,X@5\HVV>HMPYTRS8A%\+C(-C= 76] (_3>G1\E\>G2 0:&#B[)XUC'LF
M"Y#H`P2:&F-MK$4`+=8:L$'CU,.T`3Q )M/(31U,TS(S"(1Y."L^R+3!AZ&_
M$A-FUC#&`0,;3F$@I>" IV];(4 9WPY)`)B8H R]X75K"ZQL2!9 ;LB$@#B'
M8G\R:;@`$#FP`1)ZPP+]3C5B->#=ZA;113?@/T:+ZPS[OJW!WG.1'O0ZP'<G
M$/!^5B0H,L>5I35>*%Q1;>8F5 VF.]%"J>AM9:J2+[]K=JZ&!>$'1ZF*KQI8
M!E,;@XG*):YY.N-D.SDSPRCS@SB=DSE363I'ZUS-P/@8'6))HA9$$<G0,!R8
M+\06[)\+*XX58.FA^@34SX!ZP'Q$Z[7L,Y'I&23.-8,(*6<!KKD^!P!D#:(Q
M![2':#+I`5>KTFN*1,F*FLS@]*2!CP:S5(L\C)%O,1+HE)"R6:]WDDGR(^3)
MC4DB9ZD@%AZ/!;!E5&00<(6 5^RZMP:14&:E!HV$;/ZBF"$^QI-<1DT=AK=>
M#_C.4SX'TWADO"W[0AE5\*4V%[DMV&!1G)XT^O4']X^4( RWS3&7F+/W%IV-
M7\0[2"RH/-+[TLT[T&;]UZS>NDBTV:.__^:!4=8U738@;DZ_LT#&;M?"C+NN
M9UR$,EFY*&[2HC;I\PK>+88C2Z<1JS5$S/5"-^LET##B8F+/"'YOF"(Y^J!*
MV!V,EB!6O\. O5KA*00\VN7 `]P6_\TVOF9Q<;DR7<)3UPF&#MA1#5!=6,SR
MJ-K(`C^"".BZYV_ Z2@,K.%U*NN\@U79]?"TPUI61_&^LUND9,LOS^8S8C3"
M*DA-A,!O%B5PK2WY(#@U\:;%H')][O'/[%FG-.WT*W;[;,^S[!9J5KD;ZL0Y
M,K+2IKEU8.:F[6@;*K9NG.MSKQ[T'1]$I^_VP$:VGY1JPAYZ4OCJXGM?E/01
M02,:1G=%`S@6-< S1G:(MA%K76PVUHLN%Z*$C@"S/IE?=:WUWC!S;8:Z8K;%
M>!,CD,"\Z\,LL09;M^])6Y;N2]I!)NR 1[]B(B*77* ?N,V1,?3TF/=#_$E[
M_%?,GO;?6')OX*-66%JOTLT[HG]'S_AT3Y3HA+$9O*'K^')7\7>'#?[-!:N!
M;$:M'4QT`8;*?U'9.\CY34#U"]O>V;VLC[P2<X(7XD&A+2.]+P-<,QN(Q)16
M#3L]_YK8?CV75=\Q^\WV=%T7&:/^(5,4Q=4'U4,$]@4QI1HS<;7_QA"Q&W(B
M3$JABDOX)[)23D/"\H5GP!NH@(EE(F<<[<])3"7#$$+F3T=5?07K&&!33=W/
M!XU4$WL@%D?Q,9PXN5(R<7+EV)8ZK @$*/WB.;(HX&!4+[_@9!A&EL$`_/3)
M+/$2MP(/8L@MXTNBJ'I2%'9O`V-J(V[\%L2U,*_9T"JFQR4*[^8LFJ:Z_'^R
M(TCSPAHZ*FVB*(*=V':"%3 /AL1[7W#YD-YIG)6GI2@RH'LM*FN#K)OJ*+\&
MX#2Y071*V]6>H%=W!Q=DSLY+,)]#<&6W:+=VRLE,4^G1>G5**FV!B@(<L7]^
M<41?8/.N!14NS:3H(%(Q!KV;`L:4\R%B`(% OD333,)ZC/TB$PKQ)H2FHBQ3
ML?L[B6QKE0R&_1YRJVWUBL*+>TM5>^0#<3N,5%&<24>ULS*?E=YU4J'<A"U&
MVEKC'QT$.32WC6M<VSWNDED4@MHG<FU[6W,,/L"ID6WQZ^YM'W\N5:(BQK9H
MN< @HAZR9.MZD<>(:MEOC3%']09Q#9L:2DUDJUB4"N#.G2R4_,+8N$&V`@0W
ML<G'$1XVZY9AK]MZV].R]EE:U]E)OVNU=/H-2C12+ACI7/_V#O((*HIUL6E.
MP@T*FJJ<SC1[F:D`]_ @SGQTQWGX1@1/ZDWFB T>=YTOKSPY[;W1&?-("ME8
M%8.I1ZJ8&GDAM:G[>3,T=/27EUW8=:/=!9X]"GD0N3L3Q1E[1(^MB['DHJ!3
M?34F%@P6XP(I^K `H:%G,+3/#P=")@JRW9LAD-=<@Y57+M/=Q3W=*R1LWO\Q
MENEEJ-&)5EUKN]_26!Z6-#4UPM%2>DF@G\^G0Y5:!0'+:9R3U!\-7]3/'P3Z
M:D9/TPG\BAI:XW$]%?7@0N*_T7*&8IQN=8CB:@2-$Q3;'&J\]SFV\E&Y'(X'
MUZ)H+^6X/9,0%B%0M8@//Z#U2US"<\\N[KAY]UC_J2E5!-/B`6U;N=+FI[;
MTUI$-/ 'M +7_%">YU,JV^+V[C?*_["344 _,0SUG;% $:RVUJFAUNY+%_W&
M>H" 'Y*(4M2.9-O3.=9 M8MNZ\+=]R[V7)C0"R2SX<GKE%YW]*XR_*6'V%LU
MRW:WUVZ6[>Z=Z7+=,<5 2[,PH.:MVQ$BFU/_U*6]7=&.G7.]>&=!9+"QBFUZ
M=%'1Y9$]R'#&`D4I!S)'\Y(N#SH:Q]6HG)<3>Z:^BNVMX[">\>U$W,O "I#'
M9LP2>'='28U%69*/%D"8B(W -(_+A@^F=V?T4$] J/.]Z(8ZO K":IJK&:1P
M*\EU'MP/=7=8>T:"Z1'<[ R/D*M>NET4W:;U[MVS14%<<>O\WJ'!3L74Y9'5
M'F%C<^1]>'L#^!MB79/I+7.-UUS]TCG>C$[3= X13):D]B#'5W)L6:I:$()P
MAG[H1FJQQT9TMLPUNC(J\9P".%F_9*,9TJ2C<:S5)+8RGBJ5LQN\]@=TZ((8
M\1[4<(@D[:KA$X.IL/QC:RX`AXQ$A-?-\-)@P5F*5A&H8LWJ-H<X`S?<UW80
MNIVPN+-+U3!D:X9F`2\Q@M@#SWHV!O=?RN"B8('V#^$2]M),50O253'(5($@
MND%[)I.M!RV(W8AOL""-:D:CEKA@+T@6NT6QGGHN)XJMR>?#0G@?+'^(!+0#
MXB#"6 'DPHUZ3)@]T4<#VE$T4/NW9<'V`SKA=NAYP-V09C*\T)$`:+1VH6W
M]=S4T.U.B1O$HZ6"-_+&-K[RD9R;>6.UF*O!P#<'7>'EX8U'+?0N4>3=8T_!
M:3REN^3F?-OW&_*BNP3L&#8_R*#A[U7"TY_Q-S@62;Q]8$XIL9AKH)1X73DW
MLN(/\>D,7[=?WDID8:^+TF "<\ .HEX/6^S!IP6\LR!-HXD)Z/SLX" Z8$_-
MG7=SU](0ZCB5E,S*!+T##R7X0:M<M?L&1*CGE<CR<G^Y.W)$[<(D^Q.A==E8
M,#!-32M7QMVV-900XL$<'2T(3K><6:+QRM<2:^IK+SQXD2_#:[(V*'G/,X"J
M\ =.S%V^,#]TB*MKXII^%N<HN3*9&=Y8H_OI@">*AUBPP [:*("^IF4(1@O7
MX6O7YP=LVX*QR>,8S]6R\2;]@"Q^<&!XLWW37/36*XW%.^,K#UPH-:W!>%LU
M:G4R5DHW[7YM:FL+'AY(H>$:$U87[KYI\&#UT8O7?3?]+60B8S%8D@[8K&\:
M_S&\*+PNI7J%L1+$)0F$1Z+&4Z*']\.=$UJ2CK_PN"X!?T%K70(^2%F;@#63
MJX^WT2G69U?>AOJ]J-4ENW&#:W4"]SCMU4V#"[FJ6&ME0(J.='QE0IJ,FP%A
MZ05@QUH9:74,FB1LV?3;";FD=3U*]01N/1H0A*PN8#C016(N_EIO^M9R]&.0
M<G761Z$55!4?@YZKFCT&K6^3H*[\?CUJS<1U/2IAZK:ZLKO0?(VYRS4"( R7
M6\U88QA5N6UMZ)W]V:?Y$P+53S[AO1&K2TT)V-QG%%0_-;%B:UIA?Y*(Z5"0
M`-@1)GFISM>K^'Y5%DR1`1F8&B)]O<!*D%1!6@"Y+/Y:F@Z:.;OA<[KE5Y:0
MLTGS-REX0 $35#I@=T^I7]\.+9 .H3Z(FUU*Z'P$9VX`-3,8:O5_1V)Q2VEO
M%@((<;/ID8@!`I3H3JEH)8$:<"^)C2=;1_27+.H_S_\?4$L!`A0+% ````@`
M@))3,PQ8/_G^$ ``ST4``!(``````````0`@`````````$-5(%-44R!38V%N
=;F5R+G!A8U!+!08``````0`!`$ ````N$0``````
`
end

Chris,

could you first send me a walkback file so that I can see which error
you are getting?

If this wont help I can then give you FTP access on our server for
sending database files.

Best regards,

David Gorisek


Chris Uppal wrote:

David,

> could you first send me a walkback file so that I can see which error
> you are getting?

I'll attatch some extracts from the .errors file, but I doubt if they'll be
much use.  The walkbacks happen as a response to the fact that the Omnibase
state is already trashed.  And, as I mentioned, they don't always happen at
all -- sometimes everything seems fine until I disconnect and reconnect.  There
have been a variety of failures but the most common one(s) seem to the be of
the 'unknown serial id(1)' variety.

> If this wont help I can then give you FTP access on our server for
> sending database files.

Shouldn't be necessary -- I've managed to put together a reasonably small
example respository and have posted it at:

    http://ephemera.metagnostic.org/code/Repository.zip

1.8 Mb zipped.

Please note -- that repository was generated from my real one by stripping out
lots of stuff "by hand" (working at the OmniBase level rather than with STS
tools), that may mean that it isn't completely valid from the point of view of
STS's own semenatics.  However it /is/ valid (I believe) as an OmniBase db.

Now to see the problem (detailed instructions in case anyone else wants to
follow along).

First connect STS to that repository:

    StsManager startUpOn: 'somewhere'.

You should find that all the STS tools work OK (at least I didn't see any
problems despite the above caveat).  My own scanner (which is also in that
download) finds no problems.  All seems good.

Now run:

    StsManager current fixup.

where #fixup is a loose method defined in the fixup.pac package (also in the
download).  All that does is duplicates the versioning code in
StsManager>>install.  That takes a while, and -- in the couple of times I've
tried it -- produces no apparent errors.  (Remember that that sometimes happens
in the "real" case too).  Assuming that there were no errors, then the STS
tools will still be working too, and my scanner:

    STSScanner new scanWithProgress.

will not find any problems.  If the versioning loop /did/ encounter errors then
the OmniBase state should already be corrupted.

Now disconnect:

    StsManager shutdown.

close any browsers, etc, and then reconnect.  If your system has done the same
thing as mine, we are now in difficulties.

Try (working just at the OmniBase level) looking at the data:

    txn := StsManager current databaseConnection newTransaction.
    txn root at: 'sts.packages'.    "inspect it"
    txn root at: 'sts.classes'.    "inspect it"
    "close the inspectors then: "
    txn abort.

You'll probably encounter a lot of errors. If you open the package edition
browser then it'll throw lots of errors too, and may be difficult to kill.

If you inspect OmniBase's class manager:

    StsManager current databaseConnection classManager.

then you'll see that many of the class definitions are corrupt.

Remember to:

    StsManager shutdown.

before attempting to use any of the browsers!

Now it's possible that some of the errors in the above are because I have
mangled the database contents -- but the point is that nothing we've done
should have actually corrupted the OmniBase data.  In my real image, exactly
the same kind of corruption happens.

The single package in the respository /may/ be connected with the problem, or
there again it may just be coincidence, but without that package I wasn't
seeing the same kind of corruption.  The package is unusual in two ways.  One
is that it has a global variable defined.  The other is that that variable
holds an instance of one of the classes in the package.  Indeed, because I've
changed the shape of that class a few times without creating suitable STB fixup
code, the STB-ed binary data for that global can only be reconstituted (without
errors) using the code in the same version of the package.  I repeat that it
may just be coincidence, but if not then oddness of that package (and some
others like it in my real image) might explain why I'm seeing a problem when
you are not.

Anyway, thanks for looking into this.  Please let me know when you want more
data.

    -- chris


begin 666 Default.zip
M4$L#!!0````(``%T5#,X+E*87@(```X2```.````1&5F875L="YE<G)O<G/M
M5]%NVC 4?4?B'_Q6D(9$H*#BAT@$.JE2@:FTZ\.T!R>^H1Z.C6QGP+Y^C@,L
M;:D8%5L[AA\BQ?><Z^MKGZ.D5OOSHUSRZKC9P.<7'Y!W@4:1D2$HU*C76QC=
MB0<B* >*8!'!S# I4 T1@4;]X%(IJ2IGHT2P@&A %#0H1CC[`2IC3H6<"[2>
M(X[+**IXU;-JN50N;8@1)UK[OF830;C+BFVT'_0+"7V?2T)=L%RZ$Q1B)H".
MPF\0&=^7-"R M[$M8@@+DQ-<?%N2:[O&9V)9(0>-75U7(I9X^Y(OHY\M+V">
M$X<P[V5 AQD^G5VWXK_?S]CH3R2:D@E<4N9NCLM4R>/Y2_6]%?TL7E#$1R43
MK" &!2*"`1%V:PH;180F4;;!//]XC:?]($_;;K6:[9LUT9:5D"G<_N+E*,O>
MI?-#C$RVNS"'&-DZGH>]-F[63ZYT4O'?+/IM5?SE*V+N'E_+B/ "HF+G"J^V
M;>8IO9LE"+B,ICTN=:K ]UG<2Y4AS&HE+^VU:5U7$JNZ[*2RKDS@<<>*L1FH
M6*H$SYEYZ*I)FH P^D@MJH.]/3Z<4"!30?7*I*ZL?!>HX740TTBF!LD8A0Z0
M>U&P--!5BBQ7BK&G`QESG(8Z4FQF\FSX961^)39!]P3:DYS#^KS)ZG@'D$BU
MO %"QT8!2=P19]3?DLQ^3K0?^C@LH-$Z&8 %;GI@NRI!#Z7)+H/2QM(M8@!:
MVX[ZON7/B:*WLCCYG? 4CM%([!]8"Y\WK9%T7F\D7MT[&<E[,Y+]*MP/_1;[
M.;PQWEN3L'?V7S;'71H_Q"B7?@)02P$"% L4````" `!=%0S."Y2F%X"```.
M$@``#@`````````!`" `````````1&5F875L="YE<G)O<G-02P4&``````$`
,`0`\````B@(`````
`
end