Two store questions

Charles A. Monteiro wrote:
> if you don't mind me asking, how many developers are we talking about i.e.
> that actively involved on a day to day basis in the process you describe
> below.

That fluctuates between 4 to 6 pairs.

R
-

Gentlemen,

I have started this thread, and I think now this horse is way past dead.

I think the problem with the "version" is that there is a big time gap
from the time I start the Publish dialog(which gets me a version
string), and the time I actually initiate the database transaction by
pressing the  <publish> button.

I am not a DB expert (spent 10+ years on Oracle, Sybase, Gemstone), but
I don't really see pure database solution to this.

Locking tables is not an option - what if I got interrupted and forgot
to publish, or image crashes, or... -  db gets unusable for others...

Failing and later trying to restart transactions may not be that easy
either.

What we all should do - is to analyze  our Usage Patterns of the Store
and see exactly where the Store falls short.  and then see how to
address shortcomings.

Maybe we need something like "Check out/check in" functionality --
something along the lines: you open PUblish dialog and DB gets notified
that the PAckage/Bundle is being published. So next developer will get
to know that somebody intending to publish the package at this time,
with all the relevant information like when and who.
But then we are getting to the mess with the expirations - i.e. how long
to keep that "check out" active, etc.; and those kinds of things can not
be decided programmatically.  THey are subjects of the Operation
Policies and Procedures, which humans tend to forget or ignore :-(.

we may entertain the idea of the package/bundle ownership and who can
commit package or bundle and so on - along the lines as it was done in
ENVY, but I doubt we can exactly replicate it in store.

At this time however, It is mere annoyance for my organization, not a
show stopper.  Our builds don't happen that often as for Reinout, nor it
has that many developers, so this type of problem doesn't happen too
often if ever.

My 2C.

--Mark Pirogovsky

Reinout Heeck wrote:

At 09:46 AM 2/28/2006, Reinout Heeck wrote:
A relatively simple workaround would be to save the primary key of the packages you plan to load, either instead of or as well as the version number. Or, when it goes to load, checking that the version number is in fact unique, and you don't get two or more packages back from that query, and crashing quickly if there are. Or you could build the list and then do the load all within a transaction.

A newest with blessing level above X approach also seems susceptible to corruption when the blessing level changes.


--
Alan Knight [|], Cincom Smalltalk Development
[hidden email]
[hidden email]
http://www.cincom.com/smalltalk

"The Static Typing Philosophy: Make it fast. Make it right. Make it run." - Niall Ross

From: Mark Pirogovsky [mailto:[hidden email]]
> I think the problem with the "version" is that there is a
> big time gap from the time I start the Publish dialog
> (which gets me a version string), and the time I actually
> initiate the database transaction by pressing the
> <publish> button.

The rest of Mark's suggestions are good, but how about something this
simple: when you press Publish, Store does a quick re-check of the
already used version numbers? Whilst it's not 100% watertight, it would
at least be easy to implement, and better than the current offer of "no
we can't".

Steve

Steven Kelly wrote:
Thanks,

but how about something this
> simple: when you press Publish, Store does a quick re-check of the
> already used version numbers?

It is way better, but it only shrinks the window of opportunity to
publish the same version.  For the fast and often publishing team like
Reinout has, it will not solve much.  Also it will not work too well for
the remote people, where the transaction time may be few minutes from
start to end.

Whilst it's not 100% watertight, it would
> at least be easy to implement, and better than the current offer of "no
> we can't".
>
> Steve
>

--Mark

My understanding of the current behavior when opening a publish dialog is that a new version string is generated based on the "parent" version string.  If the proposed string is already used in a published version, it is modified and rechecked as necessary until an unused string is found.  This unused string is then displayed in the UI.

I've seen applications that keep track of the next primary key when creating a new record.  If the creation is aborted, the aborted key is not reused for the next new record creation, but the next in the sequence is used instead.

I'm no expert (I avoid thinking about dbs if I can), but it seems that a "next version string" reservation table could be added to the store schema.  The records of this table could have fields for pundle, reserved version string, reservation timestamp, and user id of the person who made the reservation, with constraints that equivalent strings cannot exist in this table for the same pundle, and that a reservation cannot be made for a string already used for a published version of the pundle in question.

We could alter the current behavior such that in addition to searching for a string in published pundle versions, this table is also searched to ensure no other developer has dibbs on the proposed string.  If the string is available, a new reservation entry is created for the proposed string when the publish dialog opens.  If the user edits the string, then either leaving focus or clicking the publish button would trigger another check, and if the new string is available the previous reservation entry is either edited, or replaced with an updated reservation entry.  Cancelling the dialog and other failures would remove the reservation.  If an exceptional condition left an orphaned reservation, other recovery mechanisms can come into play such as dialogs and timeouts.  The way in which reservation contentions are resolved can be tailored to each development organization.

Wouldn't a scheme like this be feasible?  It seems to me that taking advantage of the db's concurrency and constraint features would greatly simplify the issue.

Dave

Like I said, it's not 100% watertight. Doing that either requires
DBMS-dependent code (as Alan pointed out) or a new table in the Store
schema (as Dave suggested).

If Reinout would really still hit this problem often with only 4-6
pairs, I think he could be encouraged to look at whether his work
process is a contributory factor, not just Store. If every pair is
encouraged to edit the same pundle often - to exaggerate, say an
"AllSoopsStuff" bundle, which everybody is asked to update and publish
whenever they publish a package - then that bundle will of course be a
hotspot for locking conflicts. That's bad, even if Store would recognise
the conflicts. Version control systems that allow forks are simply not
built to implement a single version path, where everyone is constantly
adding a new version.

Would the above suggestion help, Reinout? What if we extended it so at
the end of the publish, we again check to see for this version number,
and if it exists twice, give a warning: "Developer "stevek" has
published a version X.Y at the same time you published your X.Y. Please
contact him to sort out which of you will rename your version. If it is
you, please enter a new version number here: ____". In some cases the
warning would be seen by both parties, in some only by one: either way
this should be enough (although of course the check would need to be
repeated each time a new version number was entered).

The purist in me rails against such unholy hacks, but if the
Store<->DBMS architecture really has been set in stone before the
integrity constraints were fully figured out, it might be all we ever
get... And even if eventually someone is going to make Store elegant,
some kind of workaround is needed in the meantime.

Steve

Alan Knight wrote:
> A relatively simple workaround would be to save the primary key of the
> packages you plan to load, either instead of or as well as the version
> number. Or, when it goes to load, checking that the version number is in
> fact unique, and you don't get two or more packages back from that query,
> and crashing quickly if there are. Or you could build the list and then do
> the load all within a transaction.

Weighing effort versus effect I guess simply adding the DB constraint will be
my first act, it avoids the entire problem albeit at the cost of confusing
error reporting. Using a transaction sounds like a good idea ot follow up on.


> A newest with blessing level above X approach also seems susceptible to
> corruption when the blessing level changes.

Correct, hence we use Lineups in other stages of our process. (A Lineup
effectively publishes the aforementioned list of versioned prereqs).



R
-

Steven Kelly wrote:
>
> If Reinout would really still hit this problem often with only 4-6
> pairs,

I don't know how often this occurs. Personally I've seen this problem once, I
don't know how often it happens under the radar.


> I think he could be encouraged to look at whether his work
> process is a contributory factor, not just Store. If every pair is
> encouraged to edit the same pundle often - to exaggerate, say an
> "AllSoopsStuff" bundle, which everybody is asked to update and publish
> whenever they publish a package - then that bundle will of course be a
> hotspot for locking conflicts. That's bad, even if Store would recognise
> the conflicts.

See
http://www.cincomsmalltalk.com/userblogs/travis/blogView?showComments=true&entry=3265388740
for some of my views on bundles and our process.

> Version control systems that allow forks are simply not
> built to implement a single version path, where everyone is constantly
> adding a new version.

We are not simply adding, we rely on the publish dialog to tell us whether a
merge is necessary: if dotted versions appear I have to merge after the
publishing to bring my changes into the head version. The bug under
discussion endangers this process.

> Would the above suggestion help, Reinout? What if we extended it so at
> the end of the publish, we again check to see for this version number,
> and if it exists twice, give a warning: "Developer "stevek" has
> published a version X.Y at the same time you published your X.Y. Please
> contact him to sort out which of you will rename your version. If it is
> you, please enter a new version number here: ____". In some cases the
> warning would be seen by both parties, in some only by one: either way
> this should be enough (although of course the check would need to be
> repeated each time a new version number was entered).

I prefer the duplicate version not ending up in the DB at all, isolating the
burden of problem resolution to a single pair. Fail faster please :-)


> The purist in me rails against such unholy hacks, but if the
> Store<->DBMS architecture really has been set in stone before the
> integrity constraints were fully figured out, it might be all we ever
> get... And even if eventually someone is going to make Store elegant,
> some kind of workaround is needed in the meantime.

More to the heart of the matter: I would like to be able to trust Store, this
requires closing holes due to race conditions. As it stands though Store is
pretty much cast in stone in order to support backwards compatibility, hence
I understand Cincom's reluctance to alter the DB, even when only adding
constraints. A sorry state of affairs though :-(


R
-