SCM'ed VW

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

SCM'ed VW

Charles Adams
With all the talk about Store and the best database product to use, its time, once again, to bring up the issue of a published distribution. It is long past time for Cincom to produce versioned code in some sort of Store configuration. There's no technical, legal or even philosophical reason why this can't or shouldn't be done.
 
If I tried to hand my customers files instead of software under configuration management, I'd be out of business. Look... I'm not going to stand here and make the case for SCM. We are all seasoned professionals. Its time for Cincom to acknowledge their responsibilities in this area.
 
Charles Adams   
Adventa Control Technologies, Inc.
3001 E. Plano Parkway, #100 
Plano, TX 75074-7422 
http://www.adventact.com
[hidden email]
 
Reply | Threaded
Open this post in threaded view
|

Re: SCM'ed VW

Alan Knight-2
Indeed, the reasons are all procedural, logistical, and financial :-)

At 03:52 PM 1/22/2008, Charles Adams wrote:
With all the talk about Store and the best database product to use, its time, once again, to bring up the issue of a published distribution. It is long past time for Cincom to produce versioned code in some sort of Store configuration. There's no technical, legal or even philosophical reason why this can't or shouldn't be done.
 
If I tried to hand my customers files instead of software under configuration management, I'd be out of business. Look... I'm not going to stand here and make the case for SCM. We are all seasoned professionals. Its time for Cincom to acknowledge their responsibilities in this area.
 
Charles Adams   
Adventa Control Technologies, Inc.
3001 E. Plano Parkway, #100 
Plano, TX 75074-7422 
http://www.adventact.com
[hidden email]
 

--
Alan Knight [|], Cincom Smalltalk Development
Reply | Threaded
Open this post in threaded view
|

Re: SCM'ed VW

Charles Adams
In reply to this post by Charles Adams
Well then these are surmountable. Agree on a procedure; find the time and assign a resource. You'll be done before you know it.
 
I dare say there are already procedures internal to Cincom. No doubt you each find the time to CM your own work. All you need now is to find an aggregator.
 
Hmm, do you want to take our publication of your release? How about someone elses? There are probably several publications of VW 7.4.1, VW 7.5 and so on. They may vary slightly but maybe you'd like to take a donation and bless it. Something, anything to get started. Please.
----- Original Message -----
Sent: Tuesday, January 22, 2008 5:24 PM
Subject: Re: SCM'ed VW

Indeed, the reasons are all procedural, logistical, and financial :-)

At 03:52 PM 1/22/2008, Charles Adams wrote:
With all the talk about Store and the best database product to use, its time, once again, to bring up the issue of a published distribution. It is long past time for Cincom to produce versioned code in some sort of Store configuration. There's no technical, legal or even philosophical reason why this can't or shouldn't be done.
 
If I tried to hand my customers files instead of software under configuration management, I'd be out of business. Look... I'm not going to stand here and make the case for SCM. We are all seasoned professionals. Its time for Cincom to acknowledge their responsibilities in this area.
 
Charles Adams   
Adventa Control Technologies, Inc.
3001 E. Plano Parkway, #100 
Plano, TX 75074-7422 
http://www.adventact.com
[hidden email]
 

--
Alan Knight [|], Cincom Smalltalk Development
Reply | Threaded
Open this post in threaded view
|

RE: SCM'ed VW

Steven Kelly
In reply to this post by Charles Adams
Message
Hi,
From: Alan Knight [mailto:[hidden email]]
Indeed, the reasons are all procedural, logistical, and financial :-)

At 03:52 PM 1/22/2008, Charles Adams wrote:
With all the talk about Store and the best database product to use, its time, once again, to bring up the issue of a published distribution. It is long past time for Cincom to produce versioned code in some sort of Store configuration. There's no technical, legal or even philosophical reason why this can't or shouldn't be done.
 
If I tried to hand my customers files instead of software under configuration management, I'd be out of business. Look... I'm not going to stand here and make the case for SCM. We are all seasoned professionals. Its time for Cincom to acknowledge their responsibilities in this area. '
I'd second the request, something I first asked for in 5i.3 :-). Alan says "procedural, logistical and financial", so let's look at those.
 
"Procedural" - you mean someone at Cincom would have to publish the base, or more likely replicate the release pundle lineup. Currently, every customer wanting to see VW version history in Store, or wanting to know what they've accidentally changed in base VW packages, needs to do that. We've even partly automated the reconcile + publish process, and I'm sure you could do better.
 
"Logistical" - you mean Cincom has to find a way to distribute the published repository. An SQLite file sounds like an obvious solution. A method in the base to script the reconcile + publish would of course require zero effort to distribute.
 
"Financial" - there are no direct costs involved, only time: tell us how long it will take you in hours. We already spend something like that number of hours, but without the certainty that what we arrive at is "official". You even have the offer of using Adventa's (or MetaCase's) publish, if that helps.
 
Making the upgrade easier will also lower the perceived barrier to upgrading to a new VW version. The time and annoyance of having to do the reconcile and publish ourselves are one thing. Another is the worry that in a partially manual and hackily automated process, we may cause ourselves versioning problems. With a correct reconcile + publish, it's easy to check for potential problems by comparing with the previous version, and of course it's a major help in debugging such problems.
 
Cheers,
Steve
Reply | Threaded
Open this post in threaded view
|

Re: SCM'ed VW

Reinout Heeck-2
In reply to this post by Alan Knight-2
Alan Knight wrote:
> Indeed, the reasons are all procedural, logistical, and financial :-)
>
Which makes it even stranger that we don't have this...

Procedural:
  - all the customers need to do a 2 day manual labor session to get a
proper repository vs. Cincom can automate it in their weekly build once.
Logistical:
  -the distribution format /may/ need to change from CD to DVD, big deal.
Financial:
  -effort spent /once/ by Cincom vs effort spent /once every release/
times every customer

The bottom line is that there is only so much money going around in the
VW community, currently we collectively spend too much of it on
upgrading our repository contents (in parallel/duplicative efforts). If
we move that effort to Cincom we collectively spend less and Cincom can
turn over a bit more. The cream on the cake is that all our repositories
will be the same wrt. to base code.


So I guess the reasons are none of the above but rather one of
incentive, how can we make this attractive to Cincom?

R
-


Reply | Threaded
Open this post in threaded view
|

Re: SCM'ed VW

Antony Blakey-2

On 23/01/2008, at 8:54 PM, Reinout Heeck wrote:

> Alan Knight wrote:
>> Indeed, the reasons are all procedural, logistical, and financial :-)
>>
> Which makes it even stranger that we don't have this...
>
> Procedural:
> - all the customers need to do a 2 day manual labor session to get a  
> proper repository vs. Cincom can automate it in their weekly build  
> once.
> Logistical:
> -the distribution format /may/ need to change from CD to DVD, big  
> deal.
> Financial:
> -effort spent /once/ by Cincom vs effort spent /once every release/  
> times every customer


Surely it's possible to turn this issue into a purely legal one?

For example - I have put the VM sources in a Bazaar repository so I  
can merge my changes across Cincom vw-dev releases. CST08_jan08.3 had  
a lot of non-substantive changes to copyright, version and license  
comments. I went to the trouble of splitting these changes so I could  
have two different commits, one for substantive and one for non-
substantive. Anyway, as I was doing this I thought "maybe others would  
like access to my trunk branch/repository ... if only I could manage  
the legal issue with Cincom."

My point being that the community could solve these problems if it  
were legally able to do so, and hence the problem to be sorted with  
Cincom becomes a legal one, rather than 'procedural, logistical and  
financial'.


Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787

Borrow money from pessimists - they don't expect it back.
   -- Steven Wright


Reply | Threaded
Open this post in threaded view
|

Re: SCM'ed VW

giorgiof
In reply to this post by Reinout Heeck-2
Hi,
 
>
>-the distribution format /may/ need to change from CD to DVD, big deal.
>Financial:

 
this could be a zipped SQLLite file everyone needs it can download it  from a Cincom server. So no change on the distribution.
 
 
Giorgio
 
 
On 1/23/08, Reinout Heeck <[hidden email]> wrote:
Alan Knight wrote:
> Indeed, the reasons are all procedural, logistical, and financial :-)
>
Which makes it even stranger that we don't have this...

Procedural:
- all the customers need to do a 2 day manual labor session to get a
proper repository vs. Cincom can automate it in their weekly build once.
Logistical:
-the distribution format /may/ need to change from CD to DVD, big deal.
Financial:
-effort spent /once/ by Cincom vs effort spent /once every release/
times every customer

The bottom line is that there is only so much money going around in the
VW community, currently we collectively spend too much of it on
upgrading our repository contents (in parallel/duplicative efforts). If
we move that effort to Cincom we collectively spend less and Cincom can
turn over a bit more. The cream on the cake is that all our repositories
will be the same wrt. to base code.


So I guess the reasons are none of the above but rather one of
incentive, how can we make this attractive to Cincom?

R
-



Reply | Threaded
Open this post in threaded view
|

VW 7.5 text view/editor

Mike Bielser
Hello
I have a simple question: Is it possible to have different fonts (with different attributes) in one
and the same text view/editor?
I tried some code but it does not seem to work... instead of wasting hours, I'd rather ask the experts.

Thanks a lot
Mike

Reply | Threaded
Open this post in threaded view
|

RE: SCM'ed VW

Alan Knight-2
In reply to this post by Steven Kelly
At 03:22 AM 1/23/2008, Steven Kelly wrote:
Hi,
From: Alan Knight [[hidden email]]
Indeed, the reasons are all procedural, logistical, and financial :-)

At 03:52 PM 1/22/2008, Charles Adams wrote:
With all the talk about Store and the best database product to use, its time, once again, to bring up the issue of a published distribution. It is long past time for Cincom to produce versioned code in some sort of Store configuration. There's no technical, legal or even philosophical reason why this can't or shouldn't be done.
 
If I tried to hand my customers files instead of software under configuration management, I'd be out of business. Look... I'm not going to stand here and make the case for SCM. We are all seasoned professionals. Its time for Cincom to acknowledge their responsibilities in this area. '

I'd second the request, something I first asked for in 5i.3 :-). Alan says "procedural, logistical and financial", so let's look at those.
 
"Procedural" - you mean someone at Cincom would have to publish the base, or more likely replicate the release pundle lineup. Currently, every customer wanting to see VW version history in Store, or wanting to know what they've accidentally changed in base VW packages, needs to do that. We've even partly automated the reconcile + publish process, and I'm sure you could do better.
 
"Logistical" - you mean Cincom has to find a way to distribute the published repository. An SQLite file sounds like an obvious solution. A method in the base to script the reconcile + publish would of course require zero effort to distribute.
 
"Financial" - there are no direct costs involved, only time: tell us how long it will take you in hours. We already spend something like that number of hours, but without the certainty that what we arrive at is "official". You even have the offer of using Adventa's (or MetaCase's) publish, if that helps.
 
Making the upgrade easier will also lower the perceived barrier to upgrading to a new VW version. The time and annoyance of having to do the reconcile and publish ourselves are one thing. Another is the worry that in a partially manual and hackily automated process, we may cause ourselves versioning problems. With a correct reconcile + publish, it's easy to check for potential problems by comparing with the previous version, and of course it's a major help in debugging such problems.

I think there's more to it than that. There are, for example, some interesting issues about what exactly we're exporting. So, presumably we'd want to be creating such an export with each weekly build, and with release candidates. I'd certainly hate to have something that we only did once a release and expected to get precisely right that one time. So then what are exporting? Do we use our own internal version numbers or something else? What's the parent version? When we release, I think what customers would ideally like is to have version numbers like "7.6" whose parent versions are like "7.5". So should we do an export that warps the version numbers and the parents to that? And if we do, what do we do in the weekly builds? Are their parents the previously weekly builds, or are they the previous released version? And do release candidates then each provide a set of versions named 7.6, but which are different? Or should we just expose our internal names (These currently tend to look something like 7.6jan08.3 3, reflecting the build they're targeted at, plus an increment. But things that haven't changed don't get re-published, so may have version numbers from different builds, or if they haven't been changed in this release, previous releases. Our internal names also aren't always consistent). Should we change our internal naming convention to be something that's more convenient for customers, and if so, what would it be?

There's also the issue of support. If we're putting something like this into supported status, then that implies also putting into supported status and maintaining various other things. Assuming SQLLite worked out, it means a SQLLite EXDI layer, Store for SQLLite, some form of Store Replication (at least to get it from that repository into the customers, and probably to get it into that one in the first place), and the specific bits of that required for SQLLite, if any. That's not a trivial amount of stuff. If SQLLite is not supported on one of our supported platforms (how's their SGI support? MacOS9?) then that's also an issue. It's possible we could ignore that on the assumption that everyone's got a machine around somewhere that can talk to both SQLLite and their store database, but that's an interesting decision. Or we could say that we're distributing this, but not in a supported way - again, an interesting product management type of decision.

Aside: I'm assuming we'd want to replicate because I think we'd want to do this for the full set of things we're distributing (or at least the full set of supported things), not just for the things that are in the standard base image, so reconciling and publishing with regular Store, which requires things to be loaded into the image, seems less good. There are also alternative ways of distributing it. One would be as ExportDatabase files. But that needs some work, might have some scaling issues for large databases, and relies on John Brant's Replicator for its underlying support. Another would be, rather than distributing a file, to give customers access to a Store database online with the information in it and let them replicate it however they want. That would still require replication, but might address some of the issues of version naming. We don't put the versions into that database until we have a shipped release, and we rename them to correspond to the release at that point. It could also just be a Postgresql database, removing the issue of supporting another one (although SQLLite might be useful in its own right as a portable starter database for Store). And we could easily make past versions available - handling the parentage if we only ship one release's versions in the distribution is an interesting question.

So, I think this is a good idea, I think it would be valuable to customers (and a good number of customers seem to strongly agree) and I've spent time thinking about how to implement it previously. But it's also not trivial, and I don't even have a good idea of how many hours it would take to do it. I don't think that access to someone else's publish of the base is all that helpful. We do have our own. The issues are mostly around exactly what and how to distribute, and how this compares to all the other things that customers think we ought to be doing.

One of the responders asked how people could help. The most obvious way, of course, is by giving us money. It makes it ever so much easier to find time and money when people give us the latter. The more the better :-) However, barring that, useful suggestions about how people would want to receive such a distribution and what they would want it to look like are probably the most valuable. And writing code is good - e.g., if someone wanted to write "rename and reparent while replicating" functionality. But that's assuming that we knew exactly what we wanted to do.



--
Alan Knight [|], Cincom Smalltalk Development
Reply | Threaded
Open this post in threaded view
|

Re: SCM'ed VW

Stefan Schmiedl
On Wed, 23 Jan 2008 10:21:30 -0500
Alan Knight <[hidden email]> wrote:

> When we release, I think what customers would ideally like is
> to have version numbers like "7.6" whose parent versions are
> like "7.5". So should we do an export that warps the version
> numbers and the parents to that?

Indepently of the issue at hand, I'd *very much* appreciate this.
There are parcels in the distribution with a seemingly outdated
version number. How do I know if the parcel is still intended to
be used? Was it introduced in 7.2, last modified for 7.2, is it
still useful in 7.6?

Also I remember being told to use a more recent version from the
public store for this package, while for that package the
parcelled version is more recommended.

> Should we change our internal naming convention to be something
> that's more convenient for customers, and if so, what would it
> be?

Here's what I'm doing in my personal Store: Spikes are numbered
with integers, they're not for keeping anyways. All "real" code
is organized in a bundle, which is what I'm actively publishing.
Also I'm using the bundle's current version for all modified
packages, so I end up with a continuously numbered bundle and
"jumpy" version numbers for the packages within then, as modified
packages always have the bundle's version number.

> There's also the issue of support.

Actually, there's a disadvantage with using an (any) external
library for my mostly linux based development. I'm usually
running a 32-bit image (so that I can continue working on
my Windows laptop) within my 64-bit linux system. Every external
library I intend to use, I have to keep and maintain as a 32-bit
version separate from the rest. But I guess that's just me, so
not really important to the world at large.

> Or we could say that we're distributing this, but not in a
> supported way - again, an interesting product management type
> of decision.

Yeah, well. That brings up a sore point from the current history,
when Widgetry was released as "fully supported" and then
withdrawn a few weeks after that. So "distributing, but not in
a supported way" can't be much worse.

> However, barring that, useful suggestions about how people
> would want to receive such a distribution and what they would
> want it to look like are probably the most valuable.

I don't know about corporate people having to deal with IT
department regulations, but as a one-man company, I am very
comfortable downloading such things.

For starters, how about providing a Store containing the baseline
of the base image, where everything has the version of the build?

I'm sure you can think of reasons why this is too simplistic, so
let's discuss.

Hm. Wait. Discussion might take place during working hours... and
too much of it might prevent them from implementing a solution
:-)

s.

Reply | Threaded
Open this post in threaded view
|

RE: SCM'ed VW

Terry Raymond
In reply to this post by Alan Knight-2

Alan

 

With respect to VW-Dev builds, the package versions should reflect

the build version in which they were changed.  For example,

suppose Assets was last modifed in October and released in the

waldo oct07.2 build then its latest version would be oct07.2, even

in the waldo jan08.2 release.

 

When VW actually ships, all the package versions would be updated

to the shipped version, i.e. 7.6 or 7.6.0.

 

Additionally, your discussion points out the need for Store to better

trace component heritage, particularly when you consider replication.

 

Terry

===========================================================
Terry Raymond
Crafted Smalltalk
80 Lazywood Ln.
Tiverton, RI  02878
(401) 624-4517      [hidden email]
<http://www.craftedsmalltalk.com>
===========================================================


From: Alan Knight [mailto:[hidden email]]
Sent: Wednesday, January 23, 2008 10:22 AM
To: Steven Kelly; Charles Adams; [hidden email]
Subject: RE: SCM'ed VW

 

At 03:22 AM 1/23/2008, Steven Kelly wrote:

Hi,

From: Alan Knight [[hidden email]]

Indeed, the reasons are all procedural, logistical, and financial :-)

At 03:52 PM 1/22/2008, Charles Adams wrote:

With all the talk about Store and the best database product to use, its time, once again, to bring up the issue of a published distribution. It is long past time for Cincom to produce versioned code in some sort of Store configuration. There's no technical, legal or even philosophical reason why this can't or shouldn't be done.

 

If I tried to hand my customers files instead of software under configuration management, I'd be out of business. Look... I'm not going to stand here and make the case for SCM. We are all seasoned professionals. Its time for Cincom to acknowledge their responsibilities in this area. '

 

I'd second the request, something I first asked for in 5i.3 :-). Alan says "procedural, logistical and financial", so let's look at those.
 
"Procedural" - you mean someone at Cincom would have to publish the base, or more likely replicate the release pundle lineup. Currently, every customer wanting to see VW version history in Store, or wanting to know what they've accidentally changed in base VW packages, needs to do that. We've even partly automated the reconcile + publish process, and I'm sure you could do better.
 
"Logistical" - you mean Cincom has to find a way to distribute the published repository. An SQLite file sounds like an obvious solution. A method in the base to script the reconcile + publish would of course require zero effort to distribute.
 
"Financial" - there are no direct costs involved, only time: tell us how long it will take you in hours. We already spend something like that number of hours, but without the certainty that what we arrive at is "official". You even have the offer of using Adventa's (or MetaCase's) publish, if that helps.
 
Making the upgrade easier will also lower the perceived barrier to upgrading to a new VW version. The time and annoyance of having to do the reconcile and publish ourselves are one thing. Another is the worry that in a partially manual and hackily automated process, we may cause ourselves versioning problems. With a correct reconcile + publish, it's easy to check for potential problems by comparing with the previous version, and of course it's a major help in debugging such problems.


I think there's more to it than that. There are, for example, some interesting issues about what exactly we're exporting. So, presumably we'd want to be creating such an export with each weekly build, and with release candidates. I'd certainly hate to have something that we only did once a release and expected to get precisely right that one time. So then what are exporting? Do we use our own internal version numbers or something else? What's the parent version? When we release, I think what customers would ideally like is to have version numbers like "7.6" whose parent versions are like "7.5". So should we do an export that warps the version numbers and the parents to that? And if we do, what do we do in the weekly builds? Are their parents the previously weekly builds, or are they the previous released version? And do release candidates then each provide a set of versions named 7.6, but which are different? Or should we just expose our internal names (These currently tend to look something like 7.6jan08.3 3, reflecting the build they're targeted at, plus an increment. But things that haven't changed don't get re-published, so may have version numbers from different builds, or if they haven't been changed in this release, previous releases. Our internal names also aren't always consistent). Should we change our internal naming convention to be something that's more convenient for customers, and if so, what would it be?

There's also the issue of support. If we're putting something like this into supported status, then that implies also putting into supported status and maintaining various other things. Assuming SQLLite worked out, it means a SQLLite EXDI layer, Store for SQLLite, some form of Store Replication (at least to get it from that repository into the customers, and probably to get it into that one in the first place), and the specific bits of that required for SQLLite, if any. That's not a trivial amount of stuff. If SQLLite is not supported on one of our supported platforms (how's their SGI support? MacOS9?) then that's also an issue. It's possible we could ignore that on the assumption that everyone's got a machine around somewhere that can talk to both SQLLite and their store database, but that's an interesting decision. Or we could say that we're distributing this, but not in a supported way - again, an interesting product management type of decision.

Aside: I'm assuming we'd want to replicate because I think we'd want to do this for the full set of things we're distributing (or at least the full set of supported things), not just for the things that are in the standard base image, so reconciling and publishing with regular Store, which requires things to be loaded into the image, seems less good. There are also alternative ways of distributing it. One would be as ExportDatabase files. But that needs some work, might have some scaling issues for large databases, and relies on John Brant's Replicator for its underlying support. Another would be, rather than distributing a file, to give customers access to a Store database online with the information in it and let them replicate it however they want. That would still require replication, but might address some of the issues of version naming. We don't put the versions into that database until we have a shipped release, and we rename them to correspond to the release at that point. It could also just be a Postgresql database, removing the issue of supporting another one (although SQLLite might be useful in its own right as a portable starter database for Store). And we could easily make past versions available - handling the parentage if we only ship one release's versions in the distribution is an interesting question.

So, I think this is a good idea, I think it would be valuable to customers (and a good number of customers seem to strongly agree) and I've spent time thinking about how to implement it previously. But it's also not trivial, and I don't even have a good idea of how many hours it would take to do it. I don't think that access to someone else's publish of the base is all that helpful. We do have our own. The issues are mostly around exactly what and how to distribute, and how this compares to all the other things that customers think we ought to be doing.

One of the responders asked how people could help. The most obvious way, of course, is by giving us money. It makes it ever so much easier to find time and money when people give us the latter. The more the better :-) However, barring that, useful suggestions about how people would want to receive such a distribution and what they would want it to look like are probably the most valuable. And writing code is good - e.g., if someone wanted to write "rename and reparent while replicating" functionality. But that's assuming that we knew exactly what we wanted to do.


--
Alan Knight [|], Cincom Smalltalk Development
Reply | Threaded
Open this post in threaded view
|

RE: SCM'ed VW

Steven Kelly
In reply to this post by Charles Adams
Message
Thanks Alan,
 
I think doing the export for weekly builds is overkill, because of the version numbering issue. I imagine most VW users aren't in the vw-dev program anyway, or at least aren't actively downloading each week. It would be enough to test the export in the release candidates in vw-dev, with the warning not to replicate them to another db. Actually, there seems no point replicating to another db in any case, for users on Windows: just keep the VW packages linked only to the local SQLite Cincom db, which can even be used by multiple users (all access would be read-only). (BTW A nice addition to Store would be that if a package in the image doesn't have version info for the currently selected repository, but does for one other, it would automatically use that repository for compares etc.)
Assuming SQLLite worked out, it means a SQLLite EXDI layer, Store for SQLLite ...
Would an SQLite ODBC driver help, meaning you could just use the existing ODBCEXDI? There's one at <http://www.ch-werner.de/sqliteodbc/> for Windows, Linux and Mac OS X. StoreForSQLLite would presumably then be rather similar to StoreForAccess - and in any case StoreFor* doesn't look like a huge amount of work (nor should it be), mostly 1-2 line methods overriding subclassResponsibilities. (Ah, how easy this is to say when I've never tried it!)
Another would be, rather than distributing a file, to give customers access to a Store database online with the information in it and let them replicate it however they want.
Not an option! Access to the public repository is painfully slow, even after the improvements in newer VW versions and the upgrade to the server. The problem is presumably network latency, since there's plenty of bandwidth at both ends of the connection (as seen when downloading via HTTP or FTP). Even if that could be improved by custom queries for this replication, I'd imagine there'd be quite a load on your server when a new version is released. The server can easily fill its bandwidth for FTP or HTTP, but for SQL queries I imagine the processor would become the bottleneck.
 
Cheers,
Steve
-----Original Message-----
From: Alan Knight [mailto:[hidden email]]
Sent: 23. tammikuuta 2008 17:22
To: Steven Kelly; Charles Adams; [hidden email]
Subject: RE: SCM'ed VW

At 03:22 AM 1/23/2008, Steven Kelly wrote:
Hi,
From: Alan Knight [[hidden email]]
Indeed, the reasons are all procedural, logistical, and financial :-)

At 03:52 PM 1/22/2008, Charles Adams wrote:
With all the talk about Store and the best database product to use, its time, once again, to bring up the issue of a published distribution. It is long past time for Cincom to produce versioned code in some sort of Store configuration. There's no technical, legal or even philosophical reason why this can't or shouldn't be done.

 
If I tried to hand my customers files instead of software under configuration management, I'd be out of business. Look... I'm not going to stand here and make the case for SCM. We are all seasoned professionals. Its time for Cincom to acknowledge their responsibilities in this area. '

I'd second the request, something I first asked for in 5i.3 :-). Alan says "procedural, logistical and financial", so let's look at those.
 
"Procedural" - you mean someone at Cincom would have to publish the base, or more likely replicate the release pundle lineup. Currently, every customer wanting to see VW version history in Store, or wanting to know what they've accidentally changed in base VW packages, needs to do that. We've even partly automated the reconcile + publish process, and I'm sure you could do better.
 
"Logistical" - you mean Cincom has to find a way to distribute the published repository. An SQLite file sounds like an obvious solution. A method in the base to script the reconcile + publish would of course require zero effort to distribute.
 
"Financial" - there are no direct costs involved, only time: tell us how long it will take you in hours. We already spend something like that number of hours, but without the certainty that what we arrive at is "official". You even have the offer of using Adventa's (or MetaCase's) publish, if that helps.
 
Making the upgrade easier will also lower the perceived barrier to upgrading to a new VW version. The time and annoyance of having to do the reconcile and publish ourselves are one thing. Another is the worry that in a partially manual and hackily automated process, we may cause ourselves versioning problems. With a correct reconcile + publish, it's easy to check for potential problems by comparing with the previous version, and of course it's a major help in debugging such problems.

I think there's more to it than that. There are, for example, some interesting issues about what exactly we're exporting. So, presumably we'd want to be creating such an export with each weekly build, and with release candidates. I'd certainly hate to have something that we only did once a release and expected to get precisely right that one time. So then what are exporting? Do we use our own internal version numbers or something else? What's the parent version? When we release, I think what customers would ideally like is to have version numbers like "7.6" whose parent versions are like "7.5". So should we do an export that warps the version numbers and the parents to that? And if we do, what do we do in the weekly builds? Are their parents the previously weekly builds, or are they the previous released version? And do release candidates then each provide a set of versions named 7.6, but which are different? Or should we just expose our internal names (These currently tend to look something like 7.6jan08.3 3, reflecting the build they're targeted at, plus an increment. But things that haven't changed don't get re-published, so may have version numbers from different builds, or if they haven't been changed in this release, previous releases. Our internal names also aren't always consistent). Should we change our internal naming convention to be something that's more convenient for customers, and if so, what would it be?

There's also the issue of support. If we're putting something like this into supported status, then that implies also putting into supported status and maintaining various other things. Assuming SQLLite worked out, it means a SQLLite EXDI layer, Store for SQLLite, some form of Store Replication (at least to get it from that repository into the customers, and probably to get it into that one in the first place), and the specific bits of that required for SQLLite, if any. That's not a trivial amount of stuff. If SQLLite is not supported on one of our supported platforms (how's their SGI support? MacOS9?) then that's also an issue. It's possible we could ignore that on the assumption that everyone's got a machine around somewhere that can talk to both SQLLite and their store database, but that's an interesting decision. Or we could say that we're distributing this, but not in a supported way - again, an interesting product management type of decision.

Aside: I'm assuming we'd want to replicate because I think we'd want to do this for the full set of things we're distributing (or at least the full set of supported things), not just for the things that are in the standard base image, so reconciling and publishing with regular Store, which requires things to be loaded into the image, seems less good. There are also alternative ways of distributing it. One would be as ExportDatabase files. But that needs some work, might have some scaling issues for large databases, and relies on John Brant's Replicator for its underlying support. Another would be, rather than distributing a file, to give customers access to a Store database online with the information in it and let them replicate it however they want. That would still require replication, but might address some of the issues of version naming. We don't put the versions into that database until we have a shipped release, and we rename them to correspond to the release at that point. It could also just be a Postgresql database, removing the issue of supporting another one (although SQLLite might be useful in its own right as a portable starter database for Store). And we could easily make past versions available - handling the parentage if we only ship one release's versions in the distribution is an interesting question.

So, I think this is a good idea, I think it would be valuable to customers (and a good number of customers seem to strongly agree) and I've spent time thinking about how to implement it previously. But it's also not trivial, and I don't even have a good idea of how many hours it would take to do it. I don't think that access to someone else's publish of the base is all that helpful. We do have our own. The issues are mostly around exactly what and how to distribute, and how this compares to all the other things that customers think we ought to be doing.

One of the responders asked how people could help. The most obvious way, of course, is by giving us money. It makes it ever so much easier to find time and money when people give us the latter. The more the better :-) However, barring that, useful suggestions about how people would want to receive such a distribution and what they would want it to look like are probably the most valuable. And writing code is good - e.g., if someone wanted to write "rename and reparent while replicating" functionality. But that's assuming that we knew exactly what we wanted to do.



--
Alan Knight [|], Cincom Smalltalk Development
Reply | Threaded
Open this post in threaded view
|

Re: SCM'ed VW

Tim Hutchison
In reply to this post by Alan Knight-2
Currently in the Cincom Public Repository;
SQLite3EXDI-Preload (sqlite3-vw7.x-2,olaf)
SQLite3EXDI (sqlite3-vw7.x-2,olaf)
StoreForSQLite3 (sqlite3-vw7.x-2,olaf)

Could be that these may need a little more work. 

Alan Knight wrote:
At 03:22 AM 1/23/2008, Steven Kelly wrote:
Hi,
From: Alan Knight [[hidden email]]
Indeed, the reasons are all procedural, logistical, and financial :-)

At 03:52 PM 1/22/2008, Charles Adams wrote:
With all the talk about Store and the best database product to use, its time, once again, to bring up the issue of a published distribution. It is long past time for Cincom to produce versioned code in some sort of Store configuration. There's no technical, legal or even philosophical reason why this can't or shouldn't be done.
 
If I tried to hand my customers files instead of software under configuration management, I'd be out of business. Look... I'm not going to stand here and make the case for SCM. We are all seasoned professionals. Its time for Cincom to acknowledge their responsibilities in this area. '

I'd second the request, something I first asked for in 5i.3 :-). Alan says "procedural, logistical and financial", so let's look at those.
 
"Procedural" - you mean someone at Cincom would have to publish the base, or more likely replicate the release pundle lineup. Currently, every customer wanting to see VW version history in Store, or wanting to know what they've accidentally changed in base VW packages, needs to do that. We've even partly automated the reconcile + publish process, and I'm sure you could do better.
 
"Logistical" - you mean Cincom has to find a way to distribute the published repository. An SQLite file sounds like an obvious solution. A method in the base to script the reconcile + publish would of course require zero effort to distribute.
 
"Financial" - there are no direct costs involved, only time: tell us how long it will take you in hours. We already spend something like that number of hours, but without the certainty that what we arrive at is "official". You even have the offer of using Adventa's (or MetaCase's) publish, if that helps.
 
Making the upgrade easier will also lower the perceived barrier to upgrading to a new VW version. The time and annoyance of having to do the reconcile and publish ourselves are one thing. Another is the worry that in a partially manual and hackily automated process, we may cause ourselves versioning problems. With a correct reconcile + publish, it's easy to check for potential problems by comparing with the previous version, and of course it's a major help in debugging such problems.

I think there's more to it than that. There are, for example, some interesting issues about what exactly we're exporting. So, presumably we'd want to be creating such an export with each weekly build, and with release candidates. I'd certainly hate to have something that we only did once a release and expected to get precisely right that one time. So then what are exporting? Do we use our own internal version numbers or something else? What's the parent version? When we release, I think what customers would ideally like is to have version numbers like "7.6" whose parent versions are like "7.5". So should we do an export that warps the version numbers and the parents to that? And if we do, what do we do in the weekly builds? Are their parents the previously weekly builds, or are they the previous released version? And do release candidates then each provide a set of versions named 7.6, but which are different? Or should we just expose our internal names (These currently tend to look something like 7.6jan08.3 3, reflecting the build they're targeted at, plus an increment. But things that haven't changed don't get re-published, so may have version numbers from different builds, or if they haven't been changed in this release, previous releases. Our internal names also aren't always consistent). Should we change our internal naming convention to be something that's more convenient for customers, and if so, what would it be?

There's also the issue of support. If we're putting something like this into supported status, then that implies also putting into supported status and maintaining various other things. Assuming SQLLite worked out, it means a SQLLite EXDI layer, Store for SQLLite, some form of Store Replication (at least to get it from that repository into the customers, and probably to get it into that one in the first place), and the specific bits of that required for SQLLite, if any. That's not a trivial amount of stuff. If SQLLite is not supported on one of our supported platforms (how's their SGI support? MacOS9?) then that's also an issue. It's possible we could ignore that on the assumption that everyone's got a machine around somewhere that can talk to both SQLLite and their store database, but that's an interesting decision. Or we could say that we're distributing this, but not in a supported way - again, an interesting product management type of decision.

Aside: I'm assuming we'd want to replicate because I think we'd want to do this for the full set of things we're distributing (or at least the full set of supported things), not just for the things that are in the standard base image, so reconciling and publishing with regular Store, which requires things to be loaded into the image, seems less good. There are also alternative ways of distributing it. One would be as ExportDatabase files. But that needs some work, might have some scaling issues for large databases, and relies on John Brant's Replicator for its underlying support. Another would be, rather than distributing a file, to give customers access to a Store database online with the information in it and let them replicate it however they want. That would still require replication, but might address some of the issues of version naming. We don't put the versions into that database until we have a shipped release, and we rename them to correspond to the release at that point. It could also just be a Postgresql database, removing the issue of supporting another one (although SQLLite might be useful in its own right as a portable starter database for Store). And we could easily make past versions available - handling the parentage if we only ship one release's versions in the distribution is an interesting question.

So, I think this is a good idea, I think it would be valuable to customers (and a good number of customers seem to strongly agree) and I've spent time thinking about how to implement it previously. But it's also not trivial, and I don't even have a good idea of how many hours it would take to do it. I don't think that access to someone else's publish of the base is all that helpful. We do have our own. The issues are mostly around exactly what and how to distribute, and how this compares to all the other things that customers think we ought to be doing.

One of the responders asked how people could help. The most obvious way, of course, is by giving us money. It makes it ever so much easier to find time and money when people give us the latter. The more the better :-) However, barring that, useful suggestions about how people would want to receive such a distribution and what they would want it to look like are probably the most valuable. And writing code is good - e.g., if someone wanted to write "rename and reparent while replicating" functionality. But that's assuming that we knew exactly what we wanted to do.



--
Alan Knight [|], Cincom Smalltalk Development
Reply | Threaded
Open this post in threaded view
|

RE: SCM'ed VW

Alan Knight-2
In reply to this post by Steven Kelly
At 11:30 AM 1/23/2008, Steven Kelly wrote:
Thanks Alan,
 
I think doing the export for weekly builds is overkill, because of the version numbering issue. I imagine most VW users aren't in the vw-dev program anyway, or at least aren't actively downloading each week. It would be enough to test the export in the release candidates in vw-dev, with the warning not to replicate them to another db. Actually, there seems no point replicating to another db in any case, for users on Windows: just keep the VW packages linked only to the local SQLite Cincom db, which can even be used by multiple users (all access would be read-only). (BTW A nice addition to Store would be that if a package in the image doesn't have version info for the currently selected repository, but does for one other, it would automatically use that repository for compares etc.)

I don't know that that would work for all use cases. For example, one use of this that I'd see is if an application modified base packages. I certainly wouldn't want to publish my modifications into the SQLLite DB, and I wouldn't want to have the reference version in one DB and my modified version in a different one, as I then wouldn't be able to easily compare them.

And doing things once a release cycle, or only finding out that they don't work as we begin to make release candidates makes me quite nervous. I think I'd prefer to do it for each build and those who want to use it get internal version names, just as they're getting internal everything else with those builds.

Assuming SQLLite worked out, it means a SQLLite EXDI layer, Store for SQLLite ...

Would an SQLite ODBC driver help, meaning you could just use the existing ODBCEXDI? There's one at < http://www.ch-werner.de/sqliteodbc/> for Windows, Linux and Mac OS X. StoreForSQLLite would presumably then be rather similar to StoreForAccess - and in any case StoreFor* doesn't look like a huge amount of work (nor should it be), mostly 1-2 line methods overriding subclassResponsibilities. (Ah, how easy this is to say when I've never tried it!)

No, I suspect ODBC would be worse. It's pretty Windows-specific, and we always seem to get reports from people having problems trying to use ODBC on other platforms.

StoreForX isn't all that big a deal. It's just those few tiny subtle methods to deal with the interesting SQL  variations on that particular databases, and with what might go wrong if they're not correct. So it's not a monumental task, but it's not trivial either.

Another would be, rather than distributing a file, to give customers access to a Store database online with the information in it and let them replicate it however they want.

Not an option! Access to the public repository is painfully slow, even after the improvements in newer VW versions and the upgrade to the server. The problem is presumably network latency, since there's plenty of bandwidth at both ends of the connection (as seen when downloading via HTTP or FTP). Even if that could be improved by custom queries for this replication, I'd imagine there'd be quite a load on your server when a new version is released. The server can easily fill its bandwidth for FTP or HTTP, but for SQL queries I imagine the processor would become the bottleneck.

Well, my expectation would be for people to replicate the reference versions, at least those they were interested in, into their local repository. If you're replicating, and you do it once per release, it doesn't really matter how fast it is. Plus the replicator is significantly faster on high-latency connections than regular Store is. And yes, it'd be possible to make Store do better that way, and that's exactly one of the trade-offs we'd have to look at - is it more important to do that or to do this.

And if we were doing this seriously, it's not that difficult to supply additional compute power/bandwidth pretty economically these days, though that does start to run into the logistical issues of corporate approval.

 
Cheers,
Steve

-----Original Message-----
From: Alan Knight [[hidden email]]
Sent: 23. tammikuuta 2008 17:22
To: Steven Kelly; Charles Adams; [hidden email]
Subject: RE: SCM'ed VW

At 03:22 AM 1/23/2008, Steven Kelly wrote:
Hi,
From: Alan Knight [ [hidden email] ]
Indeed, the reasons are all procedural, logistical, and financial :-)
At 03:52 PM 1/22/2008, Charles Adams wrote:
With all the talk about Store and the best database product to use, its time, once again, to bring up the issue of a published distribution. It is long past time for Cincom to produce versioned code in some sort of Store configuration. There's no technical, legal or even philosophical reason why this can't or shouldn't be done.
 
If I tried to hand my customers files instead of software under configuration management, I'd be out of business. Look... I'm not going to stand here and make the case for SCM. We are all seasoned professionals. Its time for Cincom to acknowledge their responsibilities in this area. '
I'd second the request, something I first asked for in 5i.3 :-). Alan says "procedural, logistical and financial", so let's look at those.
 
"Procedural" - you mean someone at Cincom would have to publish the base, or more likely replicate the release pundle lineup. Currently, every customer wanting to see VW version history in Store, or wanting to know what they've accidentally changed in base VW packages, needs to do that. We've even partly automated the reconcile + publish process, and I'm sure you could do better.
 
"Logistical" - you mean Cincom has to find a way to distribute the published repository. An SQLite file sounds like an obvious solution. A method in the base to script the reconcile + publish would of course require zero effort to distribute.
 
"Financial" - there are no direct costs involved, only time: tell us how long it will take you in hours. We already spend something like that number of hours, but without the certainty that what we arrive at is "official". You even have the offer of using Adventa's (or MetaCase's) publish, if that helps.
 
Making the upgrade easier will also lower the perceived barrier to upgrading to a new VW version. The time and annoyance of having to do the reconcile and publish ourselves are one thing. Another is the worry that in a partially manual and hackily automated process, we may cause ourselves versioning problems. With a correct reconcile + publish, it's easy to check for potential problems by comparing with the previous version, and of course it's a major help in debugging such problems.

I think there's more to it than that. There are, for example, some interesting issues about what exactly we're exporting. So, presumably we'd want to be creating such an export with each weekly build, and with release candidates. I'd certainly hate to have something that we only did once a release and expected to get precisely right that one time. So then what are exporting? Do we use our own internal version numbers or something else? What's the parent version? When we release, I think what customers would ideally like is to have version numbers like "7.6" whose parent versions are like "7.5". So should we do an export that warps the version numbers and the parents to that? And if we do, what do we do in the weekly builds? Are their parents the previously weekly builds, or are they the previous released version? And do release candidates then each provide a set of versions named 7.6, but which are different? Or should we just expose our internal names (These currently tend to look something like 7.6jan08.3 3, reflecting the build they're targeted at, plus an increment. But things that haven't changed don't get re-published, so may have version numbers from different builds, or if they haven't been changed in this release, previous releases. Our internal names also aren't always consistent). Should we change our internal naming convention to be something that's more convenient for customers, and if so, what would it be?

There's also the issue of support. If we're putting something like this into supported status, then that implies also putting into supported status and maintaining various other things. Assuming SQLLite worked out, it means a SQLLite EXDI layer, Store for SQLLite, some form of Store Replication (at least to get it from that repository into the customers, and probably to get it into that one in the first place), and the specific bits of that required for SQLLite, if any. That's not a trivial amount of stuff. If SQLLite is not supported on one of our supported platforms (how's their SGI support? MacOS9?) then that's also an issue. It's possible we could ignore that on the assumption that everyone's got a machine around somewhere that can talk to both SQLLite and their store database, but that's an interesting decision. Or we could say that we're distributing this, but not in a supported way - again, an interesting product management type of decision.

Aside: I'm assuming we'd want to replicate because I think we'd want to do this for the full set of things we're distributing (or at least the full set of supported things), not just for the things that are in the standard base image, so reconciling and publishing with regular Store, which requires things to be loaded into the image, seems less good. There are also alternative ways of distributing it. One would be as ExportDatabase files. But that needs some work, might have some scaling issues for large databases, and relies on John Brant's Replicator for its underlying support. Another would be, rather than distributing a file, to give customers access to a Store database online with the information in it and let them replicate it however they want. That would still require replication, but might address some of the issues of version naming. We don't put the versions into that database until we have a shipped release, and we rename them to correspond to the release at that point. It could also just be a Postgresql database, removing the issue of supporting another one (although SQLLite might be useful in its own right as a portable starter database for Store). And we could easily make past versions available - handling the parentage if we only ship one release's versions in the distribution is an interesting question.

So, I think this is a good idea, I think it would be valuable to customers (and a good number of customers seem to strongly agree) and I've spent time thinking about how to implement it previously. But it's also not trivial, and I don't even have a good idea of how many hours it would take to do it. I don't think that access to someone else's publish of the base is all that helpful. We do have our own. The issues are mostly around exactly what and how to distribute, and how this compares to all the other things that customers think we ought to be doing.

One of the responders asked how people could help. The most obvious way, of course, is by giving us money. It makes it ever so much easier to find time and money when people give us the latter. The more the better :-) However, barring that, useful suggestions about how people would want to receive such a distribution and what they would want it to look like are probably the most valuable. And writing code is good - e.g., if someone wanted to write "rename and reparent while replicating" functionality. But that's assuming that we knew exactly what we wanted to do.



--
Alan Knight [|], Cincom Smalltalk Development
[hidden email]
[hidden email]
http://www.cincom.com/smalltalk

--
Alan Knight [|], Cincom Smalltalk Development
Reply | Threaded
Open this post in threaded view
|

Re: SCM'ed VW

Alan Knight-2
In reply to this post by Stefan Schmiedl
At 11:17 AM 1/23/2008, Stefan Schmiedl wrote:
On Wed, 23 Jan 2008 10:21:30 -0500
Alan Knight <[hidden email]> wrote:

> When we release, I think what customers would ideally like is
> to have version numbers like "7.6" whose parent versions are
> like "7.5". So should we do an export that warps the version
> numbers and the parents to that?

Indepently of the issue at hand, I'd *very much* appreciate this.
There are parcels in the distribution with a seemingly outdated
version number. How do I know if the parcel is still intended to
be used? Was it introduced in 7.2, last modified for 7.2, is it
still useful in 7.6?

If the parcel is in a supported area, then it's intended to be used. If it wasn't, we'd remove it. If the parcel hasn't changed since 7.2, then we leave it with that version number.

For parcels that are contributed, we don't control them, and it's up to the author/maintainer. We'll remove things if we notice, or it's brought to our attention, that they aren't being maintained and no longer work in current versions.

Parcels in preview can sometimes be difficult to determine.

Also I remember being told to use a more recent version from the
public store for this package, while for that package the
parcelled version is more recommended.

That's a general problem. Sometimes when things are updated, they are updated to require newer versions, so you're recommended to use the parcel version if you're on an older version. And sometimes newer is pre-release. Sometimes when things are updated, they don't require newer versions of the base, but just fix bugs, and if the bugs are serious, you're recommended to use the version from the public Store. I don't know what would be a solution for that problem, other than checking the public Store and reading the version comments.

> Should we change our internal naming convention to be something
> that's more convenient for customers, and if so, what would it
> be?

Here's what I'm doing in my personal Store: Spikes are numbered
with integers, they're not for keeping anyways. All "real" code
is organized in a bundle, which is what I'm actively publishing.
Also I'm using the bundle's current version for all modified
packages, so I end up with a continuously numbered bundle and
"jumpy" version numbers for the packages within then, as modified
packages always have the bundle's version number.

> There's also the issue of support.

Actually, there's a disadvantage with using an (any) external
library for my mostly linux based development. I'm usually
running a 32-bit image (so that I can continue working on
my Windows laptop) within my 64-bit linux system. Every external
library I intend to use, I have to keep and maintain as a 32-bit
version separate from the rest. But I guess that's just me, so
not really important to the world at large.

> Or we could say that we're distributing this, but not in a
> supported way - again, an interesting product management type
> of decision.

Yeah, well. That brings up a sore point from the current history,
when Widgetry was released as "fully supported" and then
withdrawn a few weeks after that. So "distributing, but not in
a supported way" can't be much worse.

> However, barring that, useful suggestions about how people
> would want to receive such a distribution and what they would
> want it to look like are probably the most valuable.

I don't know about corporate people having to deal with IT
department regulations, but as a one-man company, I am very
comfortable downloading such things.

For starters, how about providing a Store containing the baseline
of the base image, where everything has the version of the build?

I'm sure you can think of reasons why this is too simplistic, so
let's discuss.

Hm. Wait. Discussion might take place during working hours... and
too much of it might prevent them from implementing a solution
:-)

s.

--
Alan Knight [|], Cincom Smalltalk Development
Reply | Threaded
Open this post in threaded view
|

RE: SCM'ed VW

Alan Knight-2
In reply to this post by Terry Raymond
At 11:26 AM 1/23/2008, Terry Raymond wrote:
Alan
 
With respect to VW-Dev builds, the package versions should reflect
the build version in which they were changed.  For example,
suppose Assets was last modifed in October and released in the
waldo oct07.2 build then its latest version would be oct07.2, even
in the waldo jan08.2 release.
 
When VW actually ships, all the package versions would be updated
to the shipped version, i.e. 7.6 or 7.6.0.
 
Additionally, your discussion points out the need for Store to better
trace component heritage, particularly when you consider replication.

I think you would need to define "better".


 

Terry

===========================================================
Terry Raymond
Crafted Smalltalk
80 Lazywood Ln.
Tiverton, RI  02878
(401) 624-4517      [hidden email]
< http://www.craftedsmalltalk.com>
===========================================================

From: Alan Knight [[hidden email]]
Sent: Wednesday, January 23, 2008 10:22 AM
To: Steven Kelly; Charles Adams; [hidden email]
Subject: RE: SCM'ed VW
 
At 03:22 AM 1/23/2008, Steven Kelly wrote:

Hi,
From: Alan Knight [ [hidden email] ]
Indeed, the reasons are all procedural, logistical, and financial :-)
At 03:52 PM 1/22/2008, Charles Adams wrote:

With all the talk about Store and the best database product to use, its time, once again, to bring up the issue of a published distribution. It is long past time for Cincom to produce versioned code in some sort of Store configuration. There's no technical, legal or even philosophical reason why this can't or shouldn't be done.
 
If I tried to hand my customers files instead of software under configuration management, I'd be out of business. Look... I'm not going to stand here and make the case for SCM. We are all seasoned professionals. Its time for Cincom to acknowledge their responsibilities in this area. '
 
I'd second the request, something I first asked for in 5i.3 :-). Alan says "procedural, logistical and financial", so let's look at those.
 
"Procedural" - you mean someone at Cincom would have to publish the base, or more likely replicate the release pundle lineup. Currently, every customer wanting to see VW version history in Store, or wanting to know what they've accidentally changed in base VW packages, needs to do that. We've even partly automated the reconcile + publish process, and I'm sure you could do better.
 
"Logistical" - you mean Cincom has to find a way to distribute the published repository. An SQLite file sounds like an obvious solution. A method in the base to script the reconcile + publish would of course require zero effort to distribute.
 
"Financial" - there are no direct costs involved, only time: tell us how long it will take you in hours. We already spend something like that number of hours, but without the certainty that what we arrive at is "official". You even have the offer of using Adventa's (or MetaCase's) publish, if that helps.
 
Making the upgrade easier will also lower the perceived barrier to upgrading to a new VW version. The time and annoyance of having to do the reconcile and publish ourselves are one thing. Another is the worry that in a partially manual and hackily automated process, we may cause ourselves versioning problems. With a correct reconcile + publish, it's easy to check for potential problems by comparing with the previous version, and of course it's a major help in debugging such problems.

I think there's more to it than that. There are, for example, some interesting issues about what exactly we're exporting. So, presumably we'd want to be creating such an export with each weekly build, and with release candidates. I'd certainly hate to have something that we only did once a release and expected to get precisely right that one time. So then what are exporting? Do we use our own internal version numbers or something else? What's the parent version? When we release, I think what customers would ideally like is to have version numbers like "7.6" whose parent versions are like "7.5". So should we do an export that warps the version numbers and the parents to that? And if we do, what do we do in the weekly builds? Are their parents the previously weekly builds, or are they the previous released version? And do release candidates then each provide a set of versions named 7.6, but which are different? Or should we just expose our internal names (These currently tend to look something like 7.6jan08.3 3, reflecting the build they're targeted at, plus an increment. But things that haven't changed don't get re-published, so may have version numbers from different builds, or if they haven't been changed in this release, previous releases. Our internal names also aren't always consistent). Should we change our internal naming convention to be something that's more convenient for customers, and if so, what would it be?

There's also the issue of support. If we're putting something like this into supported status, then that implies also putting into supported status and maintaining various other things. Assuming SQLLite worked out, it means a SQLLite EXDI layer, Store for SQLLite, some form of Store Replication (at least to get it from that repository into the customers, and probably to get it into that one in the first place), and the specific bits of that required for SQLLite, if any. That's not a trivial amount of stuff. If SQLLite is not supported on one of our supported platforms (how's their SGI support? MacOS9?) then that's also an issue. It's possible we could ignore that on the assumption that everyone's got a machine around somewhere that can talk to both SQLLite and their store database, but that's an interesting decision. Or we could say that we're distributing this, but not in a supported way - again, an interesting product management type of decision.

Aside: I'm assuming we'd want to replicate because I think we'd want to do this for the full set of things we're distributing (or at least the full set of supported things), not just for the things that are in the standard base image, so reconciling and publishing with regular Store, which requires things to be loaded into the image, seems less good. There are also alternative ways of distributing it. One would be as ExportDatabase files. But that needs some work, might have some scaling issues for large databases, and relies on John Brant's Replicator for its underlying support. Another would be, rather than distributing a file, to give customers access to a Store database online with the information in it and let them replicate it however they want. That would still require replication, but might address some of the issues of version naming. We don't put the versions into that database until we have a shipped release, and we rename them to correspond to the release at that point. It could also just be a Postgresql database, removing the issue of supporting another one (although SQLLite might be useful in its own right as a portable starter database for Store). And we could easily make past versions available - handling the parentage if we only ship one release's versions in the distribution is an interesting question.

So, I think this is a good idea, I think it would be valuable to customers (and a good number of customers seem to strongly agree) and I've spent time thinking about how to implement it previously. But it's also not trivial, and I don't even have a good idea of how many hours it would take to do it. I don't think that access to someone else's publish of the base is all that helpful. We do have our own. The issues are mostly around exactly what and how to distribute, and how this compares to all the other things that customers think we ought to be doing.

One of the responders asked how people could help. The most obvious way, of course, is by giving us money. It makes it ever so much easier to find time and money when people give us the latter. The more the better :-) However, barring that, useful suggestions about how people would want to receive such a distribution and what they would want it to look like are probably the most valuable. And writing code is good - e.g., if someone wanted to write "rename and reparent while replicating" functionality. But that's assuming that we knew exactly what we wanted to do.


--
Alan Knight [|], Cincom Smalltalk Development
[hidden email]
[hidden email]
http://www.cincom.com/smalltalk

--
Alan Knight [|], Cincom Smalltalk Development
Reply | Threaded
Open this post in threaded view
|

Re: SCM'ed VW

Charles Adams
Well, I must say I'm delighted by how much discussion my comments have spurred. But in the thrashing around of trying to rationalize this or orgainze that, may I make a suggestion: PICK SOMETHING. Make a management decision. Just do it.
 
 
Charles Adams   
Adventa Control Technologies, Inc.
3001 E. Plano Parkway, #100 
Plano, TX 75074-7422 
Office: 972.543.1688
FAX: 972.633.9317
http://www.adventact.com
[hidden email]
 
Reply | Threaded
Open this post in threaded view
|

RE: SCM'ed VW

Steven Kelly
In reply to this post by Charles Adams
Message
 
From: Alan Knight [mailto:[hidden email]]
Sent: 23. tammikuuta 2008 19:13
At 11:30 AM 1/23/2008, Steven Kelly wrote:
I think doing the export for weekly builds is overkill, because of the version numbering issue. I imagine most VW users aren't in the vw-dev program anyway, or at least aren't actively downloading each week. It would be enough to test the export in the release candidates in vw-dev, with the warning not to replicate them to another db. Actually, there seems no point replicating to another db in any case, for users on Windows: just keep the VW packages linked only to the local SQLite Cincom db, which can even be used by multiple users (all access would be read-only). (BTW A nice addition to Store would be that if a package in the image doesn't have version info for the currently selected repository, but does for one other, it would automatically use that repository for compares etc.)

I don't know that that would work for all use cases. For example, one use of this that I'd see is if an application modified base packages. I certainly wouldn't want to publish my modifications into the SQLLite DB, and I wouldn't want to have the reference version in one DB and my modified version in a different one, as I then wouldn't be able to easily compare them.
Our experience is that it makes little sense to modify base packages. We've tried modifying base packages in Envy (where it was the only option) and in Store from 5i.3 to 7.2, but have now moved to creating our own packages with extensions and overrides. Even official Cincom patches get versioned as new packages called AR50127 etc. This seems to work significantly better, in particular wrt. new releases from Cincom. One thing to avoid is sticking too many unrelated patches into a catch-all "MyBaseFixes" - if we have to support development on multiple VW versions at once, the version tree of that package becomes a bit of a nightmare.
 
The truth is that most extensions or overrides, and even bug fixes, don't appear immediately in the next release, so it's easier to keep them separate and assume they're still needed in the next version, than to assume our VisualWorks Base 7.5-1 changes will be subsumed into 7.6. If each conceptual change is in its own package, there can be a bundle MyBaseFixes that is versioned separately for each VW version (or even different bundles MyBaseFixes741 and MyBaseFixes75).
 
Since Cincom doesn't face this problem, it would be great if others who have tried both approaches would post their experiences.
And doing things once a release cycle, or only finding out that they don't work as we begin to make release candidates makes me quite nervous. I think I'd prefer to do it for each build and those who want to use it get internal version names, just as they're getting internal everything else with those builds. 
I think that just creates more work for you, and significant versioning issues for anyone in vw-dev who tries to use them. The version names would be following two different schemes, and the repository would have to contain all intermediate versions if it were to be replicated to customers' repositories. I doubt many would want to introduce that mass of intermediate versions to their real repositories, which would mean that the main benefit you're looking for in having published repositories for intermediate builds - to get them tested by vw-dev users - wouldn't happen.
 
It might be more sensible to allow vw-dev participants access to the Cincom internal db -- actually what I initially assumed the vw-dev repository was for. The image would be reconciled with that, so there'd be no initial cost when starting with a new version.
Well, my expectation would be for people to replicate the reference versions, at least those they were interested in, into their local repository. If you're replicating, and you do it once per release, it doesn't really matter how fast it is. Plus the replicator is significantly faster on high-latency connections than regular Store is. And yes, it'd be possible to make Store do better that way, and that's exactly one of the trade-offs we'd have to look at - is it more important to do that or to do this. 
If that's the case, and you're planning on high levels of automation to be able to build the weekly snapshot repositories, wouldn't it be easier to just add the reconcile+publish script to VW? Users could then use it on whatever database they have, and their would be no need to distribute a repository or add support for SQLite.
 
It sounds like there are two effective routes:
  1. just make the reconcile+publish script. People in vw-dev can use it if they want on weekly builds, and normal users can use it if they want on the full release. It would presumably run only on packages in the image: the developers will need a development image with the right packages anyway, so choosing them and loading them is no extra work for them.
  2. have a partially manual process during the release candidate phase to produce an SQLite repository, using Olaf's packages in a similar way to PostgreSQL. Since this would need to have all packages in it (there not being a good way for users to add packages), it would be big and duplicate the information from the parcels. Being partly manual, that's open to errors and differences between the final distro parcels. If you want to get rid of the errors, you need a fully automated script -- so move to 1. Similarly if you want to have the repository for weekly builds too.
So is there only one effective route, i.e. 1. above? There could of course still be SQLite support and instructions or an installation choice to set it up or reconcile into a previous version.
 
And Alan: once again, thanks!
Steve
Reply | Threaded
Open this post in threaded view
|

Re: SCM'ed VW - fiancial incetives

Mark Pirogovsky-3
In reply to this post by Alan Knight-2


Alan Knight wrote:
>   Indeed, the reasons are all procedural, logistical, and financial :-)

Financial impact -- let us see: (pardon me for stating obvious, but)

1. Cincom makes money when we, VW developers, do make a sale.

2. So far I see about dozen of people participating in this discussion.

3. Last time I did publish the VW base and most of other supported(!)
packages to the Store to make a baseline and base development images it
took me almost a day worth of doing. (other people may be luckier then
me...).
4. a day time  multiplied by number of people in this discussion makes
about 12 man days.  Cincom should know how many paid and NC users they
have, so Alan can do the math.
5. Instead of working on the product development, which makes money for
all involved, I(and many others) was spending time on the task which
could be avoided or greatly simplified.

6. As a result of all this we were one day behind of making a sale.

7. As a result Cincom got less money in the quarter (or year).

8. As a result some Cincom developers may not get their bonuses (that
one is pure speculation on my part)

:-)
Copyright notice: The above illustration is provided free of charge any
  interested parties at Cincom to convince their upper management that
it is worthwhile building SCM'ed VW (using whatever technology they
think appropriate, such as  SQLLite file, MS Access file,  PostgreSQL
exports, Oracle dump, etc ...)




Reply | Threaded
Open this post in threaded view
|

Re: SCM'ed VW

Dave Stevenson-2
In reply to this post by Steven Kelly
 > ... the main benefit you're looking for in having published
repositories for intermediate builds - to get them tested by vw-dev
users - wouldn't happen.

Incorrect assumption. The main reasons we'd want to do this for
intermediate builds are:
        - to make them more readily available for users of weekly builds,
whether internal or external. I.E., it would be easier for me as a
remote internal user to replicated locally
        - to ensure the automated process that generates them is working
seemlessly, so unexpected glitches or delays are not introduced at the
release candidate stage

Dave

Steven Kelly wrote:

>  
>
>     *From:* Alan Knight [mailto:[hidden email]]
>     *Sent:* 23. tammikuuta 2008 19:13
>     At 11:30 AM 1/23/2008, Steven Kelly wrote:
>>     I think doing the export for weekly builds is overkill, because of
>>     the version numbering issue. I imagine most VW users aren't in the
>>     vw-dev program anyway, or at least aren't actively downloading
>>     each week. It would be enough to test the export in the release
>>     candidates in vw-dev, with the warning not to replicate them to
>>     another db. Actually, there seems no point replicating to another
>>     db in any case, for users on Windows: just keep the VW packages
>>     linked only to the local SQLite Cincom db, which can even be used
>>     by multiple users (all access would be read-only). (BTW A nice
>>     addition to Store would be that if a package in the image doesn't
>>     have version info for the currently selected repository, but does
>>     for one other, it would automatically use that repository for
>>     compares etc.)
>
>     I don't know that that would work for all use cases. For example,
>     one use of this that I'd see is if an application modified base
>     packages. I certainly wouldn't want to publish my modifications into
>     the SQLLite DB, and I wouldn't want to have the reference version in
>     one DB and my modified version in a different one, as I then
>     wouldn't be able to easily compare them.
>
> Our experience is that it makes little sense to modify base packages.
> We've tried modifying base packages in Envy (where it was the only
> option) and in Store from 5i.3 to 7.2, but have now moved to creating
> our own packages with extensions and overrides. Even official Cincom
> patches get versioned as new packages called AR50127 etc. This seems to
> work significantly better, in particular wrt. new releases from Cincom.
> One thing to avoid is sticking too many unrelated patches into a
> catch-all "MyBaseFixes" - if we have to support development on multiple
> VW versions at once, the version tree of that package becomes a bit of a
> nightmare.
>  
> The truth is that most extensions or overrides, and even bug fixes,
> don't appear immediately in the next release, so it's easier to keep
> them separate and assume they're still needed in the next version, than
> to assume our VisualWorks Base 7.5-1 changes will be subsumed into 7.6.
> If each conceptual change is in its own package, there can be a bundle
> MyBaseFixes that is versioned separately for each VW version (or even
> different bundles MyBaseFixes741 and MyBaseFixes75).
>  
> Since Cincom doesn't face this problem, it would be great if others who
> have tried both approaches would post their experiences.
>
>     And doing things once a release cycle, or only finding out that they
>     don't work as we begin to make release candidates makes me quite
>     nervous. I think I'd prefer to do it for each build and those who
>     want to use it get internal version names, just as they're getting
>     internal everything else with those builds.
>
> I think that just creates more work for you, and significant versioning
> issues for anyone in vw-dev who tries to use them. The version names
> would be following two different schemes, and the repository would have
> to contain all intermediate versions if it were to be replicated to
> customers' repositories. I doubt many would want to introduce that mass
> of intermediate versions to their real repositories, which would mean
> that the main benefit you're looking for in having published
> repositories for intermediate builds - to get them tested by vw-dev
> users - wouldn't happen.
>  
> It might be more sensible to allow vw-dev participants access to the
> Cincom internal db -- actually what I initially assumed the vw-dev
> repository was for. The image would be reconciled with that, so there'd
> be no initial cost when starting with a new version.
>
>     Well, my expectation would be for people to replicate the reference
>     versions, at least those they were interested in, into their local
>     repository. If you're replicating, and you do it once per release,
>     it doesn't really matter how fast it is. Plus the replicator is
>     significantly faster on high-latency connections than regular Store
>     is. And yes, it'd be possible to make Store do better that way, and
>     that's exactly one of the trade-offs we'd have to look at - is it
>     more important to do that or to do this.
>
> If that's the case, and you're planning on high levels of automation to
> be able to build the weekly snapshot repositories, wouldn't it be easier
> to just add the reconcile+publish script to VW? Users could then use it
> on whatever database they have, and their would be no need to distribute
> a repository or add support for SQLite.
>  
> It sounds like there are two effective routes:
>
>    1.
>       just make the reconcile+publish script. People in vw-dev can use
>       it if they want on weekly builds, and normal users can use it if
>       they want on the full release. It would presumably run only on
>       packages in the image: the developers will need a development
>       image with the right packages anyway, so choosing them and loading
>       them is no extra work for them.
>    2.
>       have a partially manual process during the release candidate phase
>       to produce an SQLite repository, using Olaf's packages in a
>       similar way to PostgreSQL. Since this would need to have all
>       packages in it (there not being a good way for users to add
>       packages), it would be big and duplicate the information from the
>       parcels. Being partly manual, that's open to errors and
>       differences between the final distro parcels. If you want to get
>       rid of the errors, you need a fully automated script -- so move to
>       1. Similarly if you want to have the repository for weekly builds too.
>
> So is there only one effective route, i.e. 1. above? There could of
> course still be SQLite support and instructions or an installation
> choice to set it up or reconcile into a previous version.
>  
> And Alan: once again, thanks!
> Steve

12