[vwnc] Bundles

My utopia:
Categorize my code in bundles/packages as I see fit.
Press "publish" on the bundle/package I want in source control.
Get one item in store.
Load that item, and have my code load itself in whatever order it  
needs to.
End up with a view identical to the one I had when I published.

Alternatively, if someone else has published modifications:
- If I've made no modifications, load the new version.
- If I've made modifications, get a view like the current merge view,  
then publish back to store as a new version after merge is complete.

In short, other than because there needs to be a load order, why  
should I have to care about the load order?
- As far as I can tell, the NewPrerequisiteEngine does quite a good  
job of figuring it out just by clicking a button...
- Make non-essential tasks like messing with the load order the  
exception rather than the rule.
- At the very least, find a way to easily define a correct one,  
without messing with my daily productivity by forcing my  
categorization of code in the tool I use every day for browsing,  
making it harder to remember where stuff is than necessary. (And no,  
prereqs doesn't cut it when your project has over 200 packages, and  
there's no centralized GUI for the ordering/you can't be sure they're  
correct before you load from store )

In any case, it would be nice to have a separation of concepts.
For any beginner previously familiar with... other languages(speaking  
for both myself and another programmer I just held a short  
introduction to), the difference between parcels/packages/bundles/
prerequisites roles when it comes to categorization/loading order/
storage can be rather confusing.

Cheers,
Henry

A short background/user story:
One of my first tasks when starting in Smalltalk was to reorganize the  
existing codebase into something loadable from Store. The previous  
method was pretty much sharing fileouts of new code between  
developers, and reconciling a "base" image for each release.
As such, the starting point consisted of bundles/packages mostly for  
code categorization, with some prereqs sparsely sprinkled where the  
developers had felt it would look nice.
The structure was one of 3 levels, 1 top bundle, 11 sub bundles, and  
about 250 leaf packages.
I started adding additional package prereqs to make it load from store  
after publishing from an image, but that quickly turned into somewhat  
of a game of ping-pong between internal project packages which might  
or might not actually need each other.
Not to mention I couldn't find any equivalent to the "Validate" button  
for prereqs, and had to republish/reload every change to find new  
errors, which for a project this size takes quite awhile.
Since no one was ever going to want to load these packages one by one  
though, i cleared all the package prereqs, and used the Validate  
button in the bundle specification to iterate into a loadable state.  
Then added the external prerequisites to the top bundle, ending up  
with a big chunck of code which was loadable from a base image.
Quite possibly it could've been split into smaller loadable modules,  
but I felt the effort that would've been needed with the current tools  
was too large to be worth it.
In essence, I ended up using the top bundle as a "Super package", with  
the sub-levels upholding a valid load order, but still providing most  
of the initial categorization.

As of now, we're always publishing changes all the way up to the top  
bundle, as that's provided for the easiest method of reconciling as  
far as our experience goes. (merge - reconcile - publish).
We've long ago given up having any meaningful version numbers on the  
sub-bundles/packages, since they bork up once you forget the middle  
step above, and that happens from time to time.
Apparently when you're merging with a higher version number (not a  
different branch), what you really want to do is to branch off the  
common base, not publish a new version including your local changes as  
well.


P.S. On the topic of categorization, am I the only one shivering  
whenever I look at the UIBasics bundle, and crying myself to sleep  
when I have to modify a component there?
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

we do use bundles and sub bundles

1. as a configuration maps(closed) for the  release versions and
modifications for them. I can always compare them and see the
differences form version to version...or I can go back to fix an error
or create a patch to the earlier released version.

2. as a way to package different versions of the application with
different feature set, for example with or without network support.

I try to stay away from prerequisites i.e. our bundle do not have
prerequisites( or just very basic ones) - we load them into the already
pre-configured "base" images, where all of the required parcels( and/or
packages)are already loaded.  Same strategy applies to the building
runtime images -I produce the parcels from my version bundle, load them
int the pre-configured base runtime image and strip... This adds one
extra step to deploy new version of VW into the development -- building
the base, but it well worth it as we know what environment we deal with
at any give time.

My 2c.

--Mark Pirogovsky

Bruce Badger wrote: _______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

> Do you use bundles?
yes

> If so, how do you use them, and why?

- To collect our code together in the browsers
- To version our code together, and provide a place for an overview of
all the individual packages' version comments
- To make loading our code easier (smaller number of components +
versions)

- We mostly have bundles that only contain packages, and each package is
only in one bundle
- We have a couple of occurrences of a bundle containing (packages and a
bundle containing packages)
- We have a couple of occurrences of a package being contained in two
different bundles

- A bundle's version history may sometimes branch temporarily to allow
patches to a release, while the main trunk continues with free
development of the next release

- We split into different bundles where appropriate or when forced to by
Store (e.g. lack of support for overrides within bundles)

- We build development and deployment images from a virgin visual.im
with scripts that load Cincom parcels, reconcile them with our local
Store, load a few packages from the public Store, and load our app from
our local Store. For deployment a script then unloads Store and other
unloadable development packages then runs RTP.

- We have added our own little tools:
  - a better list of "More recently published" that filters out packages
if they can be loaded by loading a bundle shown
  - "Unloaded published items" and "Unloaded recently published items"
to show 'relevant' new packages or bundles
  - the bundle compare versions dialog has a button "Show version
comments" that concatenates the version comments of all package changes
between the bundle versions
  - "reconcile just bundle" to quickly rebase a bundle on a more recent
published version, without trying to reconcile the code contents of the
packages
  - "More recently published contents of bundle" that lists only updated
versions of this bundle or its contents

> If setting explicit pre-reqs was easier, would you still use bundles?

Yes. We use pre-reqs very little. Pre-reqs create a network where each
package carries some of the responsibility, and the result is determined
at load time. The mechanisms in pre-reqs are not sophisticated enough to
specify which package and version of it should be loaded under which
circumstances (e.g. VW version compatibility, platform). Conversely, if
they were more complicated then getting them right would be harder - at
least to achieve the desired level of certainty.

Pre-reqs work well when there is a large mass of components shared by a
large mass of users from different organizations, and those users don't
have many components from outside the shared mass - e.g. Linux OS
components and standard apps. Bundles (or similar) work better within
one organization, or where some components are local and some shared
(e.g. our Store + the public Store).

Travis asked a few more questions, some of them rather leading :-). Yes,
we used Envy actively 1993-2000, with about a dozen programmers over two
sites. One of our bundles' structure was defined then, the rest since. I
feel bundle load order lists are often the simplest and most reliable
way to sort out load order issues. I've never noticed that bundles
suppress package post load actions - is that really true??

Cheers,
Steve

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

I would like to point out that the use of prereqs for
determining load order is overly restrictive. In fact
there are two kinds of prereqs; loadtime prereq and
runtime prereq. Separating the two and using just the
load prereqs for load order would be most desirable.

Terry
 
===========================================================
Terry Raymond
Crafted Smalltalk
80 Lazywood Ln.
Tiverton, RI  02878
(401) 624-4517      [hidden email]
<http://www.craftedsmalltalk.com>
===========================================================

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

In the project where I am working, we seem to use bundles and
prerequisites more or less the same way as Steve's group.

Steven Kelly schrieb am 14.05.2008 14:02 Uhr:
>> Do you use bundles?
> yes
>> If so, how do you use them, and why?
>
> - To collect our code together in the browsers

Same here. We develop about ten different products with ca. 40
developers, and the product I have worked on has four different variants
(two very similar clients for different user groups, a mobile/offline
version, and a batch processor). There is a group developing basic
libraries, and some products include components from up to three other
products. These components are grouped in bundles, because we usually
want to load all of their packages. If we only need some of them, that
can still be done. The development images typically contains hundreds of
packages in addition to the VisualWorks base. Bundles and sub-bundles
are necessary to be able to navigate in such a large code base, and to
define the contents of product variants in a simple way.

> - To version our code together, and provide a place for an overview of
> all the individual packages' version comments

+1

> - To make loading our code easier (smaller number of components +
> versions)

Definitely. We usually don't define prerequisites at the package level,
except when a package can be used in isolation. We use bundles to keep
related packages together, which also ensures that the have the right
versions loaded - ensuring this in a consistent way with prerequisites
only would be a maintenance nightmare. We use prerequisites at the
bundle level, and the prerequisites are mostly bundles. We have tool
support which updates the prerequisite versions from what's loaded in
the image, which makes baselining very simple.

> - We mostly have bundles that only contain packages, and each package is
> only in one bundle

We have sub-bundles, and usually a package is in only one bundle. There
are a few exceptions, but we try to avoid them.

> - We have a couple of occurrences of a bundle containing (packages and a
> bundle containing packages)
> - We have a couple of occurrences of a package being contained in two
> different bundles
>
> - A bundle's version history may sometimes branch temporarily to allow
> patches to a release, while the main trunk continues with free
> development of the next release

Same here. Sometimes, we also have branches for one or more prototypes,
or for a larger refactoring which is intergrated into the trunk after
maybe several weeks of development. There have been situations where we
had up to four parallel lines of development, and branches of bundle
versions made this easy to manage.

> - We build development and deployment images from a virgin visual.im
> with scripts that load Cincom parcels, reconcile them with our local
> Store, load a few packages from the public Store, and load our app from
> our local Store. For deployment a script then unloads Store and other
> unloadable development packages then runs RTP.

More or less the same here, except that we use a base image with
preloaded and already reconciled parcels, and we don't use the public Store.

> - We have added our own little tools:

+100 ;-)

>   - a better list of "More recently published" that filters out packages
> if they can be loaded by loading a bundle shown
>   - "Unloaded published items" and "Unloaded recently published items"
> to show 'relevant' new packages or bundles
>   - the bundle compare versions dialog has a button "Show version
> comments" that concatenates the version comments of all package changes
> between the bundle versions

Good idea. May we steal it? :-)

>   - "reconcile just bundle" to quickly rebase a bundle on a more recent
> published version, without trying to reconcile the code contents of the
> packages
>   - "More recently published contents of bundle" that lists only updated
> versions of this bundle or its contents

Same here, plus the option to do this for a configurable list of bundles
which go into a product with a single click, and load what has been
found. With this tool, we have "open configuration maps" - no need to
publish the bundles every time we publish a package, and very easy to
stay up to date. Bundles are only published for baselines, and when
their structure changes (addition/removal of packages). Some teams still
prefer to publish the bundles every time, but this leads to very long
version histories and slows down the load process for updating an image.

>> If setting explicit pre-reqs was easier, would you still use bundles?
>
> Yes. We use pre-reqs very little. Pre-reqs create a network where each
> package carries some of the responsibility, and the result is determined
> at load time. The mechanisms in pre-reqs are not sophisticated enough to
> specify which package and version of it should be loaded under which
> circumstances (e.g. VW version compatibility, platform). Conversely, if
> they were more complicated then getting them right would be harder - at
> least to achieve the desired level of certainty.

+1, see above.

I agree, and as I described above, we work along this line.

> I've never noticed that bundles
> suppress package post load actions - is that really true??

I also don't remember such a behavior. I am almost certain that our
application (and JNIPort, which uses bundles, too) wouldn't work if this
were true.

Joachim
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Andre Schnoor schrieb am 13.05.2008 12:15 Uhr:
> Yes. I would rather see bundles in their "configuration map" role,  
> i.e. associate a specific set of packages (with versions) and their  
> load order for the purpose of loading products into an image and for  
> deploying them.
>
> Bundles should be safe for archiving: Loading an older product from an  
> archived configuration map for maintenance should be possible even  
> after years (maybe it already is and I missed something).

Yes, this is already possible. You have to make sure that
- you have the right version of VisualWorks, i.e. a visual.im from the
VW version which was the base for the product version, and the right
version of the VM,
- all prerequisites have been defined with the version of the prerequisite,
- all prerequisites should have the type "bundle" or "package", but not
"parcel" or "any",
- which implies that everything which you have loaded from parcels
should be published to Store and loaded from there, simply to make sure
that you don't accidentally load a parcel which has been modified or
replaced in the meantime,
- all non-Smalltalk files (images, shared libraries, configuration files
etc.) should be attached to a bundle and versioned in Store, and VW
should be configured such that it copies them from Store to the file
system when you build the product.

That should be sufficient, if I haven't forgotten something. We have
"top level bundles" for our products, and when we load them into a base
image, they load everything needed via versioned prerequisites without
manual intervention. The process is completely deterministic, and we can
reconstruct any product version which has not been explicitly deleted
from the repository.

Joachim
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Joachim,

thanks for the good instructions on how to make a bundle safe for  
archiving.

My concern however is, that all the manual steps required to achieve  
this are tedious and error prone. For me, fixing the properties of  
140+ packages manually is a show stopper. I'm not charging my  
customers based on hourly rates ;-)

I would love to see a menu item "Make Bundle Snapshot" that  
accomplishes all this automatically in one go (especially setting all  
prerequisite versions and publishing required base packages, if  
changed). This could also be an option in the "Publish" dialog.

I still envy those who successfully manage to keep released products  
along with development streams in the same repository over a long  
period of time. Those many "Merge", "Compare" and "Reconcile" dialogs  
and confirmaton messages always confused me and I did not yet have the  
time to really understand the workflow and basic concept through  
experimentation. IMO this is a steep learning curve that could be  
flattened a lot, if the user interface was aggregated into a single  
"Store Browser" with tabs that includes it all in one place very much  
like the System Browser.

Andre


Am 16.05.2008 um 22:27 schrieb Joachim Geidel:

...

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

We do use bundles, and sub-bundles, and sub-sub-bundles, and in fact
pretty much all of our packages would be totally useless without the
other packages in their bundle.

One of the reasons for this is that our software runs on both VW and VA,
and we must be able to maintain synchronization between our Store and
Envy repositories. One of the things that makes this possible is that
the structure of packages and bundles in our Store repository and the
applications and subapplications in our Envy repository is exactly the
same for the fairly large subset of our code that is the same in both.
VW-specific code is separated into packages that don't have an analogue
in Envy, and VA-specific code is separated into subapplications that
don't have an analogue in Store.

So for us bundles act as configuration maps, as containers for
deployment packaging, as browsing convenience, as providers of load and
unload actions, and as a somewhat clunky method of fulfilling
prerequisites.

If you wanted us to stop using bundles you'd have to provide something
(or things) at least as good for each of these.

Regards,

-Martin
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

We use bundles like config maps (to specify what versions to load) and
applications (to group packages together). We use packages somewhat like
sub-applications are used in ENVY (to categorize code). Where packages
declare prerequisites, they do not specify versions because that would
conflict with the version lineup specified by the bundle.

Developers version their package changes as integration-ready. Release
tools find and merge all integration-ready changes (for all contained
items) to produce an internal-release version with a ".0" suffix to
indicate an integration point that developers can base further changes
on. The ENVY 'open edition' functionality that some people like can be
achieved by just using the release tools to merge integration-ready code
and re-parent (skipping the publish part); however, developers usually
publish a new internal-release version when they feel their changes are
safe for general use. The release tools we've developed are an important
part of making this all work so well.

I don't care if it is made easier to configure package prerequisites
because we would probably still use bundles. Setting package version
prerequisites would be in conflict with the way we use bundles for
version lineups. I see no benefit to declaring an inverse relationship
between versions. This old dog thinks in terms of containment and
snapshots.

With ENVY you could create a new map edition and customize the lineup
without having contained items loaded. Much trickery needs to be done in
order configure bundles prior to load. For example, this affects us with
each new GBS release because GBS only initializes properly when loaded
from a parcel and store-load-initialization often leaves GBS inoperable.
The code relationship that is maintained with a loaded parcel (that is
also represented in StORE) makes it difficult to just replace and reload
the parcel files. Those issues are higher priority to me than easier
prerequisite setting.

Paul Baumann

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On
Behalf Of Bruce Badger
Sent: Monday, May 12, 2008 6:42 PM
To: VW NC
Subject: [vwnc] Bundles

On his Blog, Jim asked about Store Bundles:

    *  Do you use bundles?
    * If so, how do you use them, and why?
    * If setting explicit pre-reqs was easier, would you still use
bundles?

I do use bundles.

I use nested bundles with an outer master bundle for every project.

Even if setting preq versions were easier I probably still use bundles.
I like the explicit containment.

--
Make the most of your skills - with OpenSkills
http://www.openskills.org/
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
 
--------------------------------------------------------
This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc