Using Subversion - Pax and the SourceBrowser?

Having thought about STS on the other thread and figuring that I should
try using Subversion a bit more I'm giving it a spin, but am a little
unsure about how all the bits fit together and whether they make sense
anymore?

I can see what using PAX does - but am not sure why I need to use it?
What does having all the classes in seperate files really do for me -
as a decent merge tool quite happily resolves all conflicts in a big
file quite easily (this could be me being naive - haven't encountered
big merges in my experiements).

The TortoiseSVN tool - really has a great diff and merge tool, and
against Subversion it all seems to hang together very nicely. (By the
way - installing these was really easy on a single user machine, two
download and installs basically).

Then I get onto the SourceBrowser that enabling PAX gives me. I am
thinking that its very SourceSafe centric? Given the table column for
"Writable" - this sounds like what source safe did with read only flags
(Perforce does the same, CVS and Subversion don't). Am I write about
this?  And if I merge some changes outside of Dolphin - how do you get
them back into the image - I was expecting to just right click and
reload (from the file system) but thats not there. There is a file-in
button but then I have to navigate and do more work than I expected. Am
I missing something - or is this the slightly convoluted process that
Andy hinted at in the STS thread?

To me (while I take on board what Chris said about having full history
in STS) it would seem like it should be quite easy to work in your
image. Hit some sync button which writes packages out to the file
system - runs the Subversion utilities to check in, catche conflicts,
present the built in merge browser (actually thats Tortoise, but the
most of the work is in the Subversion dll's) and then just reads the
source code back in.

OF couse this is a bit more work -  so as a quick workround, saving
your packages, flipping to Tortoise, using its tools to merge/commit
and then flipping back and just reloading any changed source should be
quite trivial.

While its not going to be STS - it has the big plus that all artifacts
go into an industly standard VCS - and other non-programmers can
happily create icons etc for me, and use Tortoise to check in.

Does anyone have any comments/experience on this? Its interesting
playing around with other alternatives - especially when you come back
to Dolphin with a fresh perspective.

Tim

Tim,

> I can see what using PAX does - but am not sure why I need to use it?
> What does having all the classes in seperate files really do for me -
> as a decent merge tool quite happily resolves all conflicts in a big
> file quite easily (this could be me being naive - haven't encountered
> big merges in my experiements).

Well, the idea behind using the PAX format to split up the packages
into separate source files was certainly to reduce merge conflicts.
This was especially the case with things like view resources which, in
Dolphin 5.1 are binary artefacts.

It may well be that a modern SCCS can handle merging PAC files that
have been modified by several developers.  There should be no reason
why you need to use PAX (just don't turn it on) if you don't want to.
In that case, the Source Browser also becomes pretty superfluous since
you can see which files have changed directly from the Package Browser
and save them straight from there.

You are right, it certainly is SourceSafe centric and for good reason
-- it's the only SCCS that we use in-house.

> And if I merge some changes outside of Dolphin - how do
> you get them back into the image - I was expecting to just right
> click and reload (from the file system) but thats not there. There is
> a file-in button but then I have to navigate and do more work than I
> expected. Am I missing something - or is this the slightly convoluted
> process that Andy hinted at in the STS thread?

There is currently no "reload" facility in the Dolphin Package Browser.
You will have to unload a package and then reload it. The problem with
this, of course, is that unloading a package may also have to unload
all of the dependent packages first and then reload them later.  STS
must already solve this reload problem so you could take a look at the
code in there and see how it does it.

Note that using "File In" is not sufficient. This will just take the
contents of the file and compile it on top of the contents that are
already loaded.  Hence, if any methods or resources have been deleted
in the new file they will still be present after the file in.

The "unofficial" workflow would be something like this:

1) Begin new change by synchronising PAC/PAX source files and loading
these into a virgin image.

2) Make changes in image, occasionally saving out PAC/PAC sources but
not checking them in.

3) When change is complete, check in and merge changes with repository
database.

4) Goto 1)

This is certainly how we have been working with Dolphin for the last 10
years (albeit using SourceSafe and not Tortoise).  If you follow the
workflow above you will find that this method works fine but it
certainly would benefit from the idea of a package "reload".

Personally I'm quite excited about STS being a part of the base system
in Dolphin 6.  I think the method level change history and differencing
will be well worth while and it thenbecomes very easy to see what has
changed between versions of the base system as well as your own code.
Also, I don't really see why you can't use two systems; keeping the
Dolphin source inside STS and other artefacts inside another VCS.

You are right that there will probably be some people, most likely
larger corporations, who would prefer the "security" of keeping all the
source inside a non proprietary database.  As you are discovering this
is still possible, although it entails somewhat more work on the part
of the programmer in manually syncing the package source files with the
working image.

Best Regards,

--
Andy Bower
Dolphin Support
www.object-arts.com

Thanks for taking the time to patiently explain this - it also better
hilites what STS is doing under the covers as well.

> It may well be that a modern SCCS can handle merging PAC files that
> have been modified by several developers.  There should be no reason
> why you need to use PAX (just don't turn it on) if you don't want to.

Actually after sending this - it did occur to me that by having seperate
files to check in, you have got the ability to find individual classes in
your VCS and diff them as well, so actually I have come around to liking the
PAX format.


> There is currently no "reload" facility in the Dolphin Package Browser.
> You will have to unload a package and then reload it. The problem with
> this, of course, is that unloading a package may also have to unload
> all of the dependent packages first and then reload them later.  STS
> must already solve this reload problem so you could take a look at the
> code in there and see how it does it.

Yes thats the crux of the problem - and so I guess it comes round to whether
its worth going to all the trouble to figure it out, or just use STS as it
arrives. Still I have to say the Subversion+Tortoise is a pretty impressive
combination. If you're not using it for your artifacts - its definitely
worth considering.

> Personally I'm quite excited about STS being a part of the base system
> in Dolphin 6.  I think the method level change history and differencing
> will be well worth while and it thenbecomes very easy to see what has
> changed between versions of the base system as well as your own code.
> Also, I don't really see why you can't use two systems; keeping the
> Dolphin source inside STS and other artefacts inside another VCS.

Don't get me wrong - I loved Envy when I had it all those years ago - and
this sounds very much the same. However, having used Eclipse and IntelliJ
for quite a bit now - you don't really notice how some of those cool
features aren't there anymore. Methd level history is definitely important -
but you only need it for the last few days, not normally for all of time -
so actually in practice Ian B's method history utility works quite well.

Having said all of this, I'm looking forward to giving STS a spin, and as a
little self learning excercise will play around with Subversion just to
brush up my VCS integration skills.

Thanks again,

Tim

[I've seen Andy's reply to this, and indeed TimM's reply to that, but I want to
give this my own spin]

TimM wrote:

> I can see what using PAX does - but am not sure why I need to use it?
> What does having all the classes in seperate files really do for me -
> as a decent merge tool quite happily resolves all conflicts in a big
> file quite easily (this could be me being naive - haven't encountered
> big merges in my experiements).

Merging is not the be-all-and-end-all of SCC.  It may not even be particularly
important if your development is reasonably linear (e.g. if there is only one
of you ;-).

The important point is granularity.  The external SCC doesn't know anything
about what's "inside" a given version of a file (it doesn't know Smalltalk
syntax, etc), so it can't give a semantically rich account of the history of
the /contents/ of a particular artefact.  If can only give a history of the
artefact, and it's up to /you/, looking at the diffs, to try to reconstruct the
history in terms of meaningful changes.  That's why I wouldn't want to see
Smalltalk code versioned at any lower granularity than the method.  Methods (in
typical Smalltalk code) are small enough that they can be considered atomic
from the POV of change history.  (No need to merge changes at the sub-method
level -- a "merge" is choosing one version of a method over a rival version.)

Related to that is the point that the external SCC will not (even if you do
export methods as versioned artefacts) know much about the relationships
implied by the method's source code.  You will not be able to "browse local
references" or cross reference to see the definition some class had back then.
That kind of operation is only possible if the SCC system is Smalltalk-aware --
and realistically that's only going to happen if it's provided within Dolphin.
Which is why I would want the full history of each method to be versioned --
it's a simple way to give Dolphin access to the history without taking primary
responsibility for managing changes away from the SCC.  (Other approaches are
possible if the SCC has a sufficiently rich API)


> And if I merge some changes outside of Dolphin - how do you get
> them back into the image - I was expecting to just right click and
> reload (from the file system) but thats not there. There is a file-in
> button but then I have to navigate and do more work than I expected. Am
> I missing something - or is this the slightly convoluted process that
> Andy hinted at in the STS thread?

I think that if I were to use external SCC with Dolphin, then I would also give
up on the notion of image-based development.  I would rebuild my image every
morning (and possibly at other times too).  The equivalent of doing a
"rebuild-all" in C++ development (or it would be if a C++ rebuild-all was as
quick as loading code into Dolphin -- which it would not be for non-trivial
projects.)

Of course, that's missing out on a major benefit of using Dolphin (or most any
other Smalltalk) in the first place.  (And also, I believe, encourages you to
think about /code/ rather than about /objects/ -- not a productive attitude,
IMO)



BTW, it should probably be possible to persuade STS to version external files
too.  It wouldn't give you the benefits of a "standard" SCC, so it probably
wouldn't be suitable for your purposes.  But it might be sufficient for
handling simple things like a few icons and what have you.

You create a new kind of object, VersionableFileProxy.  That takes holds a
filename (and perhaps a FileLocator), perhaps a timestamp too.  When it is
asked to STB itself, it writes the contents of the external file to the STB
stream (possibly compressed).  When it is asked to de-STB itself it recreates
the file.  Since (I presume, but haven't checked yet) STS compares objects at
least for binary equivalence it would not save duplicate versions of the same
file in the repository.  One could go further, and hold entire folders (or
selected highlights thereof) in the repository by encoding them as binary data
in ZIP format.  In the simple case, one would just make these objects be the
values of global variables that were part of the corresponding package, but --
if the idea works at all -- a more sophisticated approach (perhaps modelled on
ResourceManager) could be contemplated.  I haven't tried that approach, but I
may give it a shot -- I do have icons and whatnot that it would be nice to
place under STS.

    -- chris

Good points Chris - you raise some interesting finer points that I hadn't
considered

"Chris Uppal" <[hidden email]> wrote in message
news:43423153$2$38041$[hidden email]...
> [I've seen Andy's reply to this, and indeed TimM's reply to that, but I
> want to
> give this my own spin]
....
> implied by the method's source code.  You will not be able to "browse
> local
> references" or cross reference to see the definition some class had back
> then.

Browsing historical references is quite a neat idea. I do wonder how much it
would really get used, but interesting none the less. Like I said before -
having used Eclipse for a long while after Envy, I started to find that
local history was enough to roll back most things (in Dolphin Ian B
leverages the changes file to give this ability). As a team, when we hit
important points in the project (this was an XP team - so this was every
couple of hours) we just labeled things in the CVS repository.

The times we ever went back in time - was to diff these labeled versions
(and then again that was rare). So I agree, I like the dream - but
practically - I am a little worried that an STS solution is just Envy all
over again - and really just a niche market. Of course - keeping it open to
use both is very smart and so maybe having the best of both worlds will
prevail.

I am looking forward to giving it a spin in the next beta.

> BTW, it should probably be possible to persuade STS to version external
> files
> too.  It wouldn't give you the benefits of a "standard" SCC, so it
> probably
> wouldn't be suitable for your purposes.  But it might be sufficient for
> handling simple things like a few icons and what have you.

Something we could do that lessens the impact of this is to make STS visible
as a network drive (or some subset of it). In our Envy days - Oli Bye used a
java library that made any object visible on the network using SMTB (or some
acronym like that - I recall is was some windows network protocol). Anyway -
he wrote an adapter for envy so everything appeared as files accross the
network. I'm sure it wouldn't be so hard to do this for Dolphin. Its sort of
similar to Ari's squeak challenge to use ftp: into an image.

Tim

p.s. You mentioned some of the long quoted posts - apologies for that, was
away and using the Google news reader which doesn't show me how much it was
quoting. Will keep that in mind in the future.

TimM wrote:

> I started to
> find that local history was enough to roll back most things (in Dolphin
> Ian B leverages the changes file to give this ability).

;-)  Just a few days ago I rebuilt my image for the first time since 5th May
2004 -- just think of all the valuable history I would have lost...

(I was hoping to make it through to D6, but I've been doing too much
experimental external interfacing and must have got a rogue pointer somewhere
:-(  So I had to rebuild.  Not a big deal anyway, and it least it was a chance
to check (and find the odd error in) my rebuild process before D6 rolls
around.)


> Something we could do that lessens the impact of this is to make STS
> visible as a network drive (or some subset of it). In our Envy days - Oli
> Bye used a java library that made any object visible on the network using
> SMTB (or some acronym like that - I recall is was some windows network
> protocol). Anyway - he wrote an adapter for envy so everything appeared
> as files accross the network. I'm sure it wouldn't be so hard to do this
> for Dolphin.

You mean SMB, I think.  In theory it would be quite doable to make an SMB (over
TCP/IP) server in Dolphin.  I've speculated a few times about how to do it.
The only problem (besides actually having to read the specs and write the code
;-) and the fact that I don't use SMB over TCP on my network[*]) is that I
don't know whether its possible to run two SMB services on the same IP address.
OTOH, an NFS server would be possible, but then you could only "see" the
virtual files on a machine with an NFS client.  So Unix boxes would see the
data, but most Windows machines would be oblivious.  What the world needs is a
good, simple, cross-platform, virtual file system...

    -- chris

[*] I'm trying to stick with the old-fashioned SMB over NetBUI approach for as
long as possible since that isn't routable by accident ("accidents" like when
sp2 silently opened up file access via (IIRC) dial-up accounts....)

I haven't been very active with Dolphin for about a year, so the
following may be somewhat innaccurate...

I come from a C++ world which is file/project based and versioning
works smoothly with VSS/SVN/CVS.  A callout also to the good folks at
'Beyond Compare' who make differencing really easy for source
code/binary/images/.doc/.pdf version comparison.

Anyway, the Dolphin image based system of code control in small talk is
still rather jarring for me.  Which I think explains why some folks
here seem to prefer to start from the virgin Smalltalk image and add
patches then open their packages.

I like the idea of versioning each individual method with STS within
Dolphin.  Would this be stored in the package file for which the method
is associated??

Would it make sense to have the C++ equivalent of an external .dsp/.dsw
file which I could click on in explorer?  Then dolphin would load the
virgin smalltalk image, apply patches and open the corresponding
packages/dependencies.

Something like this would support the package loading workflow nicely.
I've never used 'Pax' workflow, but having 1 file / class seems a bit
too many files to manage.

Also as a previous person wrote, it would be nice to have a package
re-load capability.  Even nicer if Dolphin monitors the package files
for changes, and offers to load if they are changed by the external
source control.

Also to support package workflow it would be helpful to offer to save
all modified packages on exit from Dolphin.

Thanks again for this fine product.

Chris Uppal wrote:

[hidden email] wrote:

> Anyway, the Dolphin image based system of code control in small talk is
> still rather jarring for me.  Which I think explains why some folks
> here seem to prefer to start from the virgin Smalltalk image and add
> patches then open their packages.

Some people prefer to work that way even if they are completely used to the
"Smalltalk Way".  It seems like a sensible way of working to me too, although
it's not the /only/ sensible way of working, and is not the one I prefer
myself.


> I like the idea of versioning each individual method with STS within
> Dolphin.  Would this be stored in the package file for which the method
> is associated??

Well, I was just throwing ideas around.  It would obviously involve some
changes to the way the base system worked.  I doubt if that'll happen unless a
lot of people ask for it, or OA themselves decide they need it ;-)


> Would it make sense to have the C++ equivalent of an external .dsp/.dsw
> file which I could click on in explorer?  Then dolphin would load the
> virgin smalltalk image, apply patches and open the corresponding
> packages/dependencies.

I suspect it would be very easy to set something like that up. So easy,
probably, that there's no real need for it in the base system.

One idea I've been thinking about for a while (but haven't implemented --
mainly because I don't need it, given the way I work), would be to hook the
package load/save/creation events and each time the package set changed, update
an external file which contained a filein to reload exactly the packages in
that image.  That would make re-building an image very trivial.

One could go further and make the script do things like extract the latest (or
a specified) version of each package from STS, or from whatever external SCC
you use.  The sky's the limit really.  None of it's hard to do...


> Also as a previous person wrote, it would be nice to have a package
> re-load capability.  Even nicer if Dolphin monitors the package files
> for changes, and offers to load if they are changed by the external
> source control.

The current package system already has the ability to ask a package whether it
is "current" with respect to the timestamps on the external .pac files.  (The
feature was, in fact, added because I asked for it, and very useful I find it
too !)  That could perhaps be made the basis of such an idea.

Reloading installed packages is not so simple, though.  In fact I doubt whether
it's feasible.  In my own environment I have a tool to reload packages, but
that works by just unloading the package (and all its dependents).  That's much
simpler, but even that doesn't work perfectly; e.g. if one of the target
classes has existing instances, they are not "migrated" to the newly installed
class definition.  Another problem with it is that it interferes with STS if
that's running (it "forgets" that the current version of the package was
derived from STS's db, and so you have to remember to "join up" the version
history again when you next version the package -- not a major problem, just a
bit of a nuisance).


> Thanks again for this fine product.

Just in case there's any misunderstanding here, I should say that I'm not
connected with OA in any way.  But if that comment wasn't addressed to me
specifically, then I heartily second it ;-)

    -- chris