VM packaging for Cog transition

 
On 11/9/2010 10:10 AM, K. K. Subramaniam wrote:
> Bert's reply is ingenious ;-). Generated files are intermediate files, not true
> source files. True "sources" are actually Slang code in VMMaker image and these
> should be in version control.

The generated files are true source files. If you've ever had the need
to ensure that you're building the same VMs across various platforms
you'll quickly find that you're moving generated files around, not images.

> When bootstrapping VM on a new platform, a sub-set of these intermediate files
> along with hand-coded source files are compiled generate a proto-vm. The proto-
> vm runs VMMaker image to translate all Slang code into C so that the regular
> VM and plugins can now be built and the proto-vm can be discarded.
>
> The VMMaker image is a critical part of bootstrap and should be put under
> version control.

No and no. When bootstrapping a new VM on a new platform you move the
generated files and start from those. The VMMaker image is an
afterthought, only used for generation of the sources and not involved
in the bootstrap of a new platform at all.

Cheers,
   - Andreas

 
On Tuesday 09 Nov 2010 10:50:50 pm Bert Freudenberg wrote:
>  was asking how to name that cog/stack vm binary. I don't see a reason to
> have both cog and stack VMs on one architecture officially.
What if Cog VM chokes on existing image files? There has to be fallback. x86
environments are very diverse. It is hard to foresee vm/image combinations.

> On x86 it
> would be cog, on everything else stack VM for now. The question is, should
> that difference be reflected in the binary name? Is it desirable to have
> both cog and stack VMs installed next to each other? Would people be
> confused if there is no "cog" binary on their architecture?
I feel the difference should be reflected in the name, just like in qt3 and qt4.

Subbu

On 09.11.2010, at 19:25, K. K. Subramaniam wrote:

> On Tuesday 09 Nov 2010 10:50:50 pm Bert Freudenberg wrote:
>> was asking how to name that cog/stack vm binary. I don't see a reason to
>> have both cog and stack VMs on one architecture officially.
> What if Cog VM chokes on existing image files? There has to be fallback. x86
> environments are very diverse. It is hard to foresee vm/image combinations.

Sigh. You're reading selectively ;)

We *are* going to have an interpreter for existing images. This has nothing to do with the stack/cog vm.

>> On x86 it
>> would be cog, on everything else stack VM for now. The question is, should
>> that difference be reflected in the binary name? Is it desirable to have
>> both cog and stack VMs installed next to each other? Would people be
>> confused if there is no "cog" binary on their architecture?
> I feel the difference should be reflected in the name, just like in qt3 and qt4.
>
> Subbu

That is certainly one option. "squeak4" would be the interpreter and "squeak5" would be stack/cog. John's Mac VMs already use that scheme.

- Bert -

 
Just thinking...

It would be really fine if Cog allowed forks as defined in
OSProcess/CommandShell (like forkHeadlessSqueakAndDoThenQuit: )

CdAB

 
On Tue, Nov 9, 2010 at 11:54 AM, Bert Freudenberg <[hidden email]> wrote:

> Sigh. You're reading selectively ;)
>
> We *are* going to have an interpreter for existing images. This has nothing to do with the stack/cog vm.

To put it another way, there may well be need to have both Cog and
Stack VMs installed alongside each other, even on x86. It seems
superfluous, but why make that scenario difficult? I say the previous
suggestion makes the most sense:

squeakvm-cog
squeakvm-stack
squeakvm-interp.

If we're planning to maintain these three variations going forward, we
shouldn't use version numbers to differentiate between them. Someday
we'll be releasing version 6.0 of the interpreter, and it makes no
sense to have that binary be called squeak4.

Colin

 
On Tue, Nov 09, 2010 at 08:54:09PM +0100, Bert Freudenberg wrote:
I notice that that command name "turbo" seems to be unclaimed on most
unix boxes. We could name the cog version /usr/local/bin/turbo, so that
when you want your squeak.image to go faster you would just enter:

  $ turbo squeak

;-)

Dave
 

On 9 November 2010 12:25, Bert Freudenberg <[hidden email]> wrote:
i like this separation. Ookay, Andreas insists that generated files
should be version controled. No problem.
But existing files layout makes it hard for newcomers to distinguish
generated sources from hand-written ones, which often leads to
problems "i lost, i can't compile it ".
So a clear separation of generated sources from hand-written ones is
important improvement.

> However, one reason to put the generated sources inside the platform folders was because often the platform code lagged behind, and only the single maintainer could fix it. A better solution to that would be opening up VM development. As I mentioned before I'd like us to move to a DCVS. If we intend to change the repository structure this would be an excellent time.

We talked about it before. It would be good to migrate on github (or
something like this), so we could have more open development process.
Maybe its time to make it happen? :)

>
> - Bert -
>

--
Best regards,
Igor Stasenko AKA sig.

 
On 11/9/2010 12:55 PM, Igor Stasenko wrote:
I'm not insisting. I'm just saying that we have them in version control
so that we can reproduce our production VMs faithfully on all platforms.

> But existing files layout makes it hard for newcomers to distinguish
> generated sources from hand-written ones, which often leads to
> problems "i lost, i can't compile it ".
> So a clear separation of generated sources from hand-written ones is
> important improvement.

I'm missing something. The current structure allows for the above. If
you'd like to check out for example
http://squeakvm.org/win32/release/SqueakVM-Win32-4.1.1-src.zip you will
see that the structure is -guess what- almost identical:

platforms/
        Cross/
        Win32/
winbuild/
        src/

The generated sources are inside winbuild (for historical reasons) but
they are *very* clearly separated from the hand-written stuff. I'm not
sure why everyone is so excited about a layout that we already support?

Cheers,
   - Andreas

On 9 November 2010 23:17, Andreas Raab <[hidden email]> wrote:
Huh? Are we talking about same things?

Take a look at our official SVN tree:

http://squeakvm.org/svn/squeak/trunk/

where winbuild?

Take a look where generated sources located for unix:

http://squeakvm.org/svn/squeak/trunk/platforms/unix/src/


> The generated sources are inside winbuild (for historical reasons) but they
> are *very* clearly separated from the hand-written stuff. I'm not sure why
> everyone is so excited about a layout that we already support?
>

Yes. Except that if you check out from svn, there is no winbuild.
If you mean that you packaging a VM source (in zip archive) ,
and there it contains winbuild dir, this is different.
We are talking about version-controlled repository with good layout
from very starting.
No?

> Cheers,
>  - Andreas
>



--
Best regards,
Igor Stasenko AKA sig.

 
On 11/9/2010 1:42 PM, Igor Stasenko wrote:
Clearly we aren't, otherwise I wouldn't be so confused :-)

> Take a look at our official SVN tree:
>
> http://squeakvm.org/svn/squeak/trunk/
>
> where winbuild?

In platforms/win32/build. This is the proto-location and one can either
build things in there or not. I choose to keep things separate but it's
a choice I am making for my build setups. Others can make other choices.

I'm not sure what we are talking about. There appears to be an immense
enthusiasm about the fact that - Lo and behold! - VMMaker can generate
sources in different locations. What an amazing new insight :-)

Is this simply about checking the generated sources into SVN? Or the
build setups? If so, let's just do that and get over it. It's a trivial
little thing to do and really doesn't warrant this lengthy discussion.
If not, then I'm still completely unclear what this discussion is
supposedly about.

Cheers,
   - Andreas

On 10 November 2010 00:40, Andreas Raab <[hidden email]> wrote:
hehe.. actually i were always using separate dir from very starting.
But on unixes its somewhat messy. For instance, i prefer to delete the
src subdir
to know clearly that during build i using right location of generated
sources, and not interfering with ones which come by default from SVN
(platforms/unix/src).

Since there is no point to keep multiple copies of generated sources
for each platform, i think they should be put
outside plarforms dir, into a separate (as Bert suggested) 'generated' dir.
So, then all platforms could use these files as a default location for
generated sources,
while of course we're free to use different one.

> Is this simply about checking the generated sources into SVN? Or the build
> setups? If so, let's just do that and get over it. It's a trivial little
> thing to do and really doesn't warrant this lengthy discussion. If not, then
> I'm still completely unclear what this discussion is supposedly about.
>

I'd prefer to have everything we need to be stored in SVN (err.. github ;) ),
so i just checking out, cd into some subdir, do make and voila.
'one click' for geek :)

> Cheers,
>  - Andreas
>



--
Best regards,
Igor Stasenko AKA sig.

 
On 11/9/2010 4:25 PM, Igor Stasenko wrote:
> Since there is no point to keep multiple copies of generated sources
> for each platform, i think they should be put
> outside plarforms dir, into a separate (as Bert suggested) 'generated' dir.
> So, then all platforms could use these files as a default location for
> generated sources,
> while of course we're free to use different one.

Fine. I couldn't care less. If people want to check some version of the
generated sources into SVN, that's absolutely fine with me.

>> Is this simply about checking the generated sources into SVN? Or the build
>> setups? If so, let's just do that and get over it. It's a trivial little
>> thing to do and really doesn't warrant this lengthy discussion. If not, then
>> I'm still completely unclear what this discussion is supposedly about.
>
> I'd prefer to have everything we need to be stored in SVN (err.. github ;) ),
> so i just checking out, cd into some subdir, do make and voila.
> 'one click' for geek :)

You're not going to change the need for file releases. The idea that at
any given point you can just check out SVN head and get a functioning
build on all platforms has never been true in the past, and won't be
true in the future. Different people have different amounts of time for
working on the various aspects so head is expected to be a bit
problematic at any point in time. That's why we have file releases which
provide *exactly* what you are asking for. Download the file release
from http://squeakvm.org/win32/release/SqueakVM-Win32-4.1.1-src.zip,
extract it, cd into the winbuild directory, type "make" and out pops a
shiny new VM. Whether the winbuild directory is SVN-backed or not,
matters little in this context.

Cheers,
   - Andreas

 
On Tue, Nov 09, 2010 at 10:50:31PM +0530, K. K. Subramaniam wrote:
> 0 leshort 6502 Squeak Image Classic
> 0 leshort 6504 Squeak Image Standard
> 0 leshort 6505 Squeak Image with Closure and Float reordering

Ah yes, that's it, thanks!

The image that you run on an Intel box might have been saved on an
older Mac with reversed byte ordering, and we may as well throw in
the theoretically possible 64-bit variations for completeness, so
the magic file entries could be extended like this:

0       lelong          6502            Squeak Image Classic
0       lelong          6504            Squeak Image Standard
0       lelong          6505            Squeak Image with Closure and Float reordering
0       belong          6502            Squeak Image Classic
0       belong          6504            Squeak Image Standard
0       belong          6505            Squeak Image with Closure and Float reordering
0       lelong          68000           Squeak Image 64-bit Classic
0       lelong          68002           Squeak Image 64-bit Standard
0       lelong          68003           Squeak Image 64-bit with Closure and Float reordering
4       belong          68000           Squeak Image 64-bit Classic
4       belong          68002           Squeak Image 64-bit Standard
4       belong          68003           Squeak Image 64-bit with Closure and Float reordering

This works with the original 64-bit image (saved on a Mac) as
well as a newer 64-bit image saved on Intel, so I think the rules
should be right.

Note, the format number is saved in the first 8 bytes (not 4) for
a 64-bit image. A recent Linux supports "lequad" and "bequad", but
this is not portable, hence the "belong" with 4-byte offsets in the
last three lines.

Dave

On Tuesday 09 Nov 2010 11:50:42 pm Andreas Raab wrote:
> On 11/9/2010 10:10 AM, K. K. Subramaniam wrote:
> > Bert's reply is ingenious ;-). Generated files are intermediate files,
> > not true source files. True "sources" are actually Slang code in VMMaker
> > image and these should be in version control.
>
> The generated files are true source files. If you've ever had the need
> to ensure that you're building the same VMs across various platforms
> you'll quickly find that you're moving generated files around, not images.
I meant "source" in the sense of being able to copyright, apply patches and
track version history, not just in the sense of using them to compile into
object code.

While it is possible to ‌bypass VMMaker and maintain sources manually, it is
not how Squeak is being maintained today.  AFAIK, If there is a bug in
interp.c or in any of the internal plugin code that causes build to fail,
patches go into VMMaker, not into generated source. The generated code
contains a reference to VMMaker version for applying patches.

Of course, once the patched code is regenerated, VMMaker is not required for
subsequent builds. Then, what you say applies.

Subbu

 
On Wed, Nov 10, 2010 at 11:33:28PM +0530, K. K. Subramaniam wrote:
I think we are mixing words here. From the point of view of development,
Smalltalk is the source. From the point of view of rebuilding the VM
in a reliable and reproducable manner, the generated C files are the
source. Both are valid and important uses. It is important for people
to be aware of the distinction so that they do not mistakenly "fix" a
problem by modifying generated C sources, but aside from that there
is no conflict.

The various suggestions for keeping generated sources outside of
the ./platforms tree are, I presume, intended to reduce this kind
of confusion.

Dave