TEST

TEST.

Hello, I am contemplating forking the VM due to my frustration over
having to live at Ian's mercy with regards to enhancements to the VM.  I
don't mean to fault Ian, he has clearly done some excellant work with
Squeak and is superbly talented. My issue is that I am unable to work
with his build environment to the point that I can successfully
implement my own extensions such as the SDL driver that I've been
working on. -- I can't figure out how to make the build script work.

I am therefore contemplating an attempt to fork a new set of platform
code that I can work with. I don't expect the quality of my code will be
able to equal that of Ian's in terms of compatibility, only that it will
be better suited to my skillset. -- A version that I can change quickly,
easily, and with impunity.

In researching this effort I came across an extremely interesting bit of
code in my Croquet image:

#######
glActiveTextureARB(texture)
    "This method was automatically generated."

### By what software???

    "void glActiveTextureARB(GLenum texture);"
    <apicall: void 'glActiveTextureARB' (ulong) module: 'opengl32.dll'>

### An initial investigation seems to indicate that this is accomplished
through the "FFI" plugin. How does this really work???
If it's even 1/10th as powerful as it appears to be, most of the
existing platform and cross code should be scrapped in favor of in-image
wrappers! =P
####

    ^self externalCallFailed
########

On Monday 28 March 2005 1:50 am, Alan Grimes wrote:
> Hello, I am contemplating forking the VM due to my frustration over
> having to live at Ian's mercy with regards to enhancements to the VM.  I
> don't mean to fault Ian, he has clearly done some excellant work with
> Squeak and is superbly talented. My issue is that I am unable to work
> with his build environment to the point that I can successfully
> implement my own extensions such as the SDL driver that I've been
> working on. -- I can't figure out how to make the build script work.

I find this a bit odd. I know of many people who have downloaded the sources,
VMMaker, etc. and have had no problems building a VM.

Assuming your system has a properly matched set of tools, installed properly,
the configure script should work properly.

--
Ned Konz
http://bike-nomad.com

Ned Konz <[hidden email]> wrote:

I think he was talking about modifying the build scripts, not running
them as is.

Even so, you must admit that this is a lot of if's.  If you have the
wrong awk, you don't get gnuification.  If you have the wrong version of
autoconf, then the configure script may not be rebuildable.  Wrong
version of gcc?  Your compile can fail.  etc.

FWIW, the scripts are still not perfect even with all the if's
satisfied.  My current system  builds without complaint, but the
resulting VM cannot open an X window.  (This is why I haven't yet posted
a deb of Ian's latest VM release.)  vm-display-X11 is created, and I can
compile other X programs just fine, so I don't know what the problem is.
 I intend to eventually dig to the bottom of this; in fact, I've seen it
before, but cannot remember what the problem was!  I'm not complaining--
bugs happen!  Let's not get so overconfident that we are surprised that
it does not go perfectly for someone.

Regarding the complexity of the script, I have several ideas.  In fact,
I have actually coded several of them up in the past, for whatever
credibility that gives me.  Since those changes have been rejected--
many times--I won't work further on them myself, but maybe some
people have thicker skin and would like to take a try at it.  Here are
a few from the top of my head:

        1. Dump make.  We already depend on GNU make.  It is no more onerous to
depend on Jam, which would make it easy to encode more rules in the
Jamfiles.  Jam is easier to understand, and further, it removes a major
temptation to violate #2:

        2. Use autoconf only for its purpose: *guessing* about configuration.
Always let the user override the guess.  And don't overload it with
other functions, just because make doesn't have if statements or
subroutine calls.  A quick test for whether you are overloading it is:
if you change part of the build scripts unrelated to configuration
settings, you should not have to re-run the configure script.

        3. Reconsider libtool.  It's a nice idea, but it is really confusing,
and I'm not sure it is truly helping out.  It sucks having to read an
extra manual just because we are too afraid to hard code "gcc -shared".
It would not be terrible, if the standard build scripts can only build
external plugins if you are using gcc.

        4. Heck, while at it, reconsider the default of external plugins.
There are clear advantages to external plugins, but they cause an
installation hassle.  Especially it is annoying, if you want to compile
an experimental VM and test it out without installing it on your system.
 Make internal the default, if VMMaker allows this.

Don't take me wrong -- the build scripts generally work fine.  Let's
just not overestimate them.  They are complicated, few people understand
them (maybe just 1?), they sometimes fail, they often require people to
install extra dependencies, and when the the build fails, the error
messages (or lack thereof) are frequently impenetrable.  Most of this is
not necessary, and so we should welcome discussion (and code!) about how
to improve them.

-Lex


PS -- Alan, there are no Squeak Police that will come after you if you just do
something like gcc *.c */*.c.  :)   Please don't dump great VM work just
because you can't use the official build scripts.  Build script crud can
always be fixed up later.

On Apr 7, 2005, at 8:22 AM, Lex Spoon wrote:
> Even so, you must admit that this is a lot of if's.  If you have the
> wrong awk, you don't get gnuification.

I'll point out that is a smalltalk version of the gunification  
"Gnuifier.5.cs", which could be added to the VMMaker logic
removing that awk step. I'll note it would need a bit of change since  
it points to a different powerpc register
than the unix sqGnu.h uses, hence I"ve a much older/modified sqGnu.h  
that is used in the carbon mac compile.

--
========================================================================
===
John M. McIntosh <[hidden email]> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

In message <[hidden email]>
          "Lex Spoon" <[hidden email]> wrote:

>
> 4. Heck, while at it, reconsider the default of external plugins.
> There are clear advantages to external plugins, but they cause an
> installation hassle.  Especially it is annoying, if you want to compile
> an experimental VM and test it out without installing it on your system.
>  Make internal the default, if VMMaker allows this.
VMMaker lets you do whatever you want about which plugins are internal,
external or left out altogether. I was under the impression that the unix VMs
normally had everything internal on the grounds of simplest installation; *nix
seems to lack any sensible way to gather a bunch of files together to assemble
an app. OSX has at last grasped the idea of an application directory in the
manner of 18 year old RISC OS ideas, something that makes it really simple to
have a single user-level entity that can have an arbitrary collection of code,
resource, text and application files under it. No need for that idiotic windows
registry stuff either. I can't imagine that it would be hard to implement
something similar to make life a bit easier.

tim
--
Tim Rowledge, [hidden email], http://sumeru.stanford.edu/tim
If at first you don't succeed, destroy all evidence that you tried.

I refuse to get drawn into an(other) argument about any of this.  But
here's a little history/rationale anyway, for those who have
short/selective memories.

On Apr 7, 2005, at 08:22, Lex Spoon wrote:

> Even so, you must admit that this is a lot of if's.

Well...

> Ned Konz <[hidden email]> wrote:
>
> Assuming your system has a properly matched set of tools, installed
> properly,
> the configure script should work properly.

Not really.  If you install completely incompatible versions of
autoconf, automake and libtool then you can expect the compilation of
lots of things (not just squeak) to break.

> If you have the wrong awk, you don't get gnuification.

A while back you told me about Debian not using GNU awk, so I fixed the
problem:

2002-09-27
         * platforms/unix/config/acinclude.m4 (AC_GNU_INTERP): force
         AWK=gawk if --with-gnu-awk.

A little later I modified the search rules to find gawk before [mn]awk
on almost every conceivable platform.  Even then, setting AWK=blah on
the configure command line would override automatic detection (and
setting INTERP=gnu-interp would force gnuification even if configure
thought you were crazy to attempt it).

More recently you told me about a few things in the gnuify script which
were not particularly portable, so I fixed them too:

2004-04-10
         * platforms/unix/config/gnuify: Remove gnu specifics.

2004-04-11
         * platforms/unix/config/gnuify: Escape all occurences of '{'
         within regular expressions.

I don't think gnuify cares which awk you use any more.

> If you have the wrong version of autoconf, then the configure script
> may not be rebuildable.

autoconf has gone through at least two phase transitions since I
adopted it (under duress) for squeak.  At each transition, previous
autoconf.{in,ac} became incompatible with current and future versions
of autoconf, and automake (which provides aclocal) and libtool
contained attendant more-or-less configure-breaking changes.  I have a
fully-updated Debian 'testing' system on which autoconf produces a
configure script that runs fine on Debian, Yellow Dog,
{Free,Open,Net}BSD, Solaris and Mac OS X.  (I routinely regenerate
configure on the latter too, which is also a fully-updated system.)  As
Ned says: It Just Works.  If it's broken, then either you have an
incompatible (or obsolete) collection of autothings, or Debian and
Apple are both shipping broken development toolsets (which I doubt is
the case).

> Wrong version of gcc?  Your compile can fail.  etc.

There are only a handful of broken versions of gcc.  The bugs were
fixed long (18 months or more) ago.  The current 3.3 and 3.4 series
gccs can compile it all just fine on at least four different CPU
architectures (and a myriad of different software architectures), and
2.95 is still working fine the last time I checked.  You cannot hold up
a compiler bug and then point to the source code you're feeding to the
compiler as the cause of the bug.

> FWIW, the scripts are still not perfect even with all the if's
> satisfied.  My current system  builds without complaint, but the
> resulting VM cannot open an X window.

Again, all I can say is that it works fine on my three fully-updated
Debian systems running on a couple of different architectures.

Most likely you have an unresolved dependency on the plugin (due to a
bug in the source code, or something being rearranged in the x11 libs).
  These are hard to find because error reporting is turned way down
while loading the display and sound drivers.  Fetch the SVN version of
sqUnixExternalPrims.c (which has better diagnostics), set DEBUG to 1
near the top, and check if it's failing to load the plugin (and
report/fix the error indicated by the message it spews).

> Regarding the complexity of the script, I have several ideas.

Cast your mind back to 1996 or so.  The build process was wonderfully
simple.  There was one Makefile parameterised with bizarre compiler
flags for each supported architecture.  Several source files began with
sequences of symbol tests to determine which architecture they were on,
and how to get definitions (or workarounds) for idiosyncratic (or
missing) features.  As the number of architectures grew, the number of
combinations of bizarre flags and #ifdef trees grew.  When plugins cam
along the build process became more complex because the Makefile had to
include instructions for building only the plugins that were generated,
while supporting 'plug and play' addition of home-made plugins.  This
spawned a handful of scripts for stitching everything together into a
working Makefile.  Then a big grass roots movement (I will name no
names) starting screaming and yelling very publicly on squeak-dev that
it was all horribly complex and not being done the Open Source Way
(apparently that equates with The GNU Way for many people) and that
life would be oh so simple if we would all move to using autoconf (and
some even suggested automake -- no [publicly publishable] comment).  
When I got fed up of the derogatory remarks, I moved it all to
autoconf.  I doubt I'm the only person who has noticed that It has been
several times more complex, and many times more fragile, ever since.  
But whatever else happened, the derogatory remarks ceased (and so I was
lots happier).

> In fact, I have actually coded several of them up in the past, for
> whatever
> credibility that gives me.  Since those changes have been rejected--

I can't comment on them, since I only ever recall seeing (parts of) one
of them.  The main reasons I reject things are non-portability,
invasiveness on code outside the Unix tree, shifting (rather than
reducing) complexity, or reliance on non-standard tools.  (Although I'm
as guilty as anyone for relying on autoconf and friends, which have
failed spectacularly to standardise on anything in any given 5-year
period.)

> 1. Dump make.  We already depend on GNU make.  It is no more onerous to
> depend on Jam

Unless this is a demonstrable net reduction in complexity (rather than
just moving onerousity from one place to another), and until writing
Jamfiles becomes as natural to a randomly-selected developer as writing
a Makefile, this is not a good idea.  (I would reject Imake for the
same reason.)  I will, however, look at using Jam again.  Maybe a
parallel Jamfile in the trunk would be a good thing to at least
generate some discussion on whether to dump make entirely.

> 2. Use autoconf only for its purpose: *guessing* about configuration.
> Always let the user override the guess.

For the things that really matter, you can.  --with-x-includes=blah,
etc., plus the various environment variables (CC, CFLAGS, LDFLAGS,
etc.) that you can set on the command line when running configure.  I
routinely use the latter to do sanity checks of the configuration,
where I want to turn off all optimisation and just config/compile as
fast as possible.

> 3. Reconsider libtool.  It's a nice idea, but it is really confusing,
> and I'm not sure it is truly helping out.  It sucks having to read an
> extra manual just because we are too afraid to hard code "gcc -shared".

'-shared' works with gcc on GNU-based operating systems only.  I know
of no other Unix-like system on which generating a shared lib, with or
without gcc, is that simple.  I think you might benefit greatly from
using a Unix-like system from another (non-GNU-based) vendor for a
while.

> It would not be terrible, if the standard build scripts can only build
> external plugins if you are using gcc.

IOW, it would not be so terrible if building external plugins (which
include the display and sound drivers, remember) only worked on
GNU/Linux?

(I have explained elsewhere the details of why the display drivers
absolutely cannot be compiled internally because of logical [not
procedural] conflicts.  I will not repeat myself here.)

'-shared' is not a trivial panacea provided by gcc.  There are plenty
of platforms on which gcc runs fine, but '-shared' is totally
unsupported.  '-shared' is a feature that relies far more on the
linker's behaviour than the compiler's.  If the object format is not
quite right (or, obviously, if you're forced to use the vendor's ld
instead of gnu ld) then '-shared' is immediately a non-starter.

The move to libtool came after the move to autoconf and (again) was in
reaction to many complaints about builds failing on obscure platforms
or compilers that were not detected during configuration.  The problem
with explicit configuration of shared library compiler/linker flags is
that you need an expert on each platform who can determine just the
right flags to use.  On several platforms at the time, this was not the
case.  (And adopting flags supplied by one person could break the
compile for another person on the same platform running a slightly
different version of the OS or compiler -- this happened several times
for MIPS and Irix.)  The number of libtool-related complaints I have
received since moving to it are significantly less than the number of
library-related build failure complaints that I used to receive before
moving to libtool.

FWIW, I hate libtool as much as anyone.  But it has created a local
minimum in the quantity of hate mail that I receive.

> 4. Heck, while at it, reconsider the default of external plugins.
> There are clear advantages to external plugins, but they cause an
> installation hassle.  Especially it is annoying, if you want to compile
> an experimental VM and test it out without installing it on your
> system.
> Make internal the default, if VMMaker allows this.

You can make anything you like internal when you run VMMaker.  Internal
is the default in the VMs that I ship.  The only external plugins are
FFI (so that you can secure your system by removing the plugin, as
requested by lots of people) and things that link with non-standard
libraries (such as X11DpyCtl) or which have to defer their choice of
platform support to runtime (such as B3D, which requires OpenGL from
whatever source corresponds to the display driver you've chosen).

> They are complicated,

They are what people wanted (passionately and vocally) a long time ago.
  (But wants are somewhat like political parties: the grass is always
greener on the side you're currently facing...  But the devil almost
always comes out in the details once you get there.)

> few people understand them (maybe just 1?)

The autoconf stuff and attendant scripts are as simple as I could make
them.  Not one line of code in there is redundant.

The entire process is described to the best of my ability in the
HowToBuildFromSource document.  If something critical is not explained
in there, then nobody has complained about the omission.  So either the
document suffices, or it's so impenetrable that nobody (other than 1
person) understands any of it.

> they sometimes fail, they often require people to
> install extra dependencies, and when the the build fails, the error
> messages (or lack thereof) are frequently impenetrable.  Most of this
> is
> not necessary

I think I already made it quite clear that this complexity is not
necessary (and was not there in the pre-autoconf days, when the
complexity that people were complaining about was mostly perceived
rather than actual, and likely inspired by having to do something other
than './configure' as the first part of building a VM).

> and so we should welcome discussion (and code!) about how to improve
> them.

I will look again at Jam, but don't hold your breath.  (If it entirely
eradicates configure scripts [for features, headers and libraries] and
compiler/linker flag complexities, and supports dynamic selection of
build products based on the stuff VMM spits out [without having to run
external scripts] then I probably would consider that a net reduction
in complexity.)

Cheers,
Ian

PS --

> PS -- Alan, there are no Squeak Police that will come after you if you
> just do
> something like gcc *.c */*.c.

Try it.  It won't work.

There are a lot of threads here.  This is a discussion that would
probably work better in a bug tracker, with one thread per issue.  Also,
none of these are critical issues at the moment -- these are just all
possibilities for doing even better than the current system.

That said, for anyone who wants to perservere on these issues, here are
a few more cents.


> Ned wrote:
> > Assuming your system has a properly matched set of tools, installed
> > properly,
> > the configure script should work properly.
>
> Not really.  If you install completely incompatible versions of
> autoconf, automake and libtool then you can expect the compilation of
> lots of things (not just squeak) to break.

Yes.  All I'm saying is that it has proven easy to get incompatible
versions.  To some extent that's just life with autoconf, but also it is
probably possible to be more widely compatible than we are now.  Anyway,
let's not be so excited with what we do have, that we hide from what we
don't have.


> > If you have the wrong awk, you don't get gnuification.
>
> A while back you told me about Debian not using GNU awk, so I fixed the
> problem:
>
> 2002-09-27

Well, I just tried a few days ago and it did not work.  This was also
reported a few months ago on the mailing list, with a similar response
from you that the person must be mistaken.

I appreciate that you have worked on this, and I'm as baffled as you.
Anyway, given the existing work, it is surely just a one-line tweak
somewhere at this point.



> I don't think gnuify cares which awk you use any more.

Excellent.




> Ned says: It Just Works.
>  If it's broken, then either you have an
> incompatible (or obsolete) collection of autothings, or Debian and
> Apple are both shipping broken development toolsets (which I doubt is
> the case).

I think you'll retract this argument if you think about it a minute.
Blaming the user doesn't help!  I'd rather not go there, but if you
really stand by this, I can point out three holes in it.


> You cannot hold up
> a compiler bug and then point to the source code you're feeding to the
> compiler as the cause of the bug.

Likewise.  I agree that in some cases it is gcc's fault, but still it is
nice to support as many users as possible.




> Most likely you have an unresolved dependency on the plugin (due to a
> bug in the source code, or something being rearranged in the x11 libs).

Yes.  I probably have whatever X library the autoconf script is checking
for, but am missing some other ones.  Thanks for the debugging tip
(snipped), on how to track this down.




> > Regarding the complexity of the script, I have several ideas.
>
> Cast your mind back to 1996 or so.  The build process was wonderfully
> simple.  

Yes, I know.  There are tradeoffs.  And I know you did not make it
complicated on purpose.  The current situation is an improvement --
there are more people, now, for whom the build process just works.

It's nice to do even better, though, right?


> When I got fed up of the derogatory remarks, I moved it all to
> autoconf.

Just to be clear, I deeply appreciate all the work you've done.  If
Squeak-on-Unix had come out 6 months later, I probably wouldn't have
started using Squeak, and that in turn would likely have led me to being
a static weeny like *all* the other programming language guys I know
outside the Squeak group.  I'd probably be writing greek letters and
chasing the Curry-Howard isomorphism as a way to better computerdom.
*shudder*




> > 1. Dump make.  We already depend on GNU make.  It is no more onerous to
> > depend on Jam
>
> Unless this is a demonstrable net reduction in complexity (rather than
> just moving onerousity from one place to another), and until writing
> Jamfiles becomes as natural to a randomly-selected developer as writing
> a Makefile, this is not a good idea.

I agree entirely.  No tool is a silver bullet.  And even a good tool,
needs to be used tastefully, or it will grow into a new monster to
replace the old one.

That said, make is easy to beat.  subroutines....  if statements that
look like if statements....  lexical scope....  Need I say more?  These
describe almost every language on the planet, but not make, and not even
GNU make.

Aside from the general pleasantness of the language, Jam can remove
pressure on the shell scripts in the autoconf part of the build setup,
and give you a nicer separation between configuration scanning
(autoconf's job) and dependencies plus build rules (jam).


>  (I would reject Imake for the same reason.)

Agreed again.  I don't know imake well, but from what I've seen it is a
monster worse than the one it kills.  Good idea, bad implementation.


> > It would not be terrible, if the standard build scripts can only build
> > external plugins if you are using gcc.
>
> IOW, it would not be so terrible if building external plugins (which
> include the display and sound drivers, remember) only worked on
> GNU/Linux?

Good point.  FWIW, it is more than GNU systems, though.  I just tried on
Solaris, and -shared worked fine.  Yes, there are platform differences,
but I'm sure the gcc people are aware of this.




> The move to libtool came after the move to autoconf and (again) was in
> reaction to many complaints about builds failing on obscure platforms
> or compilers that were not detected during configuration.  
>
>   The number of libtool-related complaints I have
> received since moving to it are significantly less than the number of
> library-related build failure complaints that I used to receive before
> moving to libtool.

A good argument for keeping it.

I guess I am finding libtool itself to be messed up, even though it
solves an important problem--another good idea with a bad
implementation.  At one point, to give an example, the author was
religiously opposed to having libraries which link to other libraries.
Yet, many of Squeak's plugins--such as X11 and NAS--need to have .so's
linked to other dynamic libraries.

The general complexity of the interface is what really bothers me,
though.  I have skimmed the manual twice now -- and one day, I'm sure,
I'll read it again and really try to learn it -- but it seems like an
aweful lot of effort is required to learn this dratted tool.



> (I have explained elsewhere the details of why the display drivers
> absolutely cannot be compiled internally because of logical [not
> procedural] conflicts.  I will not repeat myself here.)

I missed that discussion, but how can this be?  X11 used to be compiled
internally, and that other Squeak ports can compile their display driver
internally.  I don't see how it can be a logical problem, when other
systems do it, and when Unix used to do it.

Also, what about systems that don't have dynamic linking?  Does
Squeak/Unix now *require* dynamic linking?


> > 4. Heck, while at it, reconsider the default of external plugins.
> > There are clear advantages to external plugins, but they cause an
> > installation hassle.  Especially it is annoying, if you want to compile
> > an experimental VM and test it out without installing it on your
> > system.
> > Make internal the default, if VMMaker allows this.
>
> You can make anything you like internal when you run VMMaker.


Fine, if VMMaker shackles what the build scripts can accomplish, then
there is nothing the Unix code can do.

None of this has to be the case, of course.  I actually worked on this a
bit at one time, but have not pursued it.  It really ought to be
possible to compile things internally or externally or even both, even
after the sources are generated.



-Lex

Tim Rowledge <[hidden email]> wrote:
> I was under the impression that the unix VMs
> normally had everything internal on the grounds of simplest installation; *nix
> seems to lack any sensible way to gather a bunch of files together to assemble
> an app. OSX has at last grasped the idea of an application directory in the
> manner of 18 year old RISC OS ideas, something that makes it really simple to
> have a single user-level entity that can have an arbitrary collection of code,
> resource, text and application files under it.

Exactly.  OSX's bundles made me see the light about what's been screamed
about by Mac guys and that RISC OS guy.  Surely nothing stops someone
from adding a "directory" or "bundle" executable format in Linux (and
other Unices).  Such a thing would be analagous to the "interpreter"
executable format that supports the #! hack.

But it's not there now.


-Lex