Why the hell directives for compiling with clang were not taken into account

 

 


The cost of abstraction... I am not a fan of the "build" scripts (a lot of code cloning for what[1] gain?) and the travis logs don't show us the invocation of the actual commands.

* One would need to pass CC to configure and make
* But even for configure CC=clang seems to go missing (https://api.travis-ci.org/v3/job/361174043/log.txt): "checking whether gcc -std=gnu99 is Clang... no"


Try executing the script with shell tracing on "sh -x ./build..." and see if CC is being passed a long? And cat config.log to see which compiler was used during the configuration step?


[1] I have worked with ARM, MIPS, PowerPC, x86 CPUs in my career and I know no other project that needs such build scripts to support different architectures and build flags...

 

 

 
On Monday 02 April 2018 07:21 PM, Nicolas Cellier wrote:
>
> Gah!
>
> despite my stuttutututter command:
>      CC=clang ./configure CC=clang
> the travis build is completely ignoring the directives and keep using gcc

I found the following clue in build.win32x86/common/Makefile.tools
-----
  10 ifeq ($(COMPILER_TO_USE),)
  11 COMPILER_TO_USE:=gcc
  12 endif
--------------------------

HTH .. Subbu

 

 

 
The extract was from win32 build only but on the Cog branch. My bad :-(.

ifeq can be tricky. Try replacing the ifeq in Makefile.tools with:

COMPILER_TO_USE ?= gcc

and then try

$ make COMPILER_TO_USE=clang

You could also add this target at the end of Makefile.

echo.% :
        @echo $($(*))

and then run

$ make COMPILER_TO_USE=clang echo.COMPILER_TO_USE

to debug your rules.

HTH .. Subbu

On Monday 02 April 2018 09:15 PM, Nicolas Cellier wrote:

 

Have you seen this?
        https://travis-ci.org/OpenSmalltalk/opensmalltalk-vm/jobs/361174043#L448

OTOH, I did it that way:
        https://github.com/OpenSmalltalk/opensmalltalk-vm/compare/krono/upgrade-gcc#diff-f36a4fd868404ca84d806453aabb6f3bR65

 

 

 
Hi Holger,

The complexity is due to the arcane nature of the Linux/Unix build system and its use of autoconf and several scripts to generate makefiles.  I have been saying for years that we need to rewrite the Linux build system to use convention makefiles as per Andreas' Windows makefiles and my subsequent Mac OS Makefiles.  I rewrote the Mac OS build system to use makefiles (*) and you'll notice that
- all share a small suite of makefiles in build.macosXXX/common
- all have a single top-level Makefile that defines the vm variant (the vm source directory) and some branding
- all mvm scripts under build.macosXXX are identical and serve only to invoke make with the three variants, production, assert & debug VMs
Indeed were it not for no symbolic link support for git/svn et al on windows the mvm script would be a symbolic link to a single file

Note that using the makefiles approach eliminates the separate build directories for production, assert & debug VMs


(*) rewriting the Mac OS build system to use makefiles was the path of least resistance when adding building of 64-bit Spur VMs because Xcode project files offer no abstraction and no sharing and are much much worse to edit than the Linux mvm scripts


So instead of bashing the awful little bud build system can we rewrite them together follow the windows and Mac OS scheme?  We would likely end up with one set of makefiles for all Linux builds apart from a separate rules file (compilation flags etc) per platform per word size.  And of course there would be a single mvm script and one would never need to run autoconfig and hence get the phase errors that result from some settings between my made when configure gets built, said me getting made when configure is run and yet others being made when make is run.

>
> * One would need to pass CC to configure and make
> * But even for configure CC=clang seems to go missing (https://api.travis-ci.org/v3/job/361174043/log.txt): "checking whether gcc -std=gnu99 is Clang... no"
>
>
> Try executing the script with shell tracing on "sh -x ./build..." and see if CC is being passed a long? And cat config.log to see which compiler was used during the configuration step?
>
>
> [1] I have worked with ARM, MIPS, PowerPC, x86 CPUs in my career and I know no other project that needs such build scripts to support different architectures and build flags...

 

Hi Holger,

On Apr 2, 2018, at 7:00 AM, Holger Freyther <[hidden email]> wrote:

On 2. Apr 2018, at 14:51, Nicolas Cellier <[hidden email]> wrote:

Gah!

despite my stuttutututter command:

  CC=clang ./configure CC=clang

the travis build is completely ignoring the directives and keep using gcc

See:

https://travis-ci.org/OpenSmalltalk/opensmalltalk-vm/jobs/361174043 <https://travis-ci.org/OpenSmalltalk/opensmalltalk-vm/jobs/361174043>

I suspect that it's lack of clean build, but not sure...

Building OpenSmalltalk VM in /home/travis/build/OpenSmalltalk/opensmalltalk-vm/build.linux32x86/pharo.sista.spur/build.itimerheartbeat...

<>Re-exec as x86

<>clean? ok but this isn't safe!!

Or would the clang compiler be absent?

Or what?

It ain't fun...

The cost of abstraction... I am not a fan of the "build" scripts (a lot of code cloning for what[1] gain?) and the travis logs don't show us the invocation of the actual commands.

The complexity is due to the arcane nature of the Linux/Unix build system and its use of autoconf and several scripts to generate makefiles.  I have been saying for years that we need to rewrite the Linux build system to use convention makefiles as per Andreas' Windows makefiles and my subsequent Mac OS Makefiles.  I rewrote the Mac OS build system to use makefiles (*) and you'll notice that
- all share a small suite of makefiles in build.macosXXX/common
- all have a single top-level Makefile that defines the vm variant (the vm source directory) and some branding
- all mvm scripts under build.macosXXX are identical and serve only to invoke make with the three variants, production, assert & debug VMs
Indeed were it not for no symbolic link support for git/svn et al on windows the mvm script would be a symbolic link to a single file

Note that using the makefiles approach eliminates the separate build directories for production, assert & debug VMs


(*) rewriting the Mac OS build system to use makefiles was the path of least resistance when adding building of 64-bit Spur VMs because Xcode project files offer no abstraction and no sharing and are much much worse to edit than the Linux mvm scripts


So instead of bashing the awful little bud build system can we rewrite them together follow the windows and Mac OS scheme?  We would likely end up with one set of makefiles for all Linux builds apart from a separate rules file (compilation flags etc) per platform per word size.  And of course there would be a single mvm script and one would never need to run autoconfig and hence get the phase errors that result from some settings between my made when configure gets built, said me getting made when configure is run and yet others being made when make is run.

* One would need to pass CC to configure and make

* But even for configure CC=clang seems to go missing (https://api.travis-ci.org/v3/job/361174043/log.txt): "checking whether gcc -std=gnu99 is Clang... no"

Try executing the script with shell tracing on "sh -x ./build..." and see if CC is being passed a long? And cat config.log to see which compiler was used during the configuration step?

[1] I have worked with ARM, MIPS, PowerPC, x86 CPUs in my career and I know no other project that needs such build scripts to support different architectures and build flags...

 
On Mon, Apr 02, 2018 at 03:54:15PM -0700, Eliot Miranda wrote:
Hi Eliot,

To be perfectly honest, I do not know what it meant by "we need to rewrite the Linux
build system to use conventional makefiles". Using makefiles would be a given for any
build system on unix/Linux, the differences (and pain points) are in the configuration
step. Regardless of whether you use autoconf or CMake or something else, we would still
need something to specify the environment in which the makefile is operating. In other
words, we need a build/config.h to specify the things that the makefile does not know
about. I am pretty sure I am misunderstanding something in this discussion, but I am
comfused and I may as well admit it.

Dave

 
Speaking of which, I would love it if CMake were used if for no other reason CMake can generate an Xcode project which would make for much easier debugging and porting to iOS.

 
Hi David,

I'm feeling beaten down.  Perhaps you could read the code?  Compare what's in
    platforms/unix/config
with what's in
    build.macos32x86/common
    build.macos64x64/common
    build.win32x86/common
    build.win64x64/common

And then think about the "necessary configure step".  To what extent does it make sense to run a configure step on a build machine that makes a binary that is downloaded and used on machines that may differ markedly from that build machine?



>
> Dave
>

 
On Tue, Apr 03, 2018 at 05:30:06AM -0700, Eliot Miranda wrote:
Hi Eliot,

First an apology - this was not a very good time to be asking a dumb question.

Yes I did look at the code, but probably not closely enough. I will do so.

Dave

 
On Tuesday 03 April 2018 06:00 PM, Eliot Miranda wrote:
> I'm feeling beaten down.  Perhaps you could read the code?  Compare
> what's in platforms/unix/config with what's in
> build.macos32x86/common build.macos64x64/common build.win32x86/common
> build.win64x64/common
>
> And then think about the "necessary configure step".  To what extent
> does it make sense to run a configure step on a build machine that
> makes a binary that is downloaded and used on machines that may
> differ markedly from that build machine?

The configure script in Autoconf accumulates lots of platform-fu (see
configure:5668) for all the UNIX/Linux variants and quirks.

Mac or Win ports have to deal only with a handful of variants, so
hand-crafted Makefiles are sufficient. But Unix/GNU/Linux ports are so
numerous and diverse that Ian Piumarta (?) brought in autoconf to ease
porting Squeak to all these variants.

If we don't intend to support all the 100+ variants of GNU/UNIX/Linux
out there but only a handful, then manual makefiles are sufficient.

GNU Make is available on GNU/Linux/Mac/Win so we could create a single
top level Makefile for all three platform builds and let the configure
handle other ports. CMake is another possibility but it requires meta
makefiles to be created from scratch :-(.

Regards .. Subbu

 
Hi Subbu

> On Apr 3, 2018, at 7:34 AM, K K Subbu <[hidden email]> wrote:
>
>> On Tuesday 03 April 2018 06:00 PM, Eliot Miranda wrote:
>> I'm feeling beaten down.  Perhaps you could read the code?  Compare what's in platforms/unix/config with what's in build.macos32x86/common build.macos64x64/common build.win32x86/common
>> build.win64x64/common
>> And then think about the "necessary configure step".  To what extent does it make sense to run a configure step on a build machine that makes a binary that is downloaded and used on machines that may differ markedly from that build machine?
>
> The configure script in Autoconf accumulates lots of platform-fu (see
> configure:5668) for all the UNIX/Linux variants and quirks.

Right.

>
> Mac or Win ports have to deal only with a handful of variants, so
> hand-crafted Makefiles are sufficient. But Unix/GNU/Linux ports are so
> numerous and diverse that Ian Piumarta (?) brought in autoconf to ease
> porting Squeak to all these variants.

I see things a little differently.  I don't want to speak for Ian, but I expect that the reason Ian went the way he did was more to do with the fact that to run Squeak one had to build a vm oneself. Being a Unix expert Ian could support a build of (at first just) an X11 vm across a wide range of Unix variants at a time (the early 90's) when many manufactures were oroducing Unix workstations which were widely used, especially in academia.  That predated a lot of culture we now take god granted, the downloading of prepackaged releases, automated build and test environments, and a much reduced more consumer-oriented laptop market.

> If we don't intend to support all the 100+ variants of GNU/UNIX/Linux out there but only a handful, then manual makefiles are sufficient.

It's not that we do of do not intend, it's that we simply cannot afford to and it is not relevant to our current needs.  We instead need to support a much smaller number of operating systems well, namely Linux and maybe FreeBSD, although I don't see much energy there.  We could keep the old build system for those that want to build their own vm on something exotic (although I'd still argue that they would do as well working with conventional makefiles).  But it is no longer fit for purpose in the continuous integration context we now inhabit.

>
> GNU Make is available on GNU/Linux/Mac/Win so we could create a single top level Makefile for all three platform builds and let the configure
> handle other ports.

Having modified Andreas' makefiles for windows, and having written the makefiles for the Mac build I can say that a common top level makefiles is not a good idea.  It would be even more of a mess of ifdefs.  But certain not following the way nudes and Mac OS schemes is sensible.  The important thing is to get rid of the extremely hard to modify, slow autoconf based system used for Linux builds.

> CMake is another possibility but it requires meta makefiles to be created from scratch :-(.

In a vm text where build slaves build binaries god more general consumption the specificity of CMake and autoconfig is of marginal utility.  The Windows and Mac OS sources still have config.h's; they simply aren't generated on every build.  The information in them doesn't change in practice, except when a major change, such as the addition of s 64-bit build, occurs, and at that point a lot more work than merely updating config.h is needed (although IIRC  the mac's config.h didn't change when adding 64-bit Spur).

The main utility in CMake and autoconf is /not/ capturing platform fu, but in capturing what ancillary packages have been installed to know what additional plugins to build; for example the 3D plugins are only built if OpenGL is installed.  But in our CI infrastructure we make damn sure OpenGL is installed on the build machines so we. An provide VMs comprising a full set of plugins, some of which may not work on the user's machine because they haven't installed the libraries.

Let's have a build system that suits where we are now, where the vm is a utility we want to build reliably and download, not something we get a frisson form building ourselves.  Let's leave that yo those that work on the vm and let the rest of us get timely CI vm builds we can download at our leisure.

> Regards .. Subbu

Cheers!
Eliot

 
On Tuesday 03 April 2018 08:54 PM, Eliot Miranda wrote:

Valid point. The CI pipeline requires a different approach.

A top level Makefile does not exclude platform-specific Makefiles, nor
should it attempt a cross-platform build by itself.

The purpose of top level Makefile is general help, handle version tags,
possible products and variants etc. It should delegate the actual build
to platform-specific make. Similar to how we use OSProcess as a facade
over OSUnixProcess etc.

In fact, I am making one for my own use. Once it stabilizes, I will
offer a PR.

The downsides to spreading Makefiles into build dirs is explosion of
directories under version control and duplication of code. It also means
that I cannot mount build on tmpfs/ramfs. Keeping Makefile templates
under platform/win32/.. and copying them to build/ as needed will
simplify version control and avoid duplicate code.

Regards .. Subbu