A Bounty for CMake-ifying stack/Cog vm build process

Hi Timothy,

On Thu, Nov 7, 2013 at 12:33 PM, gettimothy <[hidden email]> wrote:

Eliot,

Thank you so much for the reply.

Currently my low-level skills are atrophied and have organized my work-life to devote study time to regaining and improving them.
This is a great opportunity to contribute and get my chops up in the process.

When you refer to 'machine code' and 'machine organization' do you mean the Cog/Squeak VM's or the AMD 64 instruction set? (I lack both, but if you mean the VM, a good exercise in learning it would be writing some good documentation on it)

I mean good low-level programming skills.  Knowing how stacks and address spaces are organized, what the machine's calling convention is, what a system call looks like, etc, and what the current processor's instruction set is, it's theory etc (is it a RISC or a CISC; does it have delay slots, etc, etc).

So first things first.  Get the stack VM wokring in 64-bits.

From your reply, I guess I would best be of use getting the Stack VM working in 64 bits--its something I am currently attempting to do.

I have a tri-boot machine--Windows 7, Linux with 32 bit compat libs--where I can run smalltalk, and a pure 64 bit boot, where I cannot run smalltalk.

OK, but right now you'll want to be in Smalltalk, and for quite a while.

 

I am currently attempting  to get the source code from  http://www.squeakvm.org/squeak64/ Squeak-3.8a-2.src.tar.gz to compile on the pure 64 box.
The Quick Start steps failed, so I am now setting about studying the source code in merge64/platforms/unix/vm/sqUnixMain.c as it looks like the entry point for the VM.
In the back of my mind, I am wondering if this is even the right approach to take..

It's not :-).

 

What do you think is the best approach to take?

Read the latest two articles on my blog (A Spur gear for Cog & Lazy Become).

Make a VMMaker image (see e.g. Cog Blog :: Building a Cog Development Image).

Then read SpurMemoryManager, /especially/ its class comment.

Then read its subclasses Spur32BitMemoryManager (functional) & Spur64BitMemoryManager (completely untested).

Then read SpurBootstrap32.

Try and create a 32-bit Spur image and run it.

Then try and implement SpurBootstrap64.

Try and create a 64-bit Spur image and run it.

This is /lots/ of work.  In particular the last two steps require some significant design.

So start slowly and see how you get on.

 

Cordially,

t

---- On Thu, 07 Nov 2013 10:54:01 -0800 Eliot Miranda<[hidden email]> wrote ----

Hi Timothy,

On Thu, Nov 7, 2013 at 7:02 AM, gettimothy <[hidden email]> wrote:

Is there any chance Cog could be made to run on AMD64 natively without 32 bit libs?

Yes, that's a goal of my current work.  Spur has a common object representation between 32- and 64-bits.  Getting to a native Stack VM on AMD64 should be straight-forward.  Getting to a Cog VM on AMD64 also requires writing a code generator for AMD64.  That's actually a fun project.  It involves configuring a Bochs plugin for an AMD64/IA32 simulator, and then writing a CogIA64Compiler subclass of CogAbstractInstruction.

If so, I would like to help out with whoever takes up this task. I would need direction, however as I have been do app development for too many years and my sys skills are atrophied.

How strong are your low-level skills?  You'll need to be quite familiar with machine code, basic machine organization, etc.  If you're not then you could instead have a go at getting the Stack VM working in 64-bits, and that would involve getting the 64-bit VM simulator working in VMMaker.

It will be great to have another collaborator!

t.

---- On Thu, 07 Nov 2013 06:01:00 -0800 Bob Arning<[hidden email]> wrote ----

It's absolutely none of my business what tools others choose to use, but I will say this:

Having more of this process in Smalltalk and less in other (arcane) languages increases the confidence that, should we lose someone's services, others could step up to fill the gap.

Cheers,
Bob

On 11/7/13 8:28 AM, Igor Stasenko wrote:

[hidden email]" type="cite">

On 7 November 2013 08:58, Tobias Pape <[hidden email]> wrote:

Hey Tim and all,

On 07.11.2013, at 05:00, Ron Teitelbaum <[hidden email]> wrote:

> Hey Tim,
>
> Göran and Eliot have been discussing this.  I just talked to Eliot about it
> yesterday.  He said he was happy to support cmake but didn't want to have to
> do the work himself.  I'm hoping that the work Göran is doing on this now
> can be turned over to Eliot when it is finished.  I think there is agreement
> that combining the excellent work of pharo build with Eliot's new vm work is
> a very good thing to do.  The work to merge the build environments is
> already being done at 3D ICC.  We are working on this for the Mac also but
> have not discussed it with Ian yet.
>
> Ron Teitelbaum
>
>> -----Original Message-----
>> From: [hidden email] [mailto:[hidden email]
>> [hidden email]] On Behalf Of tim Rowledge
>> Sent: Wednesday, November 06, 2013 9:04 PM
>> To: The general-purpose Squeak developers list
>> Subject: Re: [squeak-dev] A Bounty for CMake-ifying stack/Cog vm build
> process
>>
>>
>> Wow, no interest? Is it because make is painful or because nobody needs to
>> make any more money?

I actually have interest but I am currently reluctant,
for time-capturing reasons in my personal and work life.

However, Here is what I would do:

• Start from Ians CMake work, and try to generalize it for the
  non-Linux platforms
• Windows would be kind-of straight-forward,
• For OS X, I would really want to have the Cocoa-based VM stuff
  (aka Squeak VM 5.4.7.x) integrated and proper Xcode files
  generated, especially to be able to have iOS VMs built automatically.
   Here, some information from John McIntosh would be helpfull, especially
  a simple walk-through.
   I have some experience with that[1]


Despite all admiration I have for the way the PharoVM is built, there is one
thing that bothers me:
• You first somehow generate the VM sources (which is fine, and the automated way is amazing)
• THEN, the a Script generates CMake-Files
• CMake then generates Makefiles
• and then finally a vm is compiled.
This is one generating step too much, for my taste.
Personally, I'd prefer CMake to just pick up the C-files generated by VMMaker,
so that adding a new platform does not need to change the VMMaker…

The main big issue with 'platform-neutral' static source files, that they tend to

grow with tons of #ifdef-s over time, reducing readability and creating a mess.

My point is that one or another way you have to deal with platform differences,

and the way how i prefer to do it is using smalltalk code,

but not .m4 files, or .cmake files, or writing a medium novel long configuration files

using strange (better to say weird) scripting DSL.

I can express all build process logic in smalltalk, which does not lessens the amount
of work to deal with all platform nuances and dependencies etc, but at least it allows me to organize and shape it
into some manageable form, which is easy to change & learn.

 

 And another difference is, that i prefer working with browser & classes than
with dozens of directories and files lying here and there.

But sure thing, it is a question of taste. If you prefer to maintain this:
-------------------------
MACRO (INTERNAL_PLUGIN plugin)
  SET (plugin_sources "")
  IF (DEFINED ${plugin}_sources)
    SET (plugin_sources ${${plugin}_sources})
  ELSE (DEFINED ${plugin}_sources)
    FOREACH (dir ${src}/vm/intplugins ${cross}/plugins ${unix}/plugins)
      SET (tmp "")
      AUX_SOURCE_DIRECTORY (${dir}/${plugin} tmp)
      STRING_APPEND (plugin_sources "${tmp}")
    ENDFOREACH (dir)
  ENDIF (DEFINED ${plugin}_sources)
  IF (DEFINED ${plugin}_extra_sources)
    STRING_APPEND (plugin_sources "${${plugin}_extra_sources}")
  ENDIF (DEFINED ${plugin}_extra_sources)
  FILE (WRITE ${bld}/${plugin}/CMakeLists.in "")
  FOREACH (dir ${unix}/plugins ${unix})
    FILE_APPEND (${bld}/${plugin}/CMakeLists.in ${dir}/${plugin}/build.cmake)
  ENDFOREACH (dir)
  FILE_APPEND (${bld}/${plugin}/CMakeLists.in ${config}/PluginInternal.cmake)
  CONFIGURE_FILE (${bld}/${plugin}/CMakeLists.in ${bld}/${plugin}/CMakeLists.txt @ONLY)
  ADD_SUBDIRECTORY (${bld}/${plugin} ${bld}/${plugin})
ENDMACRO (INTERNAL_PLUGIN)
--------------------------

And/or this:
--------------------------
AC_INIT([config.h.in])

SVNREV=`grep '\$Rev: ' ${srcdir}/../../../platforms/Cross/vm/sqSCCSVersion.h \
        | sed 's/.*$Rev: \([[0-9]][[0-9]]*\).*/\1/'`
AC_VM_VERSION(4,0,${SVNREV},4,2,0)

topdir=`cd ${srcdir}/../../..; pwd`
cfgdir=`cd ${srcdir}; pwd`

AC_ARG_WITH(src,
[  --with-src=dir          generated src directory [default=src]],
[ vmmsrc="${with_src}"],
[ vmmsrc="src"])

vmmdir="${topdir}/${vmmsrc}"

if test ! -d "${topdir}/${vmmsrc}"; then
  if test -d "${topdir}/platforms/unix/${vmmsrc}"; then
    vmmdir="${topdir}/platforms/unix/${vmmsrc}"
    AC_MSG_RESULT([using built-in src directory])
  fi
fi

AC_MSG_RESULT(${vmmdir})

blddir=`pwd`

AC_ARG_WITH(vmmcfg,
[  --with-vmmcfg=dir        vm configuration directory containing plugins.int and plugins.ext [default=.]],
[ vmmcfg="${with-vmmcfg}"],
[ vmmcfg="${blddir}"])

# Compiling a Cogit VM or not?  If so, need a cogit$o, cointerp, etc.

AC_ARG_ENABLE(cogit,
[  --enable-cogit            compile a cogit VM [default=yes]],
cogit="no", cogit="yes")
AC_SUBST(cogit)

#AC_ARG_ENABLE(jit,
#[  --enable-jit            enable J4 support [default=no]],
#JIT="yes", JIT="no")
#
#test $JIT = "yes" && J_CFLAGS="-DJ_ENABLED"
#AC_SUBST(J_CFLAGS)

# Check the generated src dir looks sane

AC_CHECK_VMM_DIR

AC_SUBST(topdir)
AC_SUBST(cfgdir)
AC_SUBST(vmmdir)
AC_SUBST(vmmcfg)
AC_SUBST(blddir)

SQ_VERSION=${SQ_MAJOR}.${SQ_MINOR}-${SQ_UPDATE}

AC_DEFINE_UNQUOTED(SQ_VERSION, "${SQ_VERSION}")

AC_SUBST(SQ_MAJOR)
AC_SUBST(SQ_MINOR)
AC_SUBST(SQ_UPDATE)
AC_SUBST(SQ_VERSION)

VM_VERSION=${VM_MAJOR}.${VM_MINOR}-${VM_RELEASE}

AC_DEFINE_UNQUOTED(VM_VERSION, "${VM_VERSION}")

AC_SUBST(VM_MAJOR)
AC_SUBST(VM_MINOR)
AC_SUBST(VM_RELEASE)
AC_SUBST(VM_VERSION)

# libdir contains ${exec_prefix}, so we have to default and expand early
test "x$prefix" = xNONE && prefix=$ac_default_prefix
test "x$exec_prefix" = xNONE && exec_prefix='${prefix}'
imgdir=`eval echo ${libdir}/squeak`
expanded_relative_imgdir=`eval echo lib/squeak/${VM_VERSION}`
plgdir='${imgdir}/'`eval echo ${VM_VERSION}`

AC_SUBST(imgdir)
AC_SUBST(expanded_relative_imgdir)
AC_SUBST(plgdir)

AC_DEFINE(OS_TYPE, "unix")

AC_CANONICAL_HOST

host_cpu=`echo $host | sed 's,-.*,,'`
host=`echo $host | sed 's,-unknown,,'`

AC_SUBST(host)
AC_SUBST(host_cpu)
AC_SUBST(host_vendor)
AC_SUBST(host_os)
----------------------------

instead of plain smalltalk code, it is up to you.

But that's about 'too much' for my taste.

But if the community is positive to just move to the Pharo way of
building VMs, I'd be fine with that, since that would unify a lot
of infrastructure :)

Best
        -Tobias

[1] https://github.com/krono/self/blob/cmake/vm/cmake/mac_osx.cmake

--
Best regards,
Igor Stasenko.

--
best,

Eliot

--
best,

Eliot

Hi Timothy,

On Thu, Nov 7, 2013 at 12:33 PM, gettimothy <[hidden email]> wrote:

Eliot,

Thank you so much for the reply.

Currently my low-level skills are atrophied and have organized my work-life to devote study time to regaining and improving them.
This is a great opportunity to contribute and get my chops up in the process.

When you refer to 'machine code' and 'machine organization' do you mean the Cog/Squeak VM's or the AMD 64 instruction set? (I lack both, but if you mean the VM, a good exercise in learning it would be writing some good documentation on it)

I mean good low-level programming skills.  Knowing how stacks and address spaces are organized, what the machine's calling convention is, what a system call looks like, etc, and what the current processor's instruction set is, it's theory etc (is it a RISC or a CISC; does it have delay slots, etc, etc).

So first things first.  Get the stack VM wokring in 64-bits.

From your reply, I guess I would best be of use getting the Stack VM working in 64 bits--its something I am currently attempting to do.

I have a tri-boot machine--Windows 7, Linux with 32 bit compat libs--where I can run smalltalk, and a pure 64 bit boot, where I cannot run smalltalk.

OK, but right now you'll want to be in Smalltalk, and for quite a while.

 

I am currently attempting  to get the source code from  http://www.squeakvm.org/squeak64/ Squeak-3.8a-2.src.tar.gz to compile on the pure 64 box.
The Quick Start steps failed, so I am now setting about studying the source code in merge64/platforms/unix/vm/sqUnixMain.c as it looks like the entry point for the VM.
In the back of my mind, I am wondering if this is even the right approach to take..

It's not :-).

 

What do you think is the best approach to take?

Read the latest two articles on my blog (A Spur gear for Cog & Lazy Become).

Make a VMMaker image (see e.g. Cog Blog :: Building a Cog Development Image).

Then read SpurMemoryManager, /especially/ its class comment.

Then read its subclasses Spur32BitMemoryManager (functional) & Spur64BitMemoryManager (completely untested).

Then read SpurBootstrap32.

Try and create a 32-bit Spur image and run it.

Then try and implement SpurBootstrap64.

Try and create a 64-bit Spur image and run it.

This is /lots/ of work.  In particular the last two steps require some significant design.

So start slowly and see how you get on.

 

Cordially,

t

---- On Thu, 07 Nov 2013 10:54:01 -0800 Eliot Miranda<[hidden email]> wrote ----

Hi Timothy,

On Thu, Nov 7, 2013 at 7:02 AM, gettimothy <[hidden email]> wrote:

Is there any chance Cog could be made to run on AMD64 natively without 32 bit libs?

Yes, that's a goal of my current work.  Spur has a common object representation between 32- and 64-bits.  Getting to a native Stack VM on AMD64 should be straight-forward.  Getting to a Cog VM on AMD64 also requires writing a code generator for AMD64.  That's actually a fun project.  It involves configuring a Bochs plugin for an AMD64/IA32 simulator, and then writing a CogIA64Compiler subclass of CogAbstractInstruction.

If so, I would like to help out with whoever takes up this task. I would need direction, however as I have been do app development for too many years and my sys skills are atrophied.

How strong are your low-level skills?  You'll need to be quite familiar with machine code, basic machine organization, etc.  If you're not then you could instead have a go at getting the Stack VM working in 64-bits, and that would involve getting the 64-bit VM simulator working in VMMaker.

It will be great to have another collaborator!

t.

---- On Thu, 07 Nov 2013 06:01:00 -0800 Bob Arning<[hidden email]> wrote ----

It's absolutely none of my business what tools others choose to use, but I will say this:

Having more of this process in Smalltalk and less in other (arcane) languages increases the confidence that, should we lose someone's services, others could step up to fill the gap.

Cheers,
Bob

On 11/7/13 8:28 AM, Igor Stasenko wrote:

[hidden email]" type="cite">

On 7 November 2013 08:58, Tobias Pape <[hidden email]> wrote:

Hey Tim and all,

On 07.11.2013, at 05:00, Ron Teitelbaum <[hidden email]> wrote:

> Hey Tim,
>
> Göran and Eliot have been discussing this.  I just talked to Eliot about it
> yesterday.  He said he was happy to support cmake but didn't want to have to
> do the work himself.  I'm hoping that the work Göran is doing on this now
> can be turned over to Eliot when it is finished.  I think there is agreement
> that combining the excellent work of pharo build with Eliot's new vm work is
> a very good thing to do.  The work to merge the build environments is
> already being done at 3D ICC.  We are working on this for the Mac also but
> have not discussed it with Ian yet.
>
> Ron Teitelbaum
>
>> -----Original Message-----
>> From: [hidden email] [mailto:[hidden email]
>> [hidden email]] On Behalf Of tim Rowledge
>> Sent: Wednesday, November 06, 2013 9:04 PM
>> To: The general-purpose Squeak developers list
>> Subject: Re: [squeak-dev] A Bounty for CMake-ifying stack/Cog vm build
> process
>>
>>
>> Wow, no interest? Is it because make is painful or because nobody needs to
>> make any more money?

I actually have interest but I am currently reluctant,
for time-capturing reasons in my personal and work life.

However, Here is what I would do:

• Start from Ians CMake work, and try to generalize it for the
  non-Linux platforms
• Windows would be kind-of straight-forward,
• For OS X, I would really want to have the Cocoa-based VM stuff
  (aka Squeak VM 5.4.7.x) integrated and proper Xcode files
  generated, especially to be able to have iOS VMs built automatically.
   Here, some information from John McIntosh would be helpfull, especially
  a simple walk-through.
   I have some experience with that[1]


Despite all admiration I have for the way the PharoVM is built, there is one
thing that bothers me:
• You first somehow generate the VM sources (which is fine, and the automated way is amazing)
• THEN, the a Script generates CMake-Files
• CMake then generates Makefiles
• and then finally a vm is compiled.
This is one generating step too much, for my taste.
Personally, I'd prefer CMake to just pick up the C-files generated by VMMaker,
so that adding a new platform does not need to change the VMMaker…

The main big issue with 'platform-neutral' static source files, that they tend to

grow with tons of #ifdef-s over time, reducing readability and creating a mess.

My point is that one or another way you have to deal with platform differences,

and the way how i prefer to do it is using smalltalk code,

but not .m4 files, or .cmake files, or writing a medium novel long configuration files

using strange (better to say weird) scripting DSL.

I can express all build process logic in smalltalk, which does not lessens the amount
of work to deal with all platform nuances and dependencies etc, but at least it allows me to organize and shape it
into some manageable form, which is easy to change & learn.

 

 And another difference is, that i prefer working with browser & classes than
with dozens of directories and files lying here and there.

But sure thing, it is a question of taste. If you prefer to maintain this:
-------------------------
MACRO (INTERNAL_PLUGIN plugin)
  SET (plugin_sources "")
  IF (DEFINED ${plugin}_sources)
    SET (plugin_sources ${${plugin}_sources})
  ELSE (DEFINED ${plugin}_sources)
    FOREACH (dir ${src}/vm/intplugins ${cross}/plugins ${unix}/plugins)
      SET (tmp "")
      AUX_SOURCE_DIRECTORY (${dir}/${plugin} tmp)
      STRING_APPEND (plugin_sources "${tmp}")
    ENDFOREACH (dir)
  ENDIF (DEFINED ${plugin}_sources)
  IF (DEFINED ${plugin}_extra_sources)
    STRING_APPEND (plugin_sources "${${plugin}_extra_sources}")
  ENDIF (DEFINED ${plugin}_extra_sources)
  FILE (WRITE ${bld}/${plugin}/CMakeLists.in "")
  FOREACH (dir ${unix}/plugins ${unix})
    FILE_APPEND (${bld}/${plugin}/CMakeLists.in ${dir}/${plugin}/build.cmake)
  ENDFOREACH (dir)
  FILE_APPEND (${bld}/${plugin}/CMakeLists.in ${config}/PluginInternal.cmake)
  CONFIGURE_FILE (${bld}/${plugin}/CMakeLists.in ${bld}/${plugin}/CMakeLists.txt @ONLY)
  ADD_SUBDIRECTORY (${bld}/${plugin} ${bld}/${plugin})
ENDMACRO (INTERNAL_PLUGIN)
--------------------------

And/or this:
--------------------------
AC_INIT([config.h.in])

SVNREV=`grep '\$Rev: ' ${srcdir}/../../../platforms/Cross/vm/sqSCCSVersion.h \
        | sed 's/.*$Rev: \([[0-9]][[0-9]]*\).*/\1/'`
AC_VM_VERSION(4,0,${SVNREV},4,2,0)

topdir=`cd ${srcdir}/../../..; pwd`
cfgdir=`cd ${srcdir}; pwd`

AC_ARG_WITH(src,
[  --with-src=dir          generated src directory [default=src]],
[ vmmsrc="${with_src}"],
[ vmmsrc="src"])

vmmdir="${topdir}/${vmmsrc}"

if test ! -d "${topdir}/${vmmsrc}"; then
  if test -d "${topdir}/platforms/unix/${vmmsrc}"; then
    vmmdir="${topdir}/platforms/unix/${vmmsrc}"
    AC_MSG_RESULT([using built-in src directory])
  fi
fi

AC_MSG_RESULT(${vmmdir})

blddir=`pwd`

AC_ARG_WITH(vmmcfg,
[  --with-vmmcfg=dir        vm configuration directory containing plugins.int and plugins.ext [default=.]],
[ vmmcfg="${with-vmmcfg}"],
[ vmmcfg="${blddir}"])

# Compiling a Cogit VM or not?  If so, need a cogit$o, cointerp, etc.

AC_ARG_ENABLE(cogit,
[  --enable-cogit            compile a cogit VM [default=yes]],
cogit="no", cogit="yes")
AC_SUBST(cogit)

#AC_ARG_ENABLE(jit,
#[  --enable-jit            enable J4 support [default=no]],
#JIT="yes", JIT="no")
#
#test $JIT = "yes" && J_CFLAGS="-DJ_ENABLED"
#AC_SUBST(J_CFLAGS)

# Check the generated src dir looks sane

AC_CHECK_VMM_DIR

AC_SUBST(topdir)
AC_SUBST(cfgdir)
AC_SUBST(vmmdir)
AC_SUBST(vmmcfg)
AC_SUBST(blddir)

SQ_VERSION=${SQ_MAJOR}.${SQ_MINOR}-${SQ_UPDATE}

AC_DEFINE_UNQUOTED(SQ_VERSION, "${SQ_VERSION}")

AC_SUBST(SQ_MAJOR)
AC_SUBST(SQ_MINOR)
AC_SUBST(SQ_UPDATE)
AC_SUBST(SQ_VERSION)

VM_VERSION=${VM_MAJOR}.${VM_MINOR}-${VM_RELEASE}

AC_DEFINE_UNQUOTED(VM_VERSION, "${VM_VERSION}")

AC_SUBST(VM_MAJOR)
AC_SUBST(VM_MINOR)
AC_SUBST(VM_RELEASE)
AC_SUBST(VM_VERSION)

# libdir contains ${exec_prefix}, so we have to default and expand early
test "x$prefix" = xNONE && prefix=$ac_default_prefix
test "x$exec_prefix" = xNONE && exec_prefix='${prefix}'
imgdir=`eval echo ${libdir}/squeak`
expanded_relative_imgdir=`eval echo lib/squeak/${VM_VERSION}`
plgdir='${imgdir}/'`eval echo ${VM_VERSION}`

AC_SUBST(imgdir)
AC_SUBST(expanded_relative_imgdir)
AC_SUBST(plgdir)

AC_DEFINE(OS_TYPE, "unix")

AC_CANONICAL_HOST

host_cpu=`echo $host | sed 's,-.*,,'`
host=`echo $host | sed 's,-unknown,,'`

AC_SUBST(host)
AC_SUBST(host_cpu)
AC_SUBST(host_vendor)
AC_SUBST(host_os)
----------------------------

instead of plain smalltalk code, it is up to you.

But that's about 'too much' for my taste.

But if the community is positive to just move to the Pharo way of
building VMs, I'd be fine with that, since that would unify a lot
of infrastructure :)

Best
        -Tobias

[1] https://github.com/krono/self/blob/cmake/vm/cmake/mac_osx.cmake

--
Best regards,
Igor Stasenko.

--
best,

Eliot

--
best,

Eliot

On Thu, Nov 7, 2013 at 3:16 PM, gettimothy <[hidden email]> wrote:

Eliot,

Re "I mean good low-level programming skills.  Knowing how stacks and address spaces are organized, what the machine's calling convention is, what a system call looks like, etc, and what the current processor's instruction set is, it's theory etc (is it a RISC or a CISC; does it have delay slots, etc, etc)."

Would an updated edition of Patterson and Hennessy be helpful here? My copy (like my c skills) is from 1993.

nothing fundamental that affects Cog has changed in that time... it'll do just fine.

 

t.

---- On Thu, 07 Nov 2013 13:54:03 -0800 Eliot Miranda<[hidden email]> wrote ----

Hi Timothy,

On Thu, Nov 7, 2013 at 12:33 PM, gettimothy <[hidden email]> wrote:

Eliot,

Thank you so much for the reply.

Currently my low-level skills are atrophied and have organized my work-life to devote study time to regaining and improving them.
This is a great opportunity to contribute and get my chops up in the process.

When you refer to 'machine code' and 'machine organization' do you mean the Cog/Squeak VM's or the AMD 64 instruction set? (I lack both, but if you mean the VM, a good exercise in learning it would be writing some good documentation on it)

I mean good low-level programming skills.  Knowing how stacks and address spaces are organized, what the machine's calling convention is, what a system call looks like, etc, and what the current processor's instruction set is, it's theory etc (is it a RISC or a CISC; does it have delay slots, etc, etc).

So first things first.  Get the stack VM wokring in 64-bits.

From your reply, I guess I would best be of use getting the Stack VM working in 64 bits--its something I am currently attempting to do.

I have a tri-boot machine--Windows 7, Linux with 32 bit compat libs--where I can run smalltalk, and a pure 64 bit boot, where I cannot run smalltalk.

OK, but right now you'll want to be in Smalltalk, and for quite a while.

 

I am currently attempting  to get the source code from  http://www.squeakvm.org/squeak64/ Squeak-3.8a-2.src.tar.gz to compile on the pure 64 box.
The Quick Start steps failed, so I am now setting about studying the source code in merge64/platforms/unix/vm/sqUnixMain.c as it looks like the entry point for the VM.
In the back of my mind, I am wondering if this is even the right approach to take..

It's not :-).

 

What do you think is the best approach to take?

Read the latest two articles on my blog (A Spur gear for Cog & Lazy Become).

Make a VMMaker image (see e.g. Cog Blog :: Building a Cog Development Image).

Then read SpurMemoryManager, /especially/ its class comment.

Then read its subclasses Spur32BitMemoryManager (functional) & Spur64BitMemoryManager (completely untested).

Then read SpurBootstrap32.

Try and create a 32-bit Spur image and run it.

Then try and implement SpurBootstrap64.

Try and create a 64-bit Spur image and run it.

This is /lots/ of work.  In particular the last two steps require some significant design.

So start slowly and see how you get on.

 

Cordially,

t

---- On Thu, 07 Nov 2013 10:54:01 -0800 Eliot Miranda<[hidden email]> wrote ----

Hi Timothy,

On Thu, Nov 7, 2013 at 7:02 AM, gettimothy <[hidden email]> wrote:

Is there any chance Cog could be made to run on AMD64 natively without 32 bit libs?

Yes, that's a goal of my current work.  Spur has a common object representation between 32- and 64-bits.  Getting to a native Stack VM on AMD64 should be straight-forward.  Getting to a Cog VM on AMD64 also requires writing a code generator for AMD64.  That's actually a fun project.  It involves configuring a Bochs plugin for an AMD64/IA32 simulator, and then writing a CogIA64Compiler subclass of CogAbstractInstruction.

If so, I would like to help out with whoever takes up this task. I would need direction, however as I have been do app development for too many years and my sys skills are atrophied.

How strong are your low-level skills?  You'll need to be quite familiar with machine code, basic machine organization, etc.  If you're not then you could instead have a go at getting the Stack VM working in 64-bits, and that would involve getting the 64-bit VM simulator working in VMMaker.

It will be great to have another collaborator!

t.

---- On Thu, 07 Nov 2013 06:01:00 -0800 Bob Arning<[hidden email]> wrote ----

It's absolutely none of my business what tools others choose to use, but I will say this:

Having more of this process in Smalltalk and less in other (arcane) languages increases the confidence that, should we lose someone's services, others could step up to fill the gap.

Cheers,
Bob

On 11/7/13 8:28 AM, Igor Stasenko wrote:

[hidden email]" type="cite">

On 7 November 2013 08:58, Tobias Pape <[hidden email]> wrote:

Hey Tim and all,

On 07.11.2013, at 05:00, Ron Teitelbaum <[hidden email]> wrote:

> Hey Tim,
>
> Göran and Eliot have been discussing this.  I just talked to Eliot about it
> yesterday.  He said he was happy to support cmake but didn't want to have to
> do the work himself.  I'm hoping that the work Göran is doing on this now
> can be turned over to Eliot when it is finished.  I think there is agreement
> that combining the excellent work of pharo build with Eliot's new vm work is
> a very good thing to do.  The work to merge the build environments is
> already being done at 3D ICC.  We are working on this for the Mac also but
> have not discussed it with Ian yet.
>
> Ron Teitelbaum
>
>> -----Original Message-----
>> From: [hidden email] [mailto:[hidden email]
>> [hidden email]] On Behalf Of tim Rowledge
>> Sent: Wednesday, November 06, 2013 9:04 PM
>> To: The general-purpose Squeak developers list
>> Subject: Re: [squeak-dev] A Bounty for CMake-ifying stack/Cog vm build
> process
>>
>>
>> Wow, no interest? Is it because make is painful or because nobody needs to
>> make any more money?

I actually have interest but I am currently reluctant,
for time-capturing reasons in my personal and work life.

However, Here is what I would do:

• Start from Ians CMake work, and try to generalize it for the
  non-Linux platforms
• Windows would be kind-of straight-forward,
• For OS X, I would really want to have the Cocoa-based VM stuff
  (aka Squeak VM 5.4.7.x) integrated and proper Xcode files
  generated, especially to be able to have iOS VMs built automatically.
   Here, some information from John McIntosh would be helpfull, especially
  a simple walk-through.
   I have some experience with that[1]


Despite all admiration I have for the way the PharoVM is built, there is one
thing that bothers me:
• You first somehow generate the VM sources (which is fine, and the automated way is amazing)
• THEN, the a Script generates CMake-Files
• CMake then generates Makefiles
• and then finally a vm is compiled.
This is one generating step too much, for my taste.
Personally, I'd prefer CMake to just pick up the C-files generated by VMMaker,
so that adding a new platform does not need to change the VMMaker…

The main big issue with 'platform-neutral' static source files, that they tend to

grow with tons of #ifdef-s over time, reducing readability and creating a mess.

My point is that one or another way you have to deal with platform differences,

and the way how i prefer to do it is using smalltalk code,

but not .m4 files, or .cmake files, or writing a medium novel long configuration files

using strange (better to say weird) scripting DSL.

I can express all build process logic in smalltalk, which does not lessens the amount
of work to deal with all platform nuances and dependencies etc, but at least it allows me to organize and shape it
into some manageable form, which is easy to change & learn.

 

 And another difference is, that i prefer working with browser & classes than
with dozens of directories and files lying here and there.

But sure thing, it is a question of taste. If you prefer to maintain this:
-------------------------
MACRO (INTERNAL_PLUGIN plugin)
  SET (plugin_sources "")
  IF (DEFINED ${plugin}_sources)
    SET (plugin_sources ${${plugin}_sources})
  ELSE (DEFINED ${plugin}_sources)
    FOREACH (dir ${src}/vm/intplugins ${cross}/plugins ${unix}/plugins)
      SET (tmp "")
      AUX_SOURCE_DIRECTORY (${dir}/${plugin} tmp)
      STRING_APPEND (plugin_sources "${tmp}")
    ENDFOREACH (dir)
  ENDIF (DEFINED ${plugin}_sources)
  IF (DEFINED ${plugin}_extra_sources)
    STRING_APPEND (plugin_sources "${${plugin}_extra_sources}")
  ENDIF (DEFINED ${plugin}_extra_sources)
  FILE (WRITE ${bld}/${plugin}/CMakeLists.in "")
  FOREACH (dir ${unix}/plugins ${unix})
    FILE_APPEND (${bld}/${plugin}/CMakeLists.in ${dir}/${plugin}/build.cmake)
  ENDFOREACH (dir)
  FILE_APPEND (${bld}/${plugin}/CMakeLists.in ${config}/PluginInternal.cmake)
  CONFIGURE_FILE (${bld}/${plugin}/CMakeLists.in ${bld}/${plugin}/CMakeLists.txt @ONLY)
  ADD_SUBDIRECTORY (${bld}/${plugin} ${bld}/${plugin})
ENDMACRO (INTERNAL_PLUGIN)
--------------------------

And/or this:
--------------------------
AC_INIT([config.h.in])

SVNREV=`grep '\$Rev: ' ${srcdir}/../../../platforms/Cross/vm/sqSCCSVersion.h \
        | sed 's/.*$Rev: \([[0-9]][[0-9]]*\).*/\1/'`
AC_VM_VERSION(4,0,${SVNREV},4,2,0)

topdir=`cd ${srcdir}/../../..; pwd`
cfgdir=`cd ${srcdir}; pwd`

AC_ARG_WITH(src,
[  --with-src=dir          generated src directory [default=src]],
[ vmmsrc="${with_src}"],
[ vmmsrc="src"])

vmmdir="${topdir}/${vmmsrc}"

if test ! -d "${topdir}/${vmmsrc}"; then
  if test -d "${topdir}/platforms/unix/${vmmsrc}"; then
    vmmdir="${topdir}/platforms/unix/${vmmsrc}"
    AC_MSG_RESULT([using built-in src directory])
  fi
fi

AC_MSG_RESULT(${vmmdir})

blddir=`pwd`

AC_ARG_WITH(vmmcfg,
[  --with-vmmcfg=dir        vm configuration directory containing plugins.int and plugins.ext [default=.]],
[ vmmcfg="${with-vmmcfg}"],
[ vmmcfg="${blddir}"])

# Compiling a Cogit VM or not?  If so, need a cogit$o, cointerp, etc.

AC_ARG_ENABLE(cogit,
[  --enable-cogit            compile a cogit VM [default=yes]],
cogit="no", cogit="yes")
AC_SUBST(cogit)

#AC_ARG_ENABLE(jit,
#[  --enable-jit            enable J4 support [default=no]],
#JIT="yes", JIT="no")
#
#test $JIT = "yes" && J_CFLAGS="-DJ_ENABLED"
#AC_SUBST(J_CFLAGS)

# Check the generated src dir looks sane

AC_CHECK_VMM_DIR

AC_SUBST(topdir)
AC_SUBST(cfgdir)
AC_SUBST(vmmdir)
AC_SUBST(vmmcfg)
AC_SUBST(blddir)

SQ_VERSION=${SQ_MAJOR}.${SQ_MINOR}-${SQ_UPDATE}

AC_DEFINE_UNQUOTED(SQ_VERSION, "${SQ_VERSION}")

AC_SUBST(SQ_MAJOR)
AC_SUBST(SQ_MINOR)
AC_SUBST(SQ_UPDATE)
AC_SUBST(SQ_VERSION)

VM_VERSION=${VM_MAJOR}.${VM_MINOR}-${VM_RELEASE}

AC_DEFINE_UNQUOTED(VM_VERSION, "${VM_VERSION}")

AC_SUBST(VM_MAJOR)
AC_SUBST(VM_MINOR)
AC_SUBST(VM_RELEASE)
AC_SUBST(VM_VERSION)

# libdir contains ${exec_prefix}, so we have to default and expand early
test "x$prefix" = xNONE && prefix=$ac_default_prefix
test "x$exec_prefix" = xNONE && exec_prefix='${prefix}'
imgdir=`eval echo ${libdir}/squeak`
expanded_relative_imgdir=`eval echo lib/squeak/${VM_VERSION}`
plgdir='${imgdir}/'`eval echo ${VM_VERSION}`

AC_SUBST(imgdir)
AC_SUBST(expanded_relative_imgdir)
AC_SUBST(plgdir)

AC_DEFINE(OS_TYPE, "unix")

AC_CANONICAL_HOST

host_cpu=`echo $host | sed 's,-.*,,'`
host=`echo $host | sed 's,-unknown,,'`

AC_SUBST(host)
AC_SUBST(host_cpu)
AC_SUBST(host_vendor)
AC_SUBST(host_os)
----------------------------

instead of plain smalltalk code, it is up to you.

But that's about 'too much' for my taste.

But if the community is positive to just move to the Pharo way of
building VMs, I'd be fine with that, since that would unify a lot
of infrastructure :)

Best
        -Tobias

[1] https://github.com/krono/self/blob/cmake/vm/cmake/mac_osx.cmake

--
Best regards,
Igor Stasenko.

--
best,

Eliot

--
best,

Eliot

--
best,

Eliot

On Fri, Nov 08, 2013 at 07:29:42AM -0800, gettimothy wrote:
> Eliot
>
> First, if these discussion is inappropriate for squeak-dev, I have subscribed to squeak-vm-beginners and we can move it there or elsewhere if I am creating too much noise here.
>

Hi Timothy,

The vm-dev list is here:

http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/vm-dev

It is not a beginner's list, and your discussion would be welcome there. Either
way, you are not creating "noise" and it is great to see you getting involved.


> In working the steps at http://www.mirandabanda.org/cogblog/build-image/ and I have found that running all the tests for the Alien install on the SqueakVM that comes with the all-in-one app fails all but one of its tests. (primitives fail)
>
> If I start the image using the CogVM then all the tests pass.
>
> So, is it correct to say that building a Cog Development Image on Squeak4.4 or greater depends on a pre-built CogVM from http://www.mirandabanda.org/files/Cog/VM/ ?
>
> I am assuming yes, but I want to verify with you before I document what I am doing
>

It is a good idea to use a VM from mirandaband.org, because different VMs may have
different runtime behaviour, and using a runtime environment that matches Eliot's
will prevent confusion.

Strictly speaking, you can build a Cog VM using pretty much any VM that works
with your Squeak4.4 image. So no it is not necessary to use a specific pre-built
image, but it is a very good idea to do so, especially during the time when you
are first learning how to do VM building.

I think Eliot will give you a similar answer, so hopefully he won't mind me stepping
in to answer some of your questions.

Dave

> thx.
>
> t
>

center
  ^ origin + corner / 2

Rectangle center
  0       push the value of the receiver's first instance variable (origin) onto the
          stack
   1       push the value of the receiver's second instance variable (corner) onto the
          stack
  176     send a binary message with the selector +
  119     push the Smalllnteger 2 onto the stack
  185     send a binary message with the selector /
  124     return the object on top of the stack as the value of the message (center)

Hi Timothy,

On Tue, Nov 19, 2013 at 8:33 AM, gettimothy <[hidden email]> wrote:

Part 4 of the Blue Book--chapter 26 The implementation--in the intro there is this example:

center
  ^ origin + corner / 2

With these operations

Rectangle center
  0       push the value of the receiver's first instance variable (origin) onto the
          stack
   1       push the value of the receiver's second instance variable (corner) onto the
          stack
  176     send a binary message with the selector +
  119     push the Smalllnteger 2 onto the stack
  185     send a binary message with the selector /
  124     return the object on top of the stack as the value of the message (center)

In a Workspace, when I doit on 3 + 7/2 I get 5, left, right, unary, binary....rules apply.

What I see here--and please check my reasoning--is a RPN that places things onto the stack via Smalltalk's order of operation rules and then does conventional RPN processing.

In other words, Smalltalk expressions are translated to RPN and then placed on the stack.

Yes, and so they are in most stack-based implementations of languages.  Bytecode for a stack machine is a form of RPN.

A simple 'banish the thought' or 'yep' or 'sometimes'  would be helpful as I move forward in my study.  I am just trying to avoid carrying forward an improper idea that I will have to revise later when/if the facts turn out otherwise.

thx.

t

--
best,

Eliot