Re: Creating the smallest server runtime footprint

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
39 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Pavel Krivanek-3


2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel
Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Tim Mackinnon
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec repository: '<a href="github://newapplesho/aws-sdk-smalltalk:v1.10/pharo-repository" class="">github://newapplesho/aws-sdk-smalltalk:v1.10/pharo-repository' ].
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: '<a href="filetree://./pharo-core/src" class="">filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel

Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Pavel Krivanek-3

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel


 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel


Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Guillermo Polito
Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')
    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel


 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel





--

   

Guille Polito


Research Engineer

French National Center for Scientific Research - http://www.cnrs.fr



Web: http://guillep.github.io

Phone: +33 06 52 70 66 13

Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Tim Mackinnon
Ah, I think I’m starting to get closer to what I need…

So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?

Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?

I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.

Tim

On 2 Aug 2017, at 07:05, Guillermo Polito <[hidden email]> wrote:

Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')
    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel


 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel





--
   
Guille Polito

Research Engineer
French National Center for Scientific Research - http://www.cnrs.fr


Phone: +33 06 52 70 66 13

Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Tim Mackinnon
In reply to this post by Tim Mackinnon
As an aside, I’ve just noticed that I get the same error in a normal 6.1 image when you try to load a baselineof package in Iceberg using the install menu option. So I wonder if this might actually be a bug in the image that the minimal version is getting as well?

Tim

On 1 Aug 2017, at 22:13, Tim Mackinnon <[hidden email]> wrote:

Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec repository: '<a href="github://newapplesho/aws-sdk-smalltalk:v1.10/pharo-repository" class="">github://newapplesho/aws-sdk-smalltalk:v1.10/pharo-repository' ].
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: '<a href="filetree://./pharo-core/src" class="">filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel


Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Pavel Krivanek-3
In reply to this post by Tim Mackinnon
The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):

Metacello-GitBasedRepository.package
Metacello-GitHub.package
SUnit-Core.package

and create baseline to load them. I already tried it so you can test it:

Then you will do something like:



2017-08-02 15:18 GMT+02:00 Tim Mackinnon <[hidden email]>:
Ah, I think I’m starting to get closer to what I need…

So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?

Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?

I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.

Tim

On 2 Aug 2017, at 07:05, Guillermo Polito <[hidden email]> wrote:

Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')
    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel


 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel





--
   
Guille Polito

Research Engineer
French National Center for Scientific Research - http://www.cnrs.fr


Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13


Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Pavel Krivanek-3


2017-08-03 10:14 GMT+02:00 Pavel Krivanek <[hidden email]>:
The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):

Metacello-GitBasedRepository.package
Metacello-GitHub.package
SUnit-Core.package

and create baseline to load them. I already tried it so you can test it:

Then you will do something like:

(accidental message send ;-) )

BASELINE=MetacelloGitBasedRepository
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

BASELINE=Lambda
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

As I tried that. The baseline of Lambda is then able to be loaded.

-- Pavel





2017-08-02 15:18 GMT+02:00 Tim Mackinnon <[hidden email]>:
Ah, I think I’m starting to get closer to what I need…

So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?

Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?

I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.

Tim

On 2 Aug 2017, at 07:05, Guillermo Polito <[hidden email]> wrote:

Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')
    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel


 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel





--
   
Guille Polito

Research Engineer
French National Center for Scientific Research - http://www.cnrs.fr


Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13



Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Tim Mackinnon
I really appreciate your patience and help on this - it looks very promising and I am giving it a spin now…

When you say the pharo repository - are you referring to this: https://github.com/pharo-project/pharo/tree/master/src. ? Just so I know for the future.


Tim

On 3 Aug 2017, at 09:16, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 10:14 GMT+02:00 Pavel Krivanek <[hidden email]>:
The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):

Metacello-GitBasedRepository.package
Metacello-GitHub.package
SUnit-Core.package

and create baseline to load them. I already tried it so you can test it:

Then you will do something like:

(accidental message send ;-) )

BASELINE=MetacelloGitBasedRepository
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: '<a href="filetree://./PharoLambda/src" class="">filetree://./PharoLambda/src'; load."

BASELINE=Lambda
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: '<a href="filetree://./PharoLambda/src" class="">filetree://./PharoLambda/src'; load."

As I tried that. The baseline of Lambda is then able to be loaded.

-- Pavel





2017-08-02 15:18 GMT+02:00 Tim Mackinnon <[hidden email]>:
Ah, I think I’m starting to get closer to what I need…

So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?

Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?

I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.

Tim

On 2 Aug 2017, at 07:05, Guillermo Polito <[hidden email]> wrote:

Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')
    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel


 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel





--
   
Guille Polito

Research Engineer
French National Center for Scientific Research - http://www.cnrs.fr


Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank" class="">+33 06 52 70 66 13




Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Pavel Krivanek-3


2017-08-03 13:02 GMT+02:00 Tim Mackinnon <[hidden email]>:
I really appreciate your patience and help on this - it looks very promising and I am giving it a spin now…

When you say the pharo repository - are you referring to this: https://github.com/pharo-project/pharo/tree/master/src. ? Just so I know for the future.

yes

-- Pavel 


Tim

On 3 Aug 2017, at 09:16, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 10:14 GMT+02:00 Pavel Krivanek <[hidden email]>:
The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):

Metacello-GitBasedRepository.package
Metacello-GitHub.package
SUnit-Core.package

and create baseline to load them. I already tried it so you can test it:

Then you will do something like:

(accidental message send ;-) )

BASELINE=MetacelloGitBasedRepository
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

BASELINE=Lambda
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

As I tried that. The baseline of Lambda is then able to be loaded.

-- Pavel





2017-08-02 15:18 GMT+02:00 Tim Mackinnon <[hidden email]>:
Ah, I think I’m starting to get closer to what I need…

So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?

Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?

I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.

Tim

On 2 Aug 2017, at 07:05, Guillermo Polito <[hidden email]> wrote:

Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')
    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel


 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel





--
   
Guille Polito

Research Engineer
French National Center for Scientific Research - http://www.cnrs.fr


Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13





Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Tim Mackinnon
I’m so close now (bit of a diversion with gitlab yml not liking the eval string) - that did load and it got me really close - I’m just missing FlSerializer now (I have progressed my work to fuel out the context for debugging to S3)… so looking at the repo, there are quite a few fuel packages - but in a big image I can see that FLSerialiser is in Core - so do I just need core? But then there are a few fuel packages with Core?

I’m also not quite clear on how I would load packages without a Baseline (would I create my own baseline for the fuel packages I need?)

Tim

On 3 Aug 2017, at 12:07, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 13:02 GMT+02:00 Tim Mackinnon <[hidden email]>:
I really appreciate your patience and help on this - it looks very promising and I am giving it a spin now…

When you say the pharo repository - are you referring to this: https://github.com/pharo-project/pharo/tree/master/src. ? Just so I know for the future.

yes

-- Pavel 


Tim

On 3 Aug 2017, at 09:16, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 10:14 GMT+02:00 Pavel Krivanek <[hidden email]>:
The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):

Metacello-GitBasedRepository.package
Metacello-GitHub.package
SUnit-Core.package

and create baseline to load them. I already tried it so you can test it:

Then you will do something like:

(accidental message send ;-) )

BASELINE=MetacelloGitBasedRepository
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

BASELINE=Lambda
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

As I tried that. The baseline of Lambda is then able to be loaded.

-- Pavel





2017-08-02 15:18 GMT+02:00 Tim Mackinnon <[hidden email]>:
Ah, I think I’m starting to get closer to what I need…

So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?

Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?

I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.

Tim

On 2 Aug 2017, at 07:05, Guillermo Polito <[hidden email]> wrote:

Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')
    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel


 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel





--
   
Guille Polito

Research Engineer
French National Center for Scientific Research - http://www.cnrs.fr


Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank" class="">+33 06 52 70 66 13






Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Pavel Krivanek-3


2017-08-03 13:58 GMT+02:00 Tim Mackinnon <[hidden email]>:
I’m so close now (bit of a diversion with gitlab yml not liking the eval string) - that did load and it got me really close - I’m just missing FlSerializer now (I have progressed my work to fuel out the context for debugging to S3)… so looking at the repo, there are quite a few fuel packages - but in a big image I can see that FLSerialiser is in Core - so do I just need core? But then there are a few fuel packages with Core?

you probably need:
spec package: 'Fuel'.
spec package: 'FuelPlatform-Core'.
spec package: 'FuelPlatform-Pharo-Core'.
spec package: 'FuelPlatform-Pharo-06'.

It has some dependencies on other packages that you do not load so it should cause some warnings during loading


I’m also not quite clear on how I would load packages without a Baseline (would I create my own baseline for the fuel packages I need?)

If you will copy them to your repository then add them to your baseline. For loading of packages without baseline you may use Gofer.

- Pavel
 


Tim

On 3 Aug 2017, at 12:07, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 13:02 GMT+02:00 Tim Mackinnon <[hidden email]>:
I really appreciate your patience and help on this - it looks very promising and I am giving it a spin now…

When you say the pharo repository - are you referring to this: https://github.com/pharo-project/pharo/tree/master/src. ? Just so I know for the future.

yes

-- Pavel 


Tim

On 3 Aug 2017, at 09:16, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 10:14 GMT+02:00 Pavel Krivanek <[hidden email]>:
The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):

Metacello-GitBasedRepository.package
Metacello-GitHub.package
SUnit-Core.package

and create baseline to load them. I already tried it so you can test it:

Then you will do something like:

(accidental message send ;-) )

BASELINE=MetacelloGitBasedRepository
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

BASELINE=Lambda
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

As I tried that. The baseline of Lambda is then able to be loaded.

-- Pavel





2017-08-02 15:18 GMT+02:00 Tim Mackinnon <[hidden email]>:
Ah, I think I’m starting to get closer to what I need…

So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?

Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?

I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.

Tim

On 2 Aug 2017, at 07:05, Guillermo Polito <[hidden email]> wrote:

Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')
    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel


 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel





--
   
Guille Polito

Research Engineer
French National Center for Scientific Research - http://www.cnrs.fr


Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13







Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Mariano Martinez Peck
In reply to this post by Tim Mackinnon


On Thu, Aug 3, 2017 at 8:58 AM, Tim Mackinnon <[hidden email]> wrote:
I’m so close now (bit of a diversion with gitlab yml not liking the eval string) - that did load and it got me really close - I’m just missing FlSerializer now (I have progressed my work to fuel out the context for debugging to S3)… so looking at the repo, there are quite a few fuel packages - but in a big image I can see that FLSerialiser is in Core - so do I just need core? But then there are a few fuel packages with Core?


The easiest way to understand that is either checking ConfigurationOfFuel directly or some of the related Metacello tool.
Anyway, for this case, I think you need the packages (in this order): FuelPlatform and Fuel. Bug again, Metacello allows some postLoad actions (which we use in Fuel)... so...it's not always easy to load a project without Metacello. 

Sometimes you can print the result of #record:

Metacello new 
configuration: 'Fuel';
smalltalkhubUser: 'Pharo' project: 'Fuel';
version: #stable;
record: #('Core').

->

"linear load : 
linear load : 2.1.10 [ConfigurationOfFuel]
linear load : baseline [ConfigurationOfFuelPlatform]
load : FuelPlatform-Core
load : FuelPlatform-Pharo-Core
load : FuelPlatform-Pharo-06"


------

Anyway.... I guess I would try to load these packages (in this order):

FuelPlatform-Core
FuelPlatform-Pharo-Core
FuelPlatform-Pharo-06
Fuel

And when you are down loading, then execute:

(Smalltalk at: #FLPlatform) current addHacks

 
I’m also not quite clear on how I would load packages without a Baseline (would I create my own baseline for the fuel packages I need?)

Tim

On 3 Aug 2017, at 12:07, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 13:02 GMT+02:00 Tim Mackinnon <[hidden email]>:
I really appreciate your patience and help on this - it looks very promising and I am giving it a spin now…

When you say the pharo repository - are you referring to this: https://github.com/pharo-project/pharo/tree/master/src. ? Just so I know for the future.

yes

-- Pavel 


Tim

On 3 Aug 2017, at 09:16, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 10:14 GMT+02:00 Pavel Krivanek <[hidden email]>:
The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):

Metacello-GitBasedRepository.package
Metacello-GitHub.package
SUnit-Core.package

and create baseline to load them. I already tried it so you can test it:

Then you will do something like:

(accidental message send ;-) )

BASELINE=MetacelloGitBasedRepository
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

BASELINE=Lambda
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

As I tried that. The baseline of Lambda is then able to be loaded.

-- Pavel





2017-08-02 15:18 GMT+02:00 Tim Mackinnon <[hidden email]>:
Ah, I think I’m starting to get closer to what I need…

So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?

Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?

I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.

Tim

On 2 Aug 2017, at 07:05, Guillermo Polito <[hidden email]> wrote:

Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')
    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel


 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel





--
   
Guille Polito

Research Engineer
French National Center for Scientific Research - http://www.cnrs.fr


Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13









--
Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Tim Mackinnon
Hi guys - I think I’m getting close - I’m able to load remote packages, and I’ve added fuel to my repo and its loading - however I am getting an error when I run and test writing out a context - MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames

I can see that this is a trait - is trait support missing in the minimal image?

(I also seemed to get this far just loading the Fuel package - but just to be sure I added the other platform ones like you suggested and still get the same error). Given how simple that method is, I could just compile it in - but maybe there are many more?

Tim


"errorMessage": "Command failed: ./pharo Pharo.image --no-default-preferences process Lambda '{\"debug\":true,\"data\":5}'\n
  "errorType": "Error",
  "stackTrace": [
    "Writing fuel context to S3...",
    "Using S3 bucket morethan-technology-projects",
    "\u001b[31m",
    "--- RUNNING ERROR HANDLER ---",
    "MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
    "",
    "\u001b[0m\u001b[31mMessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
    "\u001b[0mContext class(Object)>>doesNotUnderstand: #fuelIgnoredInstanceVariableNames",
    "FLVariablesMapping>>instanceVariableNamesToSerialize",
    "FLVariablesMapping>>initializeAnalyzing",
    "FLVariablesMapping class>>newAnalyzing:references:",
    "FLContextCluster(FLPointerObjectCluster)>>initializeAnalyzing:",
    "FLContextCluster class(FLObjectCluster class)>>newAnalyzing:",
    "clusterKeyedByObjectClass: t1 class: t2",
    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class: in Block: clusterKeyedByObjectClass: t1 class: t2...",
    "clusterInstanceOf: t1 keyInBucket: t2 factory: t3",
    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory: in Block: clusterInstanceOf: t1 keyInBucket: t2 factory: t3...",
    "at: t1 ifAbsentPut: t2",
    "  self error: 'No decompiler available' in IdentityDictionary(Dictionary)>>at:ifAbsentPut: in Block: at: t1 ifAbsentPut: t2...",
    "IdentityDictionary(Dictionary)>>at:ifAbsent:",
    "IdentityDictionary(Dictionary)>>at:ifAbsentPut:",
    "FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory:",
    "FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class:",
    "FLLightGeneralMapper(FLMapper)>>mapAndTraceByObjectClass:to:",
    "FLLightGeneralMapper>>visitMethodContext:",
    "Context>>fuelAccept:",
    "FLLightGeneralMapper>>mapAndTrace:",
    "FLLightGlobalMapper>>mapAndTrace:",
    "FLPluggableSubstitutionMapper>>mapAndTrace:",
    "FLAnalysis>>mapAndTrace:",
    "FLAnalysis>>run",
    "setDefaultAnalysis",
    "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",
    "FLAnalyzer>>analysisFor:",
    "FLSerialization>>analysisStep",
    "FLSerialization>>run",
    "setDefaultSerialization",
    "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",
    "serialize: t1 on: t2",
    "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",
    "on: t1 globalEnvironment: t2 do: t3",
    "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",
    "BlockClosure>>ensure:",
    "FLEncoder class>>on:globalEnvironment:do:",
    "\u001b[0m"
On 3 Aug 2017, at 13:16, Mariano Martinez Peck <[hidden email]> wrote:



On Thu, Aug 3, 2017 at 8:58 AM, Tim Mackinnon <[hidden email]> wrote:
I’m so close now (bit of a diversion with gitlab yml not liking the eval string) - that did load and it got me really close - I’m just missing FlSerializer now (I have progressed my work to fuel out the context for debugging to S3)… so looking at the repo, there are quite a few fuel packages - but in a big image I can see that FLSerialiser is in Core - so do I just need core? But then there are a few fuel packages with Core?


The easiest way to understand that is either checking ConfigurationOfFuel directly or some of the related Metacello tool.
Anyway, for this case, I think you need the packages (in this order): FuelPlatform and Fuel. Bug again, Metacello allows some postLoad actions (which we use in Fuel)... so...it's not always easy to load a project without Metacello. 

Sometimes you can print the result of #record:

Metacello new 
configuration: 'Fuel';
smalltalkhubUser: 'Pharo' project: 'Fuel';
version: #stable;
record: #('Core').

->

"linear load : 
linear load : 2.1.10 [ConfigurationOfFuel]
linear load : baseline [ConfigurationOfFuelPlatform]
load : FuelPlatform-Core
load : FuelPlatform-Pharo-Core
load : FuelPlatform-Pharo-06"


------

Anyway.... I guess I would try to load these packages (in this order):

FuelPlatform-Core
FuelPlatform-Pharo-Core
FuelPlatform-Pharo-06
Fuel

And when you are down loading, then execute:

(Smalltalk at: #FLPlatform) current addHacks

 
I’m also not quite clear on how I would load packages without a Baseline (would I create my own baseline for the fuel packages I need?)

Tim

On 3 Aug 2017, at 12:07, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 13:02 GMT+02:00 Tim Mackinnon <[hidden email]>:
I really appreciate your patience and help on this - it looks very promising and I am giving it a spin now…

When you say the pharo repository - are you referring to this: https://github.com/pharo-project/pharo/tree/master/src. ? Just so I know for the future.

yes

-- Pavel 


Tim

On 3 Aug 2017, at 09:16, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 10:14 GMT+02:00 Pavel Krivanek <[hidden email]>:
The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):

Metacello-GitBasedRepository.package
Metacello-GitHub.package
SUnit-Core.package

and create baseline to load them. I already tried it so you can test it:

Then you will do something like:

(accidental message send ;-) )

BASELINE=MetacelloGitBasedRepository
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

BASELINE=Lambda
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

As I tried that. The baseline of Lambda is then able to be loaded.

-- Pavel





2017-08-02 15:18 GMT+02:00 Tim Mackinnon <[hidden email]>:
Ah, I think I’m starting to get closer to what I need…

So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?

Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?

I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.

Tim

On 2 Aug 2017, at 07:05, Guillermo Polito <[hidden email]> wrote:

Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')
    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel


 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel





-- 
   
Guille Polito

Research Engineer
French National Center for Scientific Research - http://www.cnrs.fr


Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank" class="">+33 06 52 70 66 13









-- 

Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Mariano Martinez Peck


On Thu, Aug 3, 2017 at 1:17 PM, Tim Mackinnon <[hidden email]> wrote:
Hi guys - I think I’m getting close - I’m able to load remote packages, and I’ve added fuel to my repo and its loading - however I am getting an error when I run and test writing out a context - MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames

I can see that this is a trait - is trait support missing in the minimal image?


Uff...that's a good question hahahhah 
I don't know if the minimal image supports traits, but what I am almost sure, is the Behavior DOES need the methods defined in TBehavior. So maybe the minimal image flattens the traits. In this case you could do the same for Fuel (flattenDownAllTraits)
 
(I also seemed to get this far just loading the Fuel package - but just to be sure I added the other platform ones like you suggested and still get the same error). Given how simple that method is, I could just compile it in - but maybe there are many more?



TBehavior, TClass and TClassDescription are the only one that comes to my mind right now...  there are only a few methods from Fuel. So could compile those directly at least on a first step...at least you keep you going. 



 
Tim


"errorMessage": "Command failed: ./pharo Pharo.image --no-default-preferences process Lambda '{\"debug\":true,\"data\":5}'\n
  "errorType": "Error",
  "stackTrace": [
    "Writing fuel context to S3...",
    "Using S3 bucket morethan-technology-projects",
    "\u001b[31m",
    "--- RUNNING ERROR HANDLER ---",
    "MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
    "",
    "\u001b[0m\u001b[31mMessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
    "\u001b[0mContext class(Object)>>doesNotUnderstand: #fuelIgnoredInstanceVariableNames",
    "FLVariablesMapping>>instanceVariableNamesToSerialize",
    "FLVariablesMapping>>initializeAnalyzing",
    "FLVariablesMapping class>>newAnalyzing:references:",
    "FLContextCluster(FLPointerObjectCluster)>>initializeAnalyzing:",
    "FLContextCluster class(FLObjectCluster class)>>newAnalyzing:",
    "clusterKeyedByObjectClass: t1 class: t2",
    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class: in Block: clusterKeyedByObjectClass: t1 class: t2...",
    "clusterInstanceOf: t1 keyInBucket: t2 factory: t3",
    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory: in Block: clusterInstanceOf: t1 keyInBucket: t2 factory: t3...",
    "at: t1 ifAbsentPut: t2",
    "  self error: 'No decompiler available' in IdentityDictionary(Dictionary)>>at:ifAbsentPut: in Block: at: t1 ifAbsentPut: t2...",
    "IdentityDictionary(Dictionary)>>at:ifAbsent:",
    "IdentityDictionary(Dictionary)>>at:ifAbsentPut:",
    "FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory:",
    "FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class:",
    "FLLightGeneralMapper(FLMapper)>>mapAndTraceByObjectClass:to:",
    "FLLightGeneralMapper>>visitMethodContext:",
    "Context>>fuelAccept:",
    "FLLightGeneralMapper>>mapAndTrace:",
    "FLLightGlobalMapper>>mapAndTrace:",
    "FLPluggableSubstitutionMapper>>mapAndTrace:",
    "FLAnalysis>>mapAndTrace:",
    "FLAnalysis>>run",
    "setDefaultAnalysis",
    "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",
    "FLAnalyzer>>analysisFor:",
    "FLSerialization>>analysisStep",
    "FLSerialization>>run",
    "setDefaultSerialization",
    "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",
    "serialize: t1 on: t2",
    "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",
    "on: t1 globalEnvironment: t2 do: t3",
    "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",
    "BlockClosure>>ensure:",
    "FLEncoder class>>on:globalEnvironment:do:",
    "\u001b[0m"
On 3 Aug 2017, at 13:16, Mariano Martinez Peck <[hidden email]> wrote:



On Thu, Aug 3, 2017 at 8:58 AM, Tim Mackinnon <[hidden email]> wrote:
I’m so close now (bit of a diversion with gitlab yml not liking the eval string) - that did load and it got me really close - I’m just missing FlSerializer now (I have progressed my work to fuel out the context for debugging to S3)… so looking at the repo, there are quite a few fuel packages - but in a big image I can see that FLSerialiser is in Core - so do I just need core? But then there are a few fuel packages with Core?


The easiest way to understand that is either checking ConfigurationOfFuel directly or some of the related Metacello tool.
Anyway, for this case, I think you need the packages (in this order): FuelPlatform and Fuel. Bug again, Metacello allows some postLoad actions (which we use in Fuel)... so...it's not always easy to load a project without Metacello. 

Sometimes you can print the result of #record:

Metacello new 
configuration: 'Fuel';
smalltalkhubUser: 'Pharo' project: 'Fuel';
version: #stable;
record: #('Core').

->

"linear load : 
linear load : 2.1.10 [ConfigurationOfFuel]
linear load : baseline [ConfigurationOfFuelPlatform]
load : FuelPlatform-Core
load : FuelPlatform-Pharo-Core
load : FuelPlatform-Pharo-06"


------

Anyway.... I guess I would try to load these packages (in this order):

FuelPlatform-Core
FuelPlatform-Pharo-Core
FuelPlatform-Pharo-06
Fuel

And when you are down loading, then execute:

(Smalltalk at: #FLPlatform) current addHacks

 
I’m also not quite clear on how I would load packages without a Baseline (would I create my own baseline for the fuel packages I need?)

Tim

On 3 Aug 2017, at 12:07, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 13:02 GMT+02:00 Tim Mackinnon <[hidden email]>:
I really appreciate your patience and help on this - it looks very promising and I am giving it a spin now…

When you say the pharo repository - are you referring to this: https://github.com/pharo-project/pharo/tree/master/src. ? Just so I know for the future.

yes

-- Pavel 


Tim

On 3 Aug 2017, at 09:16, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 10:14 GMT+02:00 Pavel Krivanek <[hidden email]>:
The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):

Metacello-GitBasedRepository.package
Metacello-GitHub.package
SUnit-Core.package

and create baseline to load them. I already tried it so you can test it:

Then you will do something like:

(accidental message send ;-) )

BASELINE=MetacelloGitBasedRepository
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

BASELINE=Lambda
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

As I tried that. The baseline of Lambda is then able to be loaded.

-- Pavel





2017-08-02 15:18 GMT+02:00 Tim Mackinnon <[hidden email]>:
Ah, I think I’m starting to get closer to what I need…

So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?

Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?

I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.

Tim

On 2 Aug 2017, at 07:05, Guillermo Polito <[hidden email]> wrote:

Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')
    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel


 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel





-- 
   
Guille Polito

Research Engineer
French National Center for Scientific Research - http://www.cnrs.fr


Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13









-- 




--
Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Tim Mackinnon
Hi Mariano - I compiled in the trait methods manually and it got further - but now I’m seeing another error - Error: Explicitly required method",
    "\u001b[0mLambda class(Object)>>error:",
    "explicitRequirement”,

Any ideas?

Tim

"--- RUNNING ERROR HANDLER ---",
    "Error: Explicitly required method",
    "",
    "\u001b[0m\u001b[31mError: Explicitly required method",
    "\u001b[0mLambda class(Object)>>error:",
    "explicitRequirement",
    "  self error: 'No decompiler available' in Lambda class(Object)>>explicitRequirement in Block: explicitRequirement...",
    "False>>ifFalse:",
    "Lambda class(Object)>>explicitRequirement",
    "Lambda class(Class)>>fuelAccept:",
    "FLLightGeneralMapper>>mapAndTrace:",
    "FLLightGlobalMapper>>mapAndTrace:",
    "FLPluggableSubstitutionMapper>>mapAndTrace:",
    "FLAnalysis>>mapAndTrace:",
    "FLAnalysis>>run",
    "setDefaultAnalysis",
    "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",
    "FLAnalyzer>>analysisFor:",
    "FLSerialization>>analysisStep",
    "FLSerialization>>run",
    "setDefaultSerialization",
    "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",
    "serialize: t1 on: t2",
    "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",
    "on: t1 globalEnvironment: t2 do: t3",
    "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",
    "BlockClosure>>ensure:",
    "FLEncoder class>>on:globalEnvironment:do:",
    "FLSerializer>>serialize:on:",
    "NonInteractiveUIManager>>writeFuelContext:",
    "Lambda class>>processDebug:",
    "Lambda class>>processJSON:",
    "Lambda class>>process:",
    "activate",

On 3 Aug 2017, at 17:26, Mariano Martinez Peck <[hidden email]> wrote:



On Thu, Aug 3, 2017 at 1:17 PM, Tim Mackinnon <[hidden email]> wrote:
Hi guys - I think I’m getting close - I’m able to load remote packages, and I’ve added fuel to my repo and its loading - however I am getting an error when I run and test writing out a context - MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames

I can see that this is a trait - is trait support missing in the minimal image?


Uff...that's a good question hahahhah 
I don't know if the minimal image supports traits, but what I am almost sure, is the Behavior DOES need the methods defined in TBehavior. So maybe the minimal image flattens the traits. In this case you could do the same for Fuel (flattenDownAllTraits)
 
(I also seemed to get this far just loading the Fuel package - but just to be sure I added the other platform ones like you suggested and still get the same error). Given how simple that method is, I could just compile it in - but maybe there are many more?



TBehavior, TClass and TClassDescription are the only one that comes to my mind right now...  there are only a few methods from Fuel. So could compile those directly at least on a first step...at least you keep you going. 



 
Tim


"errorMessage": "Command failed: ./pharo Pharo.image --no-default-preferences process Lambda '{\"debug\":true,\"data\":5}'\n
  "errorType": "Error",
  "stackTrace": [
    "Writing fuel context to S3...",
    "Using S3 bucket morethan-technology-projects",
    "\u001b[31m",
    "--- RUNNING ERROR HANDLER ---",
    "MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
    "",
    "\u001b[0m\u001b[31mMessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
    "\u001b[0mContext class(Object)>>doesNotUnderstand: #fuelIgnoredInstanceVariableNames",
    "FLVariablesMapping>>instanceVariableNamesToSerialize",
    "FLVariablesMapping>>initializeAnalyzing",
    "FLVariablesMapping class>>newAnalyzing:references:",
    "FLContextCluster(FLPointerObjectCluster)>>initializeAnalyzing:",
    "FLContextCluster class(FLObjectCluster class)>>newAnalyzing:",
    "clusterKeyedByObjectClass: t1 class: t2",
    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class: in Block: clusterKeyedByObjectClass: t1 class: t2...",
    "clusterInstanceOf: t1 keyInBucket: t2 factory: t3",
    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory: in Block: clusterInstanceOf: t1 keyInBucket: t2 factory: t3...",
    "at: t1 ifAbsentPut: t2",
    "  self error: 'No decompiler available' in IdentityDictionary(Dictionary)>>at:ifAbsentPut: in Block: at: t1 ifAbsentPut: t2...",
    "IdentityDictionary(Dictionary)>>at:ifAbsent:",
    "IdentityDictionary(Dictionary)>>at:ifAbsentPut:",
    "FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory:",
    "FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class:",
    "FLLightGeneralMapper(FLMapper)>>mapAndTraceByObjectClass:to:",
    "FLLightGeneralMapper>>visitMethodContext:",
    "Context>>fuelAccept:",
    "FLLightGeneralMapper>>mapAndTrace:",
    "FLLightGlobalMapper>>mapAndTrace:",
    "FLPluggableSubstitutionMapper>>mapAndTrace:",
    "FLAnalysis>>mapAndTrace:",
    "FLAnalysis>>run",
    "setDefaultAnalysis",
    "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",
    "FLAnalyzer>>analysisFor:",
    "FLSerialization>>analysisStep",
    "FLSerialization>>run",
    "setDefaultSerialization",
    "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",
    "serialize: t1 on: t2",
    "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",
    "on: t1 globalEnvironment: t2 do: t3",
    "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",
    "BlockClosure>>ensure:",
    "FLEncoder class>>on:globalEnvironment:do:",
    "\u001b[0m"
On 3 Aug 2017, at 13:16, Mariano Martinez Peck <[hidden email]> wrote:



On Thu, Aug 3, 2017 at 8:58 AM, Tim Mackinnon <[hidden email]> wrote:
I’m so close now (bit of a diversion with gitlab yml not liking the eval string) - that did load and it got me really close - I’m just missing FlSerializer now (I have progressed my work to fuel out the context for debugging to S3)… so looking at the repo, there are quite a few fuel packages - but in a big image I can see that FLSerialiser is in Core - so do I just need core? But then there are a few fuel packages with Core?


The easiest way to understand that is either checking ConfigurationOfFuel directly or some of the related Metacello tool.
Anyway, for this case, I think you need the packages (in this order): FuelPlatform and Fuel. Bug again, Metacello allows some postLoad actions (which we use in Fuel)... so...it's not always easy to load a project without Metacello. 

Sometimes you can print the result of #record:

Metacello new 
configuration: 'Fuel';
smalltalkhubUser: 'Pharo' project: 'Fuel';
version: #stable;
record: #('Core').

->

"linear load : 
linear load : 2.1.10 [ConfigurationOfFuel]
linear load : baseline [ConfigurationOfFuelPlatform]
load : FuelPlatform-Core
load : FuelPlatform-Pharo-Core
load : FuelPlatform-Pharo-06"


------

Anyway.... I guess I would try to load these packages (in this order):

FuelPlatform-Core
FuelPlatform-Pharo-Core
FuelPlatform-Pharo-06
Fuel

And when you are down loading, then execute:

(Smalltalk at: #FLPlatform) current addHacks

 
I’m also not quite clear on how I would load packages without a Baseline (would I create my own baseline for the fuel packages I need?)

Tim

On 3 Aug 2017, at 12:07, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 13:02 GMT+02:00 Tim Mackinnon <[hidden email]>:
I really appreciate your patience and help on this - it looks very promising and I am giving it a spin now…

When you say the pharo repository - are you referring to this: https://github.com/pharo-project/pharo/tree/master/src. ? Just so I know for the future.

yes

-- Pavel 


Tim

On 3 Aug 2017, at 09:16, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 10:14 GMT+02:00 Pavel Krivanek <[hidden email]>:
The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):

Metacello-GitBasedRepository.package
Metacello-GitHub.package
SUnit-Core.package

and create baseline to load them. I already tried it so you can test it:

Then you will do something like:

(accidental message send ;-) )

BASELINE=MetacelloGitBasedRepository
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

BASELINE=Lambda
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

As I tried that. The baseline of Lambda is then able to be loaded.

-- Pavel





2017-08-02 15:18 GMT+02:00 Tim Mackinnon <[hidden email]>:
Ah, I think I’m starting to get closer to what I need…

So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?

Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?

I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.

Tim

On 2 Aug 2017, at 07:05, Guillermo Polito <[hidden email]> wrote:

Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')
    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel


 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel





-- 
   
Guille Polito

Research Engineer
French National Center for Scientific Research - http://www.cnrs.fr


Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank" class="">+33 06 52 70 66 13









-- 




-- 

Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Mariano Martinez Peck

On Thu, Aug 3, 2017 at 2:14 PM, Tim Mackinnon <[hidden email]> wrote:
Hi Mariano - I compiled in the trait methods manually and it got further - but now I’m seeing another error - Error: Explicitly required method",
    "\u001b[0mLambda class(Object)>>error:",
    "explicitRequirement”,


mmmm the stack is weird... it looks like if Class >> fuelAccept: would have the wrong implementation, that is, having the one from TClass:

fuelAccept: aGeneralMapper
^self explicitRequirement.

Rather than the correct one:

fuelAccept: aGeneralMapper

^aGeneralMapper visitClass: self


Maybe the solution is to compile all those methods from traits BUT NOT those that are "^self explicitRequirement." because those will override the good methods.

If that doesn't work, then maybe can you print to console the each impl (source) of each implementor of #fuelAccept: ?

Thoughts?



 
Any ideas?

Tim

"--- RUNNING ERROR HANDLER ---",
    "Error: Explicitly required method",
    "",
    "\u001b[0m\u001b[31mError: Explicitly required method",
    "\u001b[0mLambda class(Object)>>error:",
    "explicitRequirement",
    "  self error: 'No decompiler available' in Lambda class(Object)>>explicitRequirement in Block: explicitRequirement...",
    "False>>ifFalse:",
    "Lambda class(Object)>>explicitRequirement",
    "Lambda class(Class)>>fuelAccept:",
    "FLLightGeneralMapper>>mapAndTrace:",
    "FLLightGlobalMapper>>mapAndTrace:",
    "FLPluggableSubstitutionMapper>>mapAndTrace:",
    "FLAnalysis>>mapAndTrace:",
    "FLAnalysis>>run",
    "setDefaultAnalysis",
    "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",
    "FLAnalyzer>>analysisFor:",
    "FLSerialization>>analysisStep",
    "FLSerialization>>run",
    "setDefaultSerialization",
    "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",
    "serialize: t1 on: t2",
    "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",
    "on: t1 globalEnvironment: t2 do: t3",
    "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",
    "BlockClosure>>ensure:",
    "FLEncoder class>>on:globalEnvironment:do:",
    "FLSerializer>>serialize:on:",
    "NonInteractiveUIManager>>writeFuelContext:",
    "Lambda class>>processDebug:",
    "Lambda class>>processJSON:",
    "Lambda class>>process:",
    "activate",

On 3 Aug 2017, at 17:26, Mariano Martinez Peck <[hidden email]> wrote:



On Thu, Aug 3, 2017 at 1:17 PM, Tim Mackinnon <[hidden email]> wrote:
Hi guys - I think I’m getting close - I’m able to load remote packages, and I’ve added fuel to my repo and its loading - however I am getting an error when I run and test writing out a context - MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames

I can see that this is a trait - is trait support missing in the minimal image?


Uff...that's a good question hahahhah 
I don't know if the minimal image supports traits, but what I am almost sure, is the Behavior DOES need the methods defined in TBehavior. So maybe the minimal image flattens the traits. In this case you could do the same for Fuel (flattenDownAllTraits)
 
(I also seemed to get this far just loading the Fuel package - but just to be sure I added the other platform ones like you suggested and still get the same error). Given how simple that method is, I could just compile it in - but maybe there are many more?



TBehavior, TClass and TClassDescription are the only one that comes to my mind right now...  there are only a few methods from Fuel. So could compile those directly at least on a first step...at least you keep you going. 



 
Tim


"errorMessage": "Command failed: ./pharo Pharo.image --no-default-preferences process Lambda '{\"debug\":true,\"data\":5}'\n
  "errorType": "Error",
  "stackTrace": [
    "Writing fuel context to S3...",
    "Using S3 bucket morethan-technology-projects",
    "\u001b[31m",
    "--- RUNNING ERROR HANDLER ---",
    "MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
    "",
    "\u001b[0m\u001b[31mMessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
    "\u001b[0mContext class(Object)>>doesNotUnderstand: #fuelIgnoredInstanceVariableNames",
    "FLVariablesMapping>>instanceVariableNamesToSerialize",
    "FLVariablesMapping>>initializeAnalyzing",
    "FLVariablesMapping class>>newAnalyzing:references:",
    "FLContextCluster(FLPointerObjectCluster)>>initializeAnalyzing:",
    "FLContextCluster class(FLObjectCluster class)>>newAnalyzing:",
    "clusterKeyedByObjectClass: t1 class: t2",
    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class: in Block: clusterKeyedByObjectClass: t1 class: t2...",
    "clusterInstanceOf: t1 keyInBucket: t2 factory: t3",
    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory: in Block: clusterInstanceOf: t1 keyInBucket: t2 factory: t3...",
    "at: t1 ifAbsentPut: t2",
    "  self error: 'No decompiler available' in IdentityDictionary(Dictionary)>>at:ifAbsentPut: in Block: at: t1 ifAbsentPut: t2...",
    "IdentityDictionary(Dictionary)>>at:ifAbsent:",
    "IdentityDictionary(Dictionary)>>at:ifAbsentPut:",
    "FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory:",
    "FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class:",
    "FLLightGeneralMapper(FLMapper)>>mapAndTraceByObjectClass:to:",
    "FLLightGeneralMapper>>visitMethodContext:",
    "Context>>fuelAccept:",
    "FLLightGeneralMapper>>mapAndTrace:",
    "FLLightGlobalMapper>>mapAndTrace:",
    "FLPluggableSubstitutionMapper>>mapAndTrace:",
    "FLAnalysis>>mapAndTrace:",
    "FLAnalysis>>run",
    "setDefaultAnalysis",
    "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",
    "FLAnalyzer>>analysisFor:",
    "FLSerialization>>analysisStep",
    "FLSerialization>>run",
    "setDefaultSerialization",
    "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",
    "serialize: t1 on: t2",
    "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",
    "on: t1 globalEnvironment: t2 do: t3",
    "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",
    "BlockClosure>>ensure:",
    "FLEncoder class>>on:globalEnvironment:do:",
    "\u001b[0m"
On 3 Aug 2017, at 13:16, Mariano Martinez Peck <[hidden email]> wrote:



On Thu, Aug 3, 2017 at 8:58 AM, Tim Mackinnon <[hidden email]> wrote:
I’m so close now (bit of a diversion with gitlab yml not liking the eval string) - that did load and it got me really close - I’m just missing FlSerializer now (I have progressed my work to fuel out the context for debugging to S3)… so looking at the repo, there are quite a few fuel packages - but in a big image I can see that FLSerialiser is in Core - so do I just need core? But then there are a few fuel packages with Core?


The easiest way to understand that is either checking ConfigurationOfFuel directly or some of the related Metacello tool.
Anyway, for this case, I think you need the packages (in this order): FuelPlatform and Fuel. Bug again, Metacello allows some postLoad actions (which we use in Fuel)... so...it's not always easy to load a project without Metacello. 

Sometimes you can print the result of #record:

Metacello new 
configuration: 'Fuel';
smalltalkhubUser: 'Pharo' project: 'Fuel';
version: #stable;
record: #('Core').

->

"linear load : 
linear load : 2.1.10 [ConfigurationOfFuel]
linear load : baseline [ConfigurationOfFuelPlatform]
load : FuelPlatform-Core
load : FuelPlatform-Pharo-Core
load : FuelPlatform-Pharo-06"


------

Anyway.... I guess I would try to load these packages (in this order):

FuelPlatform-Core
FuelPlatform-Pharo-Core
FuelPlatform-Pharo-06
Fuel

And when you are down loading, then execute:

(Smalltalk at: #FLPlatform) current addHacks

 
I’m also not quite clear on how I would load packages without a Baseline (would I create my own baseline for the fuel packages I need?)

Tim

On 3 Aug 2017, at 12:07, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 13:02 GMT+02:00 Tim Mackinnon <[hidden email]>:
I really appreciate your patience and help on this - it looks very promising and I am giving it a spin now…

When you say the pharo repository - are you referring to this: https://github.com/pharo-project/pharo/tree/master/src. ? Just so I know for the future.

yes

-- Pavel 


Tim

On 3 Aug 2017, at 09:16, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 10:14 GMT+02:00 Pavel Krivanek <[hidden email]>:
The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):

Metacello-GitBasedRepository.package
Metacello-GitHub.package
SUnit-Core.package

and create baseline to load them. I already tried it so you can test it:

Then you will do something like:

(accidental message send ;-) )

BASELINE=MetacelloGitBasedRepository
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

BASELINE=Lambda
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

As I tried that. The baseline of Lambda is then able to be loaded.

-- Pavel





2017-08-02 15:18 GMT+02:00 Tim Mackinnon <[hidden email]>:
Ah, I think I’m starting to get closer to what I need…

So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?

Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?

I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.

Tim

On 2 Aug 2017, at 07:05, Guillermo Polito <[hidden email]> wrote:

Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')
    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel


 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel





-- 
   
Guille Polito

Research Engineer
French National Center for Scientific Research - http://www.cnrs.fr


Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13









-- 




-- 




--
Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Pavel Krivanek-3
In reply to this post by Tim Mackinnon


2017-08-03 18:17 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi guys - I think I’m getting close - I’m able to load remote packages, and I’ve added fuel to my repo and its loading - however I am getting an error when I run and test writing out a context - MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames

I can see that this is a trait - is trait support missing in the minimal image?

No, this image supports them 

(I also seemed to get this far just loading the Fuel package - but just to be sure I added the other platform ones like you suggested and still get the same error). Given how simple that method is, I could just compile it in - but maybe there are many more?

Tim


"errorMessage": "Command failed: ./pharo Pharo.image --no-default-preferences process Lambda '{\"debug\":true,\"data\":5}'\n
  "errorType": "Error",
  "stackTrace": [
    "Writing fuel context to S3...",
    "Using S3 bucket morethan-technology-projects",
    "\u001b[31m",
    "--- RUNNING ERROR HANDLER ---",
    "MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
    "",
    "\u001b[0m\u001b[31mMessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
    "\u001b[0mContext class(Object)>>doesNotUnderstand: #fuelIgnoredInstanceVariableNames",
    "FLVariablesMapping>>instanceVariableNamesToSerialize",
    "FLVariablesMapping>>initializeAnalyzing",
    "FLVariablesMapping class>>newAnalyzing:references:",
    "FLContextCluster(FLPointerObjectCluster)>>initializeAnalyzing:",
    "FLContextCluster class(FLObjectCluster class)>>newAnalyzing:",
    "clusterKeyedByObjectClass: t1 class: t2",
    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class: in Block: clusterKeyedByObjectClass: t1 class: t2...",
    "clusterInstanceOf: t1 keyInBucket: t2 factory: t3",
    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory: in Block: clusterInstanceOf: t1 keyInBucket: t2 factory: t3...",
    "at: t1 ifAbsentPut: t2",
    "  self error: 'No decompiler available' in IdentityDictionary(Dictionary)>>at:ifAbsentPut: in Block: at: t1 ifAbsentPut: t2...",
    "IdentityDictionary(Dictionary)>>at:ifAbsent:",
    "IdentityDictionary(Dictionary)>>at:ifAbsentPut:",
    "FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory:",
    "FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class:",
    "FLLightGeneralMapper(FLMapper)>>mapAndTraceByObjectClass:to:",
    "FLLightGeneralMapper>>visitMethodContext:",
    "Context>>fuelAccept:",
    "FLLightGeneralMapper>>mapAndTrace:",
    "FLLightGlobalMapper>>mapAndTrace:",
    "FLPluggableSubstitutionMapper>>mapAndTrace:",
    "FLAnalysis>>mapAndTrace:",
    "FLAnalysis>>run",
    "setDefaultAnalysis",
    "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",
    "FLAnalyzer>>analysisFor:",
    "FLSerialization>>analysisStep",
    "FLSerialization>>run",
    "setDefaultSerialization",
    "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",
    "serialize: t1 on: t2",
    "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",
    "on: t1 globalEnvironment: t2 do: t3",
    "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",
    "BlockClosure>>ensure:",
    "FLEncoder class>>on:globalEnvironment:do:",
    "\u001b[0m"
On 3 Aug 2017, at 13:16, Mariano Martinez Peck <[hidden email]> wrote:



On Thu, Aug 3, 2017 at 8:58 AM, Tim Mackinnon <[hidden email]> wrote:
I’m so close now (bit of a diversion with gitlab yml not liking the eval string) - that did load and it got me really close - I’m just missing FlSerializer now (I have progressed my work to fuel out the context for debugging to S3)… so looking at the repo, there are quite a few fuel packages - but in a big image I can see that FLSerialiser is in Core - so do I just need core? But then there are a few fuel packages with Core?


The easiest way to understand that is either checking ConfigurationOfFuel directly or some of the related Metacello tool.
Anyway, for this case, I think you need the packages (in this order): FuelPlatform and Fuel. Bug again, Metacello allows some postLoad actions (which we use in Fuel)... so...it's not always easy to load a project without Metacello. 

Sometimes you can print the result of #record:

Metacello new 
configuration: 'Fuel';
smalltalkhubUser: 'Pharo' project: 'Fuel';
version: #stable;
record: #('Core').

->

"linear load : 
linear load : 2.1.10 [ConfigurationOfFuel]
linear load : baseline [ConfigurationOfFuelPlatform]
load : FuelPlatform-Core
load : FuelPlatform-Pharo-Core
load : FuelPlatform-Pharo-06"


------

Anyway.... I guess I would try to load these packages (in this order):

FuelPlatform-Core
FuelPlatform-Pharo-Core
FuelPlatform-Pharo-06
Fuel

And when you are down loading, then execute:

(Smalltalk at: #FLPlatform) current addHacks

 
I’m also not quite clear on how I would load packages without a Baseline (would I create my own baseline for the fuel packages I need?)

Tim

On 3 Aug 2017, at 12:07, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 13:02 GMT+02:00 Tim Mackinnon <[hidden email]>:
I really appreciate your patience and help on this - it looks very promising and I am giving it a spin now…

When you say the pharo repository - are you referring to this: https://github.com/pharo-project/pharo/tree/master/src. ? Just so I know for the future.

yes

-- Pavel 


Tim

On 3 Aug 2017, at 09:16, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 10:14 GMT+02:00 Pavel Krivanek <[hidden email]>:
The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):

Metacello-GitBasedRepository.package
Metacello-GitHub.package
SUnit-Core.package

and create baseline to load them. I already tried it so you can test it:

Then you will do something like:

(accidental message send ;-) )

BASELINE=MetacelloGitBasedRepository
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

BASELINE=Lambda
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

As I tried that. The baseline of Lambda is then able to be loaded.

-- Pavel





2017-08-02 15:18 GMT+02:00 Tim Mackinnon <[hidden email]>:
Ah, I think I’m starting to get closer to what I need…

So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?

Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?

I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.

Tim

On 2 Aug 2017, at 07:05, Guillermo Polito <[hidden email]> wrote:

Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')
    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel


 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel





-- 
   
Guille Polito

Research Engineer
French National Center for Scientific Research - http://www.cnrs.fr


Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13









-- 


Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Tim Mackinnon
If Traits are supported then I must be doing something else wrong - maybe load order? (To be honest, I don’t fully understand how traits work - but I will keep digging).

It could be that my latest error (having just done Class compile: ‘…. Missing method code … ‘) is something deeper related to these traits methods not applying somehow… 

I’ll try a few things this morning.

It’s amazing how easily the image size jumps up when you add things though. With the original minimal v6 64bit image you sorted out for me I had a footprint of 21.6mb (image was 12mb).
Now with the v7 image - and adding fuel and some S3 handling - it jumps up to 32.4mb (image is 15mb).

I know that the S3 support has a lot of XML crud - but it seems quite astounding that this takes almost 8mb of code/objects.

AWS loading (and hence response time) seems quite sensitive to the size of the package it has to load - although equally I wonder if there is also more #startUp/initalisation cost that is going on as well as when you get a warm lambda instance (so its not just loading your package) I’m still not seeing the 400-500ms times but more 800-1000ms times. 

Its all quite interesting - and shows why the work behind a minimal image is very useful. Of course there is tiny-minimal (which I know you guys are after) and then there is small-server- minimal (which is realistically what I’m after, so I can just focus on creating my app and just carefully load code and dependencies - but still have core server services like Fuel, and Git loading).

Thanks for doing this work - as it brings a lot of useful things together.

Tim


On 4 Aug 2017, at 06:48, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 18:17 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi guys - I think I’m getting close - I’m able to load remote packages, and I’ve added fuel to my repo and its loading - however I am getting an error when I run and test writing out a context - MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames

I can see that this is a trait - is trait support missing in the minimal image?

No, this image supports them 

(I also seemed to get this far just loading the Fuel package - but just to be sure I added the other platform ones like you suggested and still get the same error). Given how simple that method is, I could just compile it in - but maybe there are many more?

Tim


"errorMessage": "Command failed: ./pharo Pharo.image --no-default-preferences process Lambda '{\"debug\":true,\"data\":5}'\n
  "errorType": "Error",
  "stackTrace": [
    "Writing fuel context to S3...",
    "Using S3 bucket morethan-technology-projects",
    "\u001b[31m",
    "--- RUNNING ERROR HANDLER ---",
    "MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
    "",
    "\u001b[0m\u001b[31mMessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
    "\u001b[0mContext class(Object)>>doesNotUnderstand: #fuelIgnoredInstanceVariableNames",
    "FLVariablesMapping>>instanceVariableNamesToSerialize",
    "FLVariablesMapping>>initializeAnalyzing",
    "FLVariablesMapping class>>newAnalyzing:references:",
    "FLContextCluster(FLPointerObjectCluster)>>initializeAnalyzing:",
    "FLContextCluster class(FLObjectCluster class)>>newAnalyzing:",
    "clusterKeyedByObjectClass: t1 class: t2",
    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class: in Block: clusterKeyedByObjectClass: t1 class: t2...",
    "clusterInstanceOf: t1 keyInBucket: t2 factory: t3",
    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory: in Block: clusterInstanceOf: t1 keyInBucket: t2 factory: t3...",
    "at: t1 ifAbsentPut: t2",
    "  self error: 'No decompiler available' in IdentityDictionary(Dictionary)>>at:ifAbsentPut: in Block: at: t1 ifAbsentPut: t2...",
    "IdentityDictionary(Dictionary)>>at:ifAbsent:",
    "IdentityDictionary(Dictionary)>>at:ifAbsentPut:",
    "FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory:",
    "FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class:",
    "FLLightGeneralMapper(FLMapper)>>mapAndTraceByObjectClass:to:",
    "FLLightGeneralMapper>>visitMethodContext:",
    "Context>>fuelAccept:",
    "FLLightGeneralMapper>>mapAndTrace:",
    "FLLightGlobalMapper>>mapAndTrace:",
    "FLPluggableSubstitutionMapper>>mapAndTrace:",
    "FLAnalysis>>mapAndTrace:",
    "FLAnalysis>>run",
    "setDefaultAnalysis",
    "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",
    "FLAnalyzer>>analysisFor:",
    "FLSerialization>>analysisStep",
    "FLSerialization>>run",
    "setDefaultSerialization",
    "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",
    "serialize: t1 on: t2",
    "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",
    "on: t1 globalEnvironment: t2 do: t3",
    "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",
    "BlockClosure>>ensure:",
    "FLEncoder class>>on:globalEnvironment:do:",
    "\u001b[0m"
On 3 Aug 2017, at 13:16, Mariano Martinez Peck <[hidden email]> wrote:



On Thu, Aug 3, 2017 at 8:58 AM, Tim Mackinnon <[hidden email]> wrote:
I’m so close now (bit of a diversion with gitlab yml not liking the eval string) - that did load and it got me really close - I’m just missing FlSerializer now (I have progressed my work to fuel out the context for debugging to S3)… so looking at the repo, there are quite a few fuel packages - but in a big image I can see that FLSerialiser is in Core - so do I just need core? But then there are a few fuel packages with Core?


The easiest way to understand that is either checking ConfigurationOfFuel directly or some of the related Metacello tool.
Anyway, for this case, I think you need the packages (in this order): FuelPlatform and Fuel. Bug again, Metacello allows some postLoad actions (which we use in Fuel)... so...it's not always easy to load a project without Metacello. 

Sometimes you can print the result of #record:

Metacello new 
configuration: 'Fuel';
smalltalkhubUser: 'Pharo' project: 'Fuel';
version: #stable;
record: #('Core').

->

"linear load : 
linear load : 2.1.10 [ConfigurationOfFuel]
linear load : baseline [ConfigurationOfFuelPlatform]
load : FuelPlatform-Core
load : FuelPlatform-Pharo-Core
load : FuelPlatform-Pharo-06"


------

Anyway.... I guess I would try to load these packages (in this order):

FuelPlatform-Core
FuelPlatform-Pharo-Core
FuelPlatform-Pharo-06
Fuel

And when you are down loading, then execute:

(Smalltalk at: #FLPlatform) current addHacks

 
I’m also not quite clear on how I would load packages without a Baseline (would I create my own baseline for the fuel packages I need?)

Tim

On 3 Aug 2017, at 12:07, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 13:02 GMT+02:00 Tim Mackinnon <[hidden email]>:
I really appreciate your patience and help on this - it looks very promising and I am giving it a spin now…

When you say the pharo repository - are you referring to this: https://github.com/pharo-project/pharo/tree/master/src. ? Just so I know for the future.

yes

-- Pavel 


Tim

On 3 Aug 2017, at 09:16, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 10:14 GMT+02:00 Pavel Krivanek <[hidden email]>:
The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):

Metacello-GitBasedRepository.package
Metacello-GitHub.package
SUnit-Core.package

and create baseline to load them. I already tried it so you can test it:

Then you will do something like:

(accidental message send ;-) )

BASELINE=MetacelloGitBasedRepository
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

BASELINE=Lambda
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

As I tried that. The baseline of Lambda is then able to be loaded.

-- Pavel





2017-08-02 15:18 GMT+02:00 Tim Mackinnon <[hidden email]>:
Ah, I think I’m starting to get closer to what I need…

So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?

Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?

I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.

Tim

On 2 Aug 2017, at 07:05, Guillermo Polito <[hidden email]> wrote:

Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')
    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel


 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel





-- 
   
Guille Polito

Research Engineer
French National Center for Scientific Research - http://www.cnrs.fr


Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank" class="">+33 06 52 70 66 13









-- 



Reply | Threaded
Open this post in threaded view
|

Re: Creating the smallest server runtime footprint

Tim Mackinnon
In reply to this post by Mariano Martinez Peck
Hi Mariano - I’ve looked at this again, and it was my fault - I was compiling in :

fuelAccept: aGeneralMapper
^self explicitRequirement.

Because when I looked at the Traits for fuel - this is what it had for TClass? - as I’m not so clear on traits, what does this mean - as I had just assumed that I should flatten the Fuel methods for TCLass, TBehaviour and TClassDescription - but I think I’ve misunderstood what I should do (and I guess from Pavel’s comment - I should really look at why that initial method was missing as it sounds like it should have worked  - so maybe its an order loading thing?).

Thanks, for helping by the way - this is proving very instructional.

Tim

On 3 Aug 2017, at 18:26, Mariano Martinez Peck <[hidden email]> wrote:


On Thu, Aug 3, 2017 at 2:14 PM, Tim Mackinnon <[hidden email]> wrote:
Hi Mariano - I compiled in the trait methods manually and it got further - but now I’m seeing another error - Error: Explicitly required method",
    "\u001b[0mLambda class(Object)>>error:",
    "explicitRequirement”,


mmmm the stack is weird... it looks like if Class >> fuelAccept: would have the wrong implementation, that is, having the one from TClass:

fuelAccept: aGeneralMapper
^self explicitRequirement.

Rather than the correct one:

fuelAccept: aGeneralMapper

^aGeneralMapper visitClass: self


Maybe the solution is to compile all those methods from traits BUT NOT those that are "^self explicitRequirement." because those will override the good methods.

If that doesn't work, then maybe can you print to console the each impl (source) of each implementor of #fuelAccept: ?

Thoughts?



 
Any ideas?

Tim

"--- RUNNING ERROR HANDLER ---",
    "Error: Explicitly required method",
    "",
    "\u001b[0m\u001b[31mError: Explicitly required method",
    "\u001b[0mLambda class(Object)>>error:",
    "explicitRequirement",
    "  self error: 'No decompiler available' in Lambda class(Object)>>explicitRequirement in Block: explicitRequirement...",
    "False>>ifFalse:",
    "Lambda class(Object)>>explicitRequirement",
    "Lambda class(Class)>>fuelAccept:",
    "FLLightGeneralMapper>>mapAndTrace:",
    "FLLightGlobalMapper>>mapAndTrace:",
    "FLPluggableSubstitutionMapper>>mapAndTrace:",
    "FLAnalysis>>mapAndTrace:",
    "FLAnalysis>>run",
    "setDefaultAnalysis",
    "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",
    "FLAnalyzer>>analysisFor:",
    "FLSerialization>>analysisStep",
    "FLSerialization>>run",
    "setDefaultSerialization",
    "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",
    "serialize: t1 on: t2",
    "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",
    "on: t1 globalEnvironment: t2 do: t3",
    "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",
    "BlockClosure>>ensure:",
    "FLEncoder class>>on:globalEnvironment:do:",
    "FLSerializer>>serialize:on:",
    "NonInteractiveUIManager>>writeFuelContext:",
    "Lambda class>>processDebug:",
    "Lambda class>>processJSON:",
    "Lambda class>>process:",
    "activate",

On 3 Aug 2017, at 17:26, Mariano Martinez Peck <[hidden email]> wrote:



On Thu, Aug 3, 2017 at 1:17 PM, Tim Mackinnon <[hidden email]> wrote:
Hi guys - I think I’m getting close - I’m able to load remote packages, and I’ve added fuel to my repo and its loading - however I am getting an error when I run and test writing out a context - MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames

I can see that this is a trait - is trait support missing in the minimal image?


Uff...that's a good question hahahhah 
I don't know if the minimal image supports traits, but what I am almost sure, is the Behavior DOES need the methods defined in TBehavior. So maybe the minimal image flattens the traits. In this case you could do the same for Fuel (flattenDownAllTraits)
 
(I also seemed to get this far just loading the Fuel package - but just to be sure I added the other platform ones like you suggested and still get the same error). Given how simple that method is, I could just compile it in - but maybe there are many more?



TBehavior, TClass and TClassDescription are the only one that comes to my mind right now...  there are only a few methods from Fuel. So could compile those directly at least on a first step...at least you keep you going. 



 
Tim


"errorMessage": "Command failed: ./pharo Pharo.image --no-default-preferences process Lambda '{\"debug\":true,\"data\":5}'\n
  "errorType": "Error",
  "stackTrace": [
    "Writing fuel context to S3...",
    "Using S3 bucket morethan-technology-projects",
    "\u001b[31m",
    "--- RUNNING ERROR HANDLER ---",
    "MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
    "",
    "\u001b[0m\u001b[31mMessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
    "\u001b[0mContext class(Object)>>doesNotUnderstand: #fuelIgnoredInstanceVariableNames",
    "FLVariablesMapping>>instanceVariableNamesToSerialize",
    "FLVariablesMapping>>initializeAnalyzing",
    "FLVariablesMapping class>>newAnalyzing:references:",
    "FLContextCluster(FLPointerObjectCluster)>>initializeAnalyzing:",
    "FLContextCluster class(FLObjectCluster class)>>newAnalyzing:",
    "clusterKeyedByObjectClass: t1 class: t2",
    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class: in Block: clusterKeyedByObjectClass: t1 class: t2...",
    "clusterInstanceOf: t1 keyInBucket: t2 factory: t3",
    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory: in Block: clusterInstanceOf: t1 keyInBucket: t2 factory: t3...",
    "at: t1 ifAbsentPut: t2",
    "  self error: 'No decompiler available' in IdentityDictionary(Dictionary)>>at:ifAbsentPut: in Block: at: t1 ifAbsentPut: t2...",
    "IdentityDictionary(Dictionary)>>at:ifAbsent:",
    "IdentityDictionary(Dictionary)>>at:ifAbsentPut:",
    "FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory:",
    "FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class:",
    "FLLightGeneralMapper(FLMapper)>>mapAndTraceByObjectClass:to:",
    "FLLightGeneralMapper>>visitMethodContext:",
    "Context>>fuelAccept:",
    "FLLightGeneralMapper>>mapAndTrace:",
    "FLLightGlobalMapper>>mapAndTrace:",
    "FLPluggableSubstitutionMapper>>mapAndTrace:",
    "FLAnalysis>>mapAndTrace:",
    "FLAnalysis>>run",
    "setDefaultAnalysis",
    "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",
    "FLAnalyzer>>analysisFor:",
    "FLSerialization>>analysisStep",
    "FLSerialization>>run",
    "setDefaultSerialization",
    "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",
    "serialize: t1 on: t2",
    "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",
    "on: t1 globalEnvironment: t2 do: t3",
    "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",
    "BlockClosure>>ensure:",
    "FLEncoder class>>on:globalEnvironment:do:",
    "\u001b[0m"
On 3 Aug 2017, at 13:16, Mariano Martinez Peck <[hidden email]> wrote:



On Thu, Aug 3, 2017 at 8:58 AM, Tim Mackinnon <[hidden email]> wrote:
I’m so close now (bit of a diversion with gitlab yml not liking the eval string) - that did load and it got me really close - I’m just missing FlSerializer now (I have progressed my work to fuel out the context for debugging to S3)… so looking at the repo, there are quite a few fuel packages - but in a big image I can see that FLSerialiser is in Core - so do I just need core? But then there are a few fuel packages with Core?


The easiest way to understand that is either checking ConfigurationOfFuel directly or some of the related Metacello tool.
Anyway, for this case, I think you need the packages (in this order): FuelPlatform and Fuel. Bug again, Metacello allows some postLoad actions (which we use in Fuel)... so...it's not always easy to load a project without Metacello. 

Sometimes you can print the result of #record:

Metacello new 
configuration: 'Fuel';
smalltalkhubUser: 'Pharo' project: 'Fuel';
version: #stable;
record: #('Core').

->

"linear load : 
linear load : 2.1.10 [ConfigurationOfFuel]
linear load : baseline [ConfigurationOfFuelPlatform]
load : FuelPlatform-Core
load : FuelPlatform-Pharo-Core
load : FuelPlatform-Pharo-06"


------

Anyway.... I guess I would try to load these packages (in this order):

FuelPlatform-Core
FuelPlatform-Pharo-Core
FuelPlatform-Pharo-06
Fuel

And when you are down loading, then execute:

(Smalltalk at: #FLPlatform) current addHacks

 
I’m also not quite clear on how I would load packages without a Baseline (would I create my own baseline for the fuel packages I need?)

Tim

On 3 Aug 2017, at 12:07, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 13:02 GMT+02:00 Tim Mackinnon <[hidden email]>:
I really appreciate your patience and help on this - it looks very promising and I am giving it a spin now…

When you say the pharo repository - are you referring to this: https://github.com/pharo-project/pharo/tree/master/src. ? Just so I know for the future.

yes

-- Pavel 


Tim

On 3 Aug 2017, at 09:16, Pavel Krivanek <[hidden email]> wrote:



2017-08-03 10:14 GMT+02:00 Pavel Krivanek <[hidden email]>:
The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):

Metacello-GitBasedRepository.package
Metacello-GitHub.package
SUnit-Core.package

and create baseline to load them. I already tried it so you can test it:

Then you will do something like:

(accidental message send ;-) )

BASELINE=MetacelloGitBasedRepository
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

BASELINE=Lambda
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

As I tried that. The baseline of Lambda is then able to be loaded.

-- Pavel





2017-08-02 15:18 GMT+02:00 Tim Mackinnon <[hidden email]>:
Ah, I think I’m starting to get closer to what I need…

So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?

Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?

I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.

Tim

On 2 Aug 2017, at 07:05, Guillermo Polito <[hidden email]> wrote:

Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')
    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec
<baseline>

spec for: #common do: [
spec configuration: 'ZTimestamp' with: [
spec 
versionString: #stable;
repository: 'http://mc.stfx.eu/Neo' ].
spec baseline: 'AWS' with: [
spec 
package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].


The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel


 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:



2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:
I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).


25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:
Metacello new
repository: 'filetree://../src';
baseline: 'Lambda';
load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim


I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."
 
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel





-- 
   
Guille Polito

Research Engineer
French National Center for Scientific Research - http://www.cnrs.fr


Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank" class="">+33 06 52 70 66 13









-- 




-- 




--

12