Re: Creating the smallest server runtime footprint

On 4 Aug 2017, at 11:01, Tim Mackinnon <[hidden email]> wrote:

Hi Mariano - I’ve looked at this again, and it was my fault - I was compiling in :

fuelAccept: aGeneralMapper

^self explicitRequirement.

Because when I looked at the Traits for fuel - this is what it had for TClass? - as I’m not so clear on traits, what does this mean - as I had just assumed that I should flatten the Fuel methods for TCLass, TBehaviour and TClassDescription - but I think I’ve misunderstood what I should do (and I guess from Pavel’s comment - I should really look at why that initial method was missing as it sounds like it should have worked  - so maybe its an order loading thing?).

Thanks, for helping by the way - this is proving very instructional.

Tim

On 3 Aug 2017, at 18:26, Mariano Martinez Peck <[hidden email]> wrote:

On Thu, Aug 3, 2017 at 2:14 PM, Tim Mackinnon <[hidden email]> wrote:

Hi Mariano - I compiled in the trait methods manually and it got further - but now I’m seeing another error - Error: Explicitly required method",

    "\u001b[0mLambda class(Object)>>error:",

    "explicitRequirement”,

mmmm the stack is weird... it looks like if Class >> fuelAccept: would have the wrong implementation, that is, having the one from TClass:

fuelAccept: aGeneralMapper

^self explicitRequirement.

Rather than the correct one:

fuelAccept: aGeneralMapper

^aGeneralMapper visitClass: self

Maybe the solution is to compile all those methods from traits BUT NOT those that are "^self explicitRequirement." because those will override the good methods.

If that doesn't work, then maybe can you print to console the each impl (source) of each implementor of #fuelAccept: ?

Thoughts?

 

Any ideas?

Tim

"--- RUNNING ERROR HANDLER ---",
    "Error: Explicitly required method",
    "",
    "\u001b[0m\u001b[31mError: Explicitly required method",
    "\u001b[0mLambda class(Object)>>error:",
    "explicitRequirement",
    "  self error: 'No decompiler available' in Lambda class(Object)>>explicitRequirement in Block: explicitRequirement...",
    "False>>ifFalse:",
    "Lambda class(Object)>>explicitRequirement",
    "Lambda class(Class)>>fuelAccept:",
    "FLLightGeneralMapper>>mapAndTrace:",
    "FLLightGlobalMapper>>mapAndTrace:",
    "FLPluggableSubstitutionMapper>>mapAndTrace:",
    "FLAnalysis>>mapAndTrace:",
    "FLAnalysis>>run",
    "setDefaultAnalysis",
    "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",
    "FLAnalyzer>>analysisFor:",
    "FLSerialization>>analysisStep",
    "FLSerialization>>run",
    "setDefaultSerialization",
    "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",
    "serialize: t1 on: t2",
    "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",
    "on: t1 globalEnvironment: t2 do: t3",
    "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",
    "BlockClosure>>ensure:",
    "FLEncoder class>>on:globalEnvironment:do:",
    "FLSerializer>>serialize:on:",
    "NonInteractiveUIManager>>writeFuelContext:",
    "Lambda class>>processDebug:",
    "Lambda class>>processJSON:",
    "Lambda class>>process:",
    "activate",

On 3 Aug 2017, at 17:26, Mariano Martinez Peck <[hidden email]> wrote:

On Thu, Aug 3, 2017 at 1:17 PM, Tim Mackinnon <[hidden email]> wrote:

Hi guys - I think I’m getting close - I’m able to load remote packages, and I’ve added fuel to my repo and its loading - however I am getting an error when I run and test writing out a context - MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames

I can see that this is a trait - is trait support missing in the minimal image?

Uff...that's a good question hahahhah 

I don't know if the minimal image supports traits, but what I am almost sure, is the Behavior DOES need the methods defined in TBehavior. So maybe the minimal image flattens the traits. In this case you could do the same for Fuel (flattenDownAllTraits)

 

(I also seemed to get this far just loading the Fuel package - but just to be sure I added the other platform ones like you suggested and still get the same error). Given how simple that method is, I could just compile it in - but maybe there are many more?

TBehavior, TClass and TClassDescription are the only one that comes to my mind right now...  there are only a few methods from Fuel. So could compile those directly at least on a first step...at least you keep you going. 

 

Tim

"errorMessage": "Command failed: ./pharo Pharo.image --no-default-preferences process Lambda '{\"debug\":true,\"data\":5}'\n

  "errorType": "Error",

  "stackTrace": [

    "Writing fuel context to S3...",

    "Using S3 bucket morethan-technology-projects",

    "\u001b[31m",

    "--- RUNNING ERROR HANDLER ---",

    "MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",

    "",

    "\u001b[0m\u001b[31mMessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",

    "\u001b[0mContext class(Object)>>doesNotUnderstand: #fuelIgnoredInstanceVariableNames",

    "FLVariablesMapping>>instanceVariableNamesToSerialize",

    "FLVariablesMapping>>initializeAnalyzing",

    "FLVariablesMapping class>>newAnalyzing:references:",

    "FLContextCluster(FLPointerObjectCluster)>>initializeAnalyzing:",

    "FLContextCluster class(FLObjectCluster class)>>newAnalyzing:",

    "clusterKeyedByObjectClass: t1 class: t2",

    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class: in Block: clusterKeyedByObjectClass: t1 class: t2...",

    "clusterInstanceOf: t1 keyInBucket: t2 factory: t3",

    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory: in Block: clusterInstanceOf: t1 keyInBucket: t2 factory: t3...",

    "at: t1 ifAbsentPut: t2",

    "  self error: 'No decompiler available' in IdentityDictionary(Dictionary)>>at:ifAbsentPut: in Block: at: t1 ifAbsentPut: t2...",

    "IdentityDictionary(Dictionary)>>at:ifAbsent:",

    "IdentityDictionary(Dictionary)>>at:ifAbsentPut:",

    "FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory:",

    "FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class:",

    "FLLightGeneralMapper(FLMapper)>>mapAndTraceByObjectClass:to:",

    "FLLightGeneralMapper>>visitMethodContext:",

    "Context>>fuelAccept:",

    "FLLightGeneralMapper>>mapAndTrace:",

    "FLLightGlobalMapper>>mapAndTrace:",

    "FLPluggableSubstitutionMapper>>mapAndTrace:",

    "FLAnalysis>>mapAndTrace:",

    "FLAnalysis>>run",

    "setDefaultAnalysis",

    "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",

    "FLAnalyzer>>analysisFor:",

    "FLSerialization>>analysisStep",

    "FLSerialization>>run",

    "setDefaultSerialization",

    "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",

    "serialize: t1 on: t2",

    "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",

    "on: t1 globalEnvironment: t2 do: t3",

    "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",

    "BlockClosure>>ensure:",

    "FLEncoder class>>on:globalEnvironment:do:",

    "\u001b[0m"

On 3 Aug 2017, at 13:16, Mariano Martinez Peck <[hidden email]> wrote:

On Thu, Aug 3, 2017 at 8:58 AM, Tim Mackinnon <[hidden email]> wrote:

I’m so close now (bit of a diversion with gitlab yml not liking the eval string) - that did load and it got me really close - I’m just missing FlSerializer now (I have progressed my work to fuel out the context for debugging to S3)… so looking at the repo, there are quite a few fuel packages - but in a big image I can see that FLSerialiser is in Core - so do I just need core? But then there are a few fuel packages with Core?

The easiest way to understand that is either checking ConfigurationOfFuel directly or some of the related Metacello tool.

Anyway, for this case, I think you need the packages (in this order): FuelPlatform and Fuel. Bug again, Metacello allows some postLoad actions (which we use in Fuel)... so...it's not always easy to load a project without Metacello. 

Sometimes you can print the result of #record:

Metacello new 

configuration: 'Fuel';

smalltalkhubUser: 'Pharo' project: 'Fuel';

version: #stable;

record: #('Core').

->

"linear load : 

linear load : 2.1.10 [ConfigurationOfFuel]

linear load : baseline [ConfigurationOfFuelPlatform]

load : FuelPlatform-Core

load : FuelPlatform-Pharo-Core

load : FuelPlatform-Pharo-06"

------

Anyway.... I guess I would try to load these packages (in this order):

FuelPlatform-Core

FuelPlatform-Pharo-Core

FuelPlatform-Pharo-06

Fuel

And when you are down loading, then execute:

(Smalltalk at: #FLPlatform) current addHacks

 

I’m also not quite clear on how I would load packages without a Baseline (would I create my own baseline for the fuel packages I need?)

Tim

On 3 Aug 2017, at 12:07, Pavel Krivanek <[hidden email]> wrote:

2017-08-03 13:02 GMT+02:00 Tim Mackinnon <[hidden email]>:

I really appreciate your patience and help on this - it looks very promising and I am giving it a spin now…

When you say the pharo repository - are you referring to this: https://github.com/pharo-project/pharo/tree/master/src. ? Just so I know for the future.

yes

-- Pavel 

Tim

On 3 Aug 2017, at 09:16, Pavel Krivanek <[hidden email]> wrote:

2017-08-03 10:14 GMT+02:00 Pavel Krivanek <[hidden email]>:

The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):

Metacello-GitBasedRepository.package

Metacello-GitHub.package

SUnit-Core.package

and create baseline to load them. I already tried it so you can test it:

https://drive.google.com/open?id=0BzSsmZhqtUTeMVJacTZtdW5UQW8

Then you will do something like:

(accidental message send ;-) )

BASELINE=MetacelloGitBasedRepository

pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \

"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

BASELINE=Lambda

pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \

"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

As I tried that. The baseline of Lambda is then able to be loaded.

-- Pavel

2017-08-02 15:18 GMT+02:00 Tim Mackinnon <[hidden email]>:

Ah, I think I’m starting to get closer to what I need…

So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?

Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?

I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.

Tim

On 2 Aug 2017, at 07:05, Guillermo Polito <[hidden email]> wrote:

Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')

    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:

Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec

<baseline>

spec for: #common do: [

spec configuration: 'ZTimestamp' with: [

spec 

versionString: #stable;

repository: 'http://mc.stfx.eu/Neo' ].

spec baseline: 'AWS' with: [

spec repository: 'github://newapplesho/aws-sdk-smalltalk:v1.10/pharo-repository' ].

spec 

package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].

].

The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:

26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel

 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:

2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:

I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).

The https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit-Conversion/lastSuccessfulBuild/artifact/latest-minimal-64.zip one gives me a walkback when trying to run my install script :

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:

26 MCRepositoryGroup>>addRepository:

27 createRepository

  | repo |

  repo := self project createRepository: self.

  ^ MCRepositoryGroup default repositories

    detect: [ :each | each = repo ]

    ifNone: [ 

      MCRepositoryGroup default addRepository: repo.

      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:

Metacello new
    repository: 'filetree://../src';
    baseline: 'Lambda';
    load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim

I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."

 

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel

-- 

Guille Polito Research Engineer French National Center for Scientific Research - http://www.cnrs.fr Web: http://guillep.github.io Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank" class="">+33 06 52 70 66 13

-- 

Mariano
http://marianopeck.wordpress.com

-- 

Mariano
http://marianopeck.wordpress.com

--

Mariano
http://marianopeck.wordpress.com

On 4 Aug 2017, at 11:10, Tim Mackinnon <[hidden email]> wrote:

Actually there is something weird going on - as now I’m getting a complaint about "FLPharoPlatform had the subclass responsibility to implement #isBigEndian”, and that’s in the FuelPlatform-Pharo-06 which you said to include (and it looks like I am).

I need to double check this - as I must be making some trivial mistake.

Tim

On 4 Aug 2017, at 11:01, Tim Mackinnon <[hidden email]> wrote:

Hi Mariano - I’ve looked at this again, and it was my fault - I was compiling in :

fuelAccept: aGeneralMapper

^self explicitRequirement.

Because when I looked at the Traits for fuel - this is what it had for TClass? - as I’m not so clear on traits, what does this mean - as I had just assumed that I should flatten the Fuel methods for TCLass, TBehaviour and TClassDescription - but I think I’ve misunderstood what I should do (and I guess from Pavel’s comment - I should really look at why that initial method was missing as it sounds like it should have worked  - so maybe its an order loading thing?).

Thanks, for helping by the way - this is proving very instructional.

Tim

On 3 Aug 2017, at 18:26, Mariano Martinez Peck <[hidden email]> wrote:

On Thu, Aug 3, 2017 at 2:14 PM, Tim Mackinnon <[hidden email]> wrote:

Hi Mariano - I compiled in the trait methods manually and it got further - but now I’m seeing another error - Error: Explicitly required method",

    "\u001b[0mLambda class(Object)>>error:",

    "explicitRequirement”,

mmmm the stack is weird... it looks like if Class >> fuelAccept: would have the wrong implementation, that is, having the one from TClass:

fuelAccept: aGeneralMapper

^self explicitRequirement.

Rather than the correct one:

fuelAccept: aGeneralMapper

^aGeneralMapper visitClass: self

Maybe the solution is to compile all those methods from traits BUT NOT those that are "^self explicitRequirement." because those will override the good methods.

If that doesn't work, then maybe can you print to console the each impl (source) of each implementor of #fuelAccept: ?

Thoughts?

 

Any ideas?

Tim

"--- RUNNING ERROR HANDLER ---",
    "Error: Explicitly required method",
    "",
    "\u001b[0m\u001b[31mError: Explicitly required method",
    "\u001b[0mLambda class(Object)>>error:",
    "explicitRequirement",
    "  self error: 'No decompiler available' in Lambda class(Object)>>explicitRequirement in Block: explicitRequirement...",
    "False>>ifFalse:",
    "Lambda class(Object)>>explicitRequirement",
    "Lambda class(Class)>>fuelAccept:",
    "FLLightGeneralMapper>>mapAndTrace:",
    "FLLightGlobalMapper>>mapAndTrace:",
    "FLPluggableSubstitutionMapper>>mapAndTrace:",
    "FLAnalysis>>mapAndTrace:",
    "FLAnalysis>>run",
    "setDefaultAnalysis",
    "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",
    "FLAnalyzer>>analysisFor:",
    "FLSerialization>>analysisStep",
    "FLSerialization>>run",
    "setDefaultSerialization",
    "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",
    "serialize: t1 on: t2",
    "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",
    "on: t1 globalEnvironment: t2 do: t3",
    "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",
    "BlockClosure>>ensure:",
    "FLEncoder class>>on:globalEnvironment:do:",
    "FLSerializer>>serialize:on:",
    "NonInteractiveUIManager>>writeFuelContext:",
    "Lambda class>>processDebug:",
    "Lambda class>>processJSON:",
    "Lambda class>>process:",
    "activate",

On 3 Aug 2017, at 17:26, Mariano Martinez Peck <[hidden email]> wrote:

On Thu, Aug 3, 2017 at 1:17 PM, Tim Mackinnon <[hidden email]> wrote:

Hi guys - I think I’m getting close - I’m able to load remote packages, and I’ve added fuel to my repo and its loading - however I am getting an error when I run and test writing out a context - MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames

I can see that this is a trait - is trait support missing in the minimal image?

Uff...that's a good question hahahhah 

I don't know if the minimal image supports traits, but what I am almost sure, is the Behavior DOES need the methods defined in TBehavior. So maybe the minimal image flattens the traits. In this case you could do the same for Fuel (flattenDownAllTraits)

 

(I also seemed to get this far just loading the Fuel package - but just to be sure I added the other platform ones like you suggested and still get the same error). Given how simple that method is, I could just compile it in - but maybe there are many more?

TBehavior, TClass and TClassDescription are the only one that comes to my mind right now...  there are only a few methods from Fuel. So could compile those directly at least on a first step...at least you keep you going. 

 

Tim

"errorMessage": "Command failed: ./pharo Pharo.image --no-default-preferences process Lambda '{\"debug\":true,\"data\":5}'\n

  "errorType": "Error",

  "stackTrace": [

    "Writing fuel context to S3...",

    "Using S3 bucket morethan-technology-projects",

    "\u001b[31m",

    "--- RUNNING ERROR HANDLER ---",

    "MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",

    "",

    "\u001b[0m\u001b[31mMessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",

    "\u001b[0mContext class(Object)>>doesNotUnderstand: #fuelIgnoredInstanceVariableNames",

    "FLVariablesMapping>>instanceVariableNamesToSerialize",

    "FLVariablesMapping>>initializeAnalyzing",

    "FLVariablesMapping class>>newAnalyzing:references:",

    "FLContextCluster(FLPointerObjectCluster)>>initializeAnalyzing:",

    "FLContextCluster class(FLObjectCluster class)>>newAnalyzing:",

    "clusterKeyedByObjectClass: t1 class: t2",

    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class: in Block: clusterKeyedByObjectClass: t1 class: t2...",

    "clusterInstanceOf: t1 keyInBucket: t2 factory: t3",

    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory: in Block: clusterInstanceOf: t1 keyInBucket: t2 factory: t3...",

    "at: t1 ifAbsentPut: t2",

    "  self error: 'No decompiler available' in IdentityDictionary(Dictionary)>>at:ifAbsentPut: in Block: at: t1 ifAbsentPut: t2...",

    "IdentityDictionary(Dictionary)>>at:ifAbsent:",

    "IdentityDictionary(Dictionary)>>at:ifAbsentPut:",

    "FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory:",

    "FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class:",

    "FLLightGeneralMapper(FLMapper)>>mapAndTraceByObjectClass:to:",

    "FLLightGeneralMapper>>visitMethodContext:",

    "Context>>fuelAccept:",

    "FLLightGeneralMapper>>mapAndTrace:",

    "FLLightGlobalMapper>>mapAndTrace:",

    "FLPluggableSubstitutionMapper>>mapAndTrace:",

    "FLAnalysis>>mapAndTrace:",

    "FLAnalysis>>run",

    "setDefaultAnalysis",

    "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",

    "FLAnalyzer>>analysisFor:",

    "FLSerialization>>analysisStep",

    "FLSerialization>>run",

    "setDefaultSerialization",

    "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",

    "serialize: t1 on: t2",

    "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",

    "on: t1 globalEnvironment: t2 do: t3",

    "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",

    "BlockClosure>>ensure:",

    "FLEncoder class>>on:globalEnvironment:do:",

    "\u001b[0m"

On 3 Aug 2017, at 13:16, Mariano Martinez Peck <[hidden email]> wrote:

On Thu, Aug 3, 2017 at 8:58 AM, Tim Mackinnon <[hidden email]> wrote:

I’m so close now (bit of a diversion with gitlab yml not liking the eval string) - that did load and it got me really close - I’m just missing FlSerializer now (I have progressed my work to fuel out the context for debugging to S3)… so looking at the repo, there are quite a few fuel packages - but in a big image I can see that FLSerialiser is in Core - so do I just need core? But then there are a few fuel packages with Core?

The easiest way to understand that is either checking ConfigurationOfFuel directly or some of the related Metacello tool.

Anyway, for this case, I think you need the packages (in this order): FuelPlatform and Fuel. Bug again, Metacello allows some postLoad actions (which we use in Fuel)... so...it's not always easy to load a project without Metacello. 

Sometimes you can print the result of #record:

Metacello new 

configuration: 'Fuel';

smalltalkhubUser: 'Pharo' project: 'Fuel';

version: #stable;

record: #('Core').

->

"linear load : 

linear load : 2.1.10 [ConfigurationOfFuel]

linear load : baseline [ConfigurationOfFuelPlatform]

load : FuelPlatform-Core

load : FuelPlatform-Pharo-Core

load : FuelPlatform-Pharo-06"

------

Anyway.... I guess I would try to load these packages (in this order):

FuelPlatform-Core

FuelPlatform-Pharo-Core

FuelPlatform-Pharo-06

Fuel

And when you are down loading, then execute:

(Smalltalk at: #FLPlatform) current addHacks

 

I’m also not quite clear on how I would load packages without a Baseline (would I create my own baseline for the fuel packages I need?)

Tim

On 3 Aug 2017, at 12:07, Pavel Krivanek <[hidden email]> wrote:

2017-08-03 13:02 GMT+02:00 Tim Mackinnon <[hidden email]>:

I really appreciate your patience and help on this - it looks very promising and I am giving it a spin now…

When you say the pharo repository - are you referring to this: https://github.com/pharo-project/pharo/tree/master/src. ? Just so I know for the future.

yes

-- Pavel 

Tim

On 3 Aug 2017, at 09:16, Pavel Krivanek <[hidden email]> wrote:

2017-08-03 10:14 GMT+02:00 Pavel Krivanek <[hidden email]>:

The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):

Metacello-GitBasedRepository.package

Metacello-GitHub.package

SUnit-Core.package

and create baseline to load them. I already tried it so you can test it:

https://drive.google.com/open?id=0BzSsmZhqtUTeMVJacTZtdW5UQW8

Then you will do something like:

(accidental message send ;-) )

BASELINE=MetacelloGitBasedRepository

pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \

"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

BASELINE=Lambda

pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \

"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src'; load."

As I tried that. The baseline of Lambda is then able to be loaded.

-- Pavel

2017-08-02 15:18 GMT+02:00 Tim Mackinnon <[hidden email]>:

Ah, I think I’m starting to get closer to what I need…

So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?

Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?

I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.

Tim

On 2 Aug 2017, at 07:05, Guillermo Polito <[hidden email]> wrote:

Yes, you should be able to load an mcz in that image by doing:

(MCDirectoryRepository new directory: 'where-your-mcz-is')

    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.

Guille

On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <[hidden email]> wrote:

2017-08-01 23:13 GMT+02:00 Tim Mackinnon <[hidden email]>:

Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.

The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:

baseline: spec

<baseline>

spec for: #common do: [

spec configuration: 'ZTimestamp' with: [

spec 

versionString: #stable;

repository: 'http://mc.stfx.eu/Neo' ].

spec baseline: 'AWS' with: [

spec repository: 'github://newapplesho/aws-sdk-smalltalk:v1.10/pharo-repository' ].

spec 

package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].

].

The “spec configuration: ….” Specifications cause the error:

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:

26 MCRepositoryGroup>>addRepository:

If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.

So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 

The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.

-- Pavel

 

Tim

On 1 Aug 2017, at 10:06, Pavel Krivanek <[hidden email]> wrote:

2017-07-31 22:51 GMT+02:00 Tim Mackinnon <[hidden email]>:

I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip one still shows as being last updated 7 days ago.

The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).

The https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit-Conversion/lastSuccessfulBuild/artifact/latest-minimal-64.zip one gives me a walkback when trying to run my install script :

25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:

26 MCRepositoryGroup>>addRepository:

27 createRepository

  | repo |

  repo := self project createRepository: self.

  ^ MCRepositoryGroup default repositories

    detect: [ :each | each = repo ]

    ifNone: [ 

      MCRepositoryGroup default addRepository: repo.

      repo ] in MetacelloRepositorySpec>>createRepository

I think this is because the image doesn’t support Metacello? As in:

Metacello new
    repository: 'filetree://../src';
    baseline: 'Lambda';
    load.

How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)

I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.

Tim

I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src'; load."

 

./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"

Can you test it too?

-- Pavel

-- 

Guille Polito Research Engineer French National Center for Scientific Research - http://www.cnrs.fr Web: http://guillep.github.io Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank" class="">+33 06 52 70 66 13

-- 

Mariano
http://marianopeck.wordpress.com

-- 

Mariano
http://marianopeck.wordpress.com

--

Mariano
http://marianopeck.wordpress.com

Run the following script:

Metacello new
    baseline: 'MetacelloGitBasedRepository';
    repository: '<a href="filetree://../bootstrap" class="">filetree://../bootstrap';
    load.

“Replace missing traits"
Behavior compiledMethodAt: #fuelIgnoredInstanceVariableNames
    ifPresent: [ :m | logger cr; nextPutAll: 'Trait method is present' ]
    ifAbsent: [
        logger cr; nextPutAll: 'Recompile missing traits ...'.

        Behavior
            compile: 'fuelIgnoredInstanceVariableNames
                       "Indicates which variables have to be ignored during serialization."

                       ^#()';
            compile: 'fuelNew
                       "Answer an instance of mine in which serialized references will be injected."

                       ^ self basicNew';
            compile: 'fuelNew: sizeRequested
                       "Answer an instance of mine in which serialized references will be injected."

                       ^ self basicNew: sizeRequested'.

        "Ignore this one"
        "Class
            compile: 'fuelAccept: aGeneralMapper
                       ^self explicitRequirement.'."

        ClassDescription
            compile: 'instanceVariableNamesDo: anUnaryBlock
                       "This is part of the interface between the compiler and a class''s instance or field names.
                        The class should enumerate anUnaryBlock with the instance variable name strings.  The order is important. Names evaluated later will override the
                        same names occurring earlier."

                       | superInstSize |
                       (superInstSize := self superclass notNil ifTrue: [self superclass instSize] ifFalse: [0]) > 0 ifTrue:
                          [self superclass instanceVariableNamesDo: anUnaryBlock].
                       1 to: self instSize - superInstSize do:
                          [:i| anUnaryBlock value: (self instVarNames at: i)]'.
    ].

“Add a tacky Pharo7Platform"
Smalltalk at: #FLPharo7Platform ifAbsent: [
    logger cr; nextPutAll: '>Creating FLPharo7Platform'.
    (Smalltalk at: #FLPharo6Platform) subclass: #FLPharo7Platform.
    (Smalltalk at: #FLPharo6Platform) class compile: 'isResponsibleForCurrentPlatform
            ^true'.
].


logger cr; nextPutAll: 'Evaluating post load initialization...'.
(Smalltalk at: #FLPlatform) current addHacks.

On 4 Aug 2017, at 14:24, Mariano Martinez Peck <[hidden email]> wrote:

On Fri, Aug 4, 2017 at 9:51 AM, Tim Mackinnon <[hidden email]> wrote:

HURRAH - I got it to work, and Fuel out an error to S3 and read it back into a fat 6.1 image with a debug stack (which is so cool!).

COOOL!!! I want to see your blogpost once you make it!!! 

Congratulations, this was a very cool thread to follow :)

 

My footprint size is still a bit larger than I would like (31mb - image of 22.7mb - of which 1mb if fuel and 7mb is S3 and XMLParser support) - so I guess I need to find ways to trim down some of the addons (and any additional image size reductions would be helpful - if there are any to be had).

Is there any easy tools/scripts to explore a headless image and figure out what takes the most size (maybe at a package level?). But definitely this has got me into a good place for experimentation.

I know this is a bit offtopic... but maybe to give you some perspective... For my PhD, one of the things I did (lets say the closest to the PhD topic) was called Marea [1]   (I can also send you the slides and the PhD defense video on youtube if you want). In Marea what I did was to modify the VM to implement a very basic (with lots of limitations) object usage tracking...I simply flagged when an object was used and cleared the flag every in a while. Then, at image side, I would find graphs of unused objects, and replaced the boundary objects with proxies using Ghost proxies (note that an anused object is not the same as unreferenced object,...GC does nothing here). The graphs where then serialized with Fuel. Finally, if those graphs happened to be needed, then the proxy would intercept the message, materialize from Fuel, and plug back the original graph.

Anyway... all of what was to said that I some point I did an experiment. I was able to already proxify and serialize classes, methods, etc. So I took the whole image and I swapped out all classes and their instances (each class with its instances in a different graph) but only a really small core. This image was a Seaside image running DBX Pier website.  So I swapped out everything, and then I lazily started to navigate DBX website, causing the needed graphs to be swapped in. I was able to naviagat all webapp and use it perfectly. I even saved the image.  And all this email is to say that such an image was 3MB.  

Cheers,

[1] https://www.slideshare.net/MarianoMartinezPeck/thesis-presentation-14987645

 

 

 

For the record I had to do the following to get things working:

Define a new baseline:

baseline: spec
   <baseline>

   spec for: #common do: [
      spec 
         package: 'Metacello-GitBasedRepository';
         package: 'Metacello-GitHub';
          package: 'FuelPlatform-Core';
            package: 'FuelPlatform-Pharo-Core';
            package: 'FuelPlatform-Pharo-06';
            package: 'Fuel'.      
   ].    

Run the following script:

Metacello new
    baseline: 'MetacelloGitBasedRepository';
    repository: 'filetree://../bootstrap';
    load.

“Replace missing traits"
Behavior compiledMethodAt: #fuelIgnoredInstanceVariableNames
    ifPresent: [ :m | logger cr; nextPutAll: 'Trait method is present' ]
    ifAbsent: [
        logger cr; nextPutAll: 'Recompile missing traits ...'.

        Behavior
            compile: 'fuelIgnoredInstanceVariableNames
                       "Indicates which variables have to be ignored during serialization."

                       ^#()';
            compile: 'fuelNew
                       "Answer an instance of mine in which serialized references will be injected."

                       ^ self basicNew';
            compile: 'fuelNew: sizeRequested
                       "Answer an instance of mine in which serialized references will be injected."

                       ^ self basicNew: sizeRequested'.

        "Ignore this one"
        "Class
            compile: 'fuelAccept: aGeneralMapper
                       ^self explicitRequirement.'."

        ClassDescription
            compile: 'instanceVariableNamesDo: anUnaryBlock
                       "This is part of the interface between the compiler and a class''s instance or field names.
                        The class should enumerate anUnaryBlock with the instance variable name strings.  The order is important. Names evaluated later will override the
                        same names occurring earlier."

                       | superInstSize |
                       (superInstSize := self superclass notNil ifTrue: [self superclass instSize] ifFalse: [0]) > 0 ifTrue:
                          [self superclass instanceVariableNamesDo: anUnaryBlock].
                       1 to: self instSize - superInstSize do:
                          [:i| anUnaryBlock value: (self instVarNames at: i)]'.
    ].

“Add a tacky Pharo7Platform"
Smalltalk at: #FLPharo7Platform ifAbsent: [
    logger cr; nextPutAll: '>Creating FLPharo7Platform'.
    (Smalltalk at: #FLPharo6Platform) subclass: #FLPharo7Platform.
    (Smalltalk at: #FLPharo6Platform) class compile: 'isResponsibleForCurrentPlatform
            ^true'.
].


logger cr; nextPutAll: 'Evaluating post load initialization...'.
(Smalltalk at: #FLPlatform) current addHacks.

On 4 Aug 2017, at 12:48, Mariano Martinez Peck <[hidden email]> wrote:

On Fri, Aug 4, 2017 at 8:42 AM, Tim Mackinnon <[hidden email]> wrote:

Sadly - false call, that was my logging message - so I’m stumped why these traits aren’t being compiled when I load in Fuel? 

However I have figured out why I get the BigEndian error - this is because the platform is now 7 and so the FlPharo06Platform is not being identified and the more generic FlPharoPlatform is then the default and it doesn’t implement #isBigEndian.

Max, should we do something to the Fuel Platform for Pharo 7?

 

Tim

On 4 Aug 2017, at 12:20, Tim Mackinnon <[hidden email]> wrote:

I’ve spotted something interesting in my logs - when I added one of the missing methods to ClassDescription - I see in the log the message:

Loaded -> Fuel-cypress.1 --- filetree:///builds/macta/PharoLambda/bootstrap --- cache
...finished baseline
Compiling missing traits...
Evaluating post load initialization...
Finished Script.

However when I remove all the compiles I don’t see this message:

Loaded -> Fuel-cypress.1 --- filetree:///builds/macta/PharoLambda/bootstrap --- cache
...finished baseline
Evaluating post load initialization...
Finished Script.

So I wonder if this is similar to the problem I had in the previous 6.0 image - where some things didn’t seem to compile properly (in that case, it was loading a missing class that didn’t seem to cause recompilation).

Tim

On 4 Aug 2017, at 11:10, Tim Mackinnon <[hidden email]> wrote:

Actually there is something weird going on - as now I’m getting a complaint about "FLPharoPlatform had the subclass responsibility to implement #isBigEndian”, and that’s in the FuelPlatform-Pharo-06 which you said to include (and it looks like I am).

I need to double check this - as I must be making some trivial mistake.

Tim

On 4 Aug 2017, at 11:01, Tim Mackinnon <[hidden email]> wrote:

Hi Mariano - I’ve looked at this again, and it was my fault - I was compiling in :

fuelAccept: aGeneralMapper

^self explicitRequirement.

Because when I looked at the Traits for fuel - this is what it had for TClass? - as I’m not so clear on traits, what does this mean - as I had just assumed that I should flatten the Fuel methods for TCLass, TBehaviour and TClassDescription - but I think I’ve misunderstood what I should do (and I guess from Pavel’s comment - I should really look at why that initial method was missing as it sounds like it should have worked  - so maybe its an order loading thing?).

Thanks, for helping by the way - this is proving very instructional.

Tim

On 3 Aug 2017, at 18:26, Mariano Martinez Peck <[hidden email]> wrote:

On Thu, Aug 3, 2017 at 2:14 PM, Tim Mackinnon <[hidden email]> wrote:

Hi Mariano - I compiled in the trait methods manually and it got further - but now I’m seeing another error - Error: Explicitly required method",

    "\u001b[0mLambda class(Object)>>error:",

    "explicitRequirement”,

mmmm the stack is weird... it looks like if Class >> fuelAccept: would have the wrong implementation, that is, having the one from TClass:

fuelAccept: aGeneralMapper

^self explicitRequirement.

Rather than the correct one:

fuelAccept: aGeneralMapper

^aGeneralMapper visitClass: self

Maybe the solution is to compile all those methods from traits BUT NOT those that are "^self explicitRequirement." because those will override the good methods.

If that doesn't work, then maybe can you print to console the each impl (source) of each implementor of #fuelAccept: ?

Thoughts?

 

Any ideas?

Tim

"--- RUNNING ERROR HANDLER ---",
    "Error: Explicitly required method",
    "",
    "\u001b[0m\u001b[31mError: Explicitly required method",
    "\u001b[0mLambda class(Object)>>error:",
    "explicitRequirement",
    "  self error: 'No decompiler available' in Lambda class(Object)>>explicitRequirement in Block: explicitRequirement...",
    "False>>ifFalse:",
    "Lambda class(Object)>>explicitRequirement",
    "Lambda class(Class)>>fuelAccept:",
    "FLLightGeneralMapper>>mapAndTrace:",
    "FLLightGlobalMapper>>mapAndTrace:",
    "FLPluggableSubstitutionMapper>>mapAndTrace:",
    "FLAnalysis>>mapAndTrace:",
    "FLAnalysis>>run",
    "setDefaultAnalysis",
    "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",
    "FLAnalyzer>>analysisFor:",
    "FLSerialization>>analysisStep",
    "FLSerialization>>run",
    "setDefaultSerialization",
    "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",
    "serialize: t1 on: t2",
    "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",
    "on: t1 globalEnvironment: t2 do: t3",
    "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",
    "BlockClosure>>ensure:",
    "FLEncoder class>>on:globalEnvironment:do:",
    "FLSerializer>>serialize:on:",
    "NonInteractiveUIManager>>writeFuelContext:",
    "Lambda class>>processDebug:",
    "Lambda class>>processJSON:",
    "Lambda class>>process:",
    "activate",

--

Mariano
http://marianopeck.wordpress.com