Loads of Pharo development process questions

I'm having trouble wrapping my head around how the Pharo development/
maintenance process is supposed to work.
Most of these seem to be missing info on the wiki which merely need  
fleshing out there, I would update the wiki pages myself if only I  
knew what to write...

Please bear in mind that I'm a newbie to Squeak, so some of these  
questions may be rather naive.
One of my motivations is to evaluate the future viability of Pharo as  
a vehicle for commercial applications and whether it warrants to lobby  
for resources being allocated to its development.



So here goes:

The wiki mentions 'kernel' and 'MiniImage' as not being in scope of  
Pharo.
What does that entail, does this mean that discussions about allowing  
nil in Sets are out of scope?



What is the status of the dev-image? As delivered the tools are broken  
(because context menus on the source pane raise an UHE that is induced  
by OB-Enhancements, issue 904), is it actively used by the main  
committers or is the core image the place to be?
Even more confusingly the above issue has been marked as fixed this  
night but if I do a World->System->Software Update the bug persists in  
my image.



When I looked into fixing it myself I got confused (because I looked  
at InstallScript.st), it seems the code comes from various  
repositories -- not a single Pharo respository/branch. Are these  
various repositories Pharo-specific and maintained over Pharo-Inbox or  
do I need to submit fixes to different places?
I guess as a Squeak newbie I need a high-level overview on the wiki on  
how code and fixes flow within Pharo, towards Squeak and towards the  
various tool maintainers (and back to Pharo of course, the upstream/
downstream story). What is the story when such tools and Pharo become  
incompatible and need to fork (or are they already forked)?



Is there a vision on long-term support?
Pharo wants to be agile and be free to change at will (IMO a good  
decision) but if it is to be viable at my job there need to be  
versions that receive long-term support (the 'professional' aspect of  
Pharo).
At least bug fixes (and hopefully speed enhancements) need to be  
ported back to an 'LTS' release for quite a while (I'm thinking three  
to four years here) and on a consistent/reliable basis if Pharo is to  
be viable for the kind of projects I work on. Porting our projects  
(and project specific tool enhancements) forwards turns out to be very  
expensive (even in a 'conservative' environment like VisualWorks) so a  
mechanism needs to be in place to avoid needing too many porting  
moments over the lifetime of our projects (think 10+ years).


The process of updating the dev image is excruciatingly slow, is this  
normal?
The process seems to be CPU bound and takes ages for what I consider a  
small amount of changes - in its current state it would be unusable  
for the process we use at work, it will need to be brought back to  
well under a minute as opposed to the twenty (or so) minutes it took  
me yesterday. What is the background  and vision for the future on  
this issue?


Let me finish with a quick and easy newbie question:
where do I find the profiler tools for Pharo?



Thanks!

Reinout
-------


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Reinout,

The profiling tools are exposed through MessageTally.

        http://onsmalltalk.com/com/profiling-smalltalk

should get you started.

Please note that Pharo is not even yet in beta, so some of the snags you mention will no doubt get fixed as the feature list finalizes.  As an open source project, there will be a certain amount of bias toward what is important to the heavy contributors.  I hope those will include organizations such as yours to push Pharo in the direction of a stable performance-oriented environment on top of which a mix of commercial and academic projects will flourish.

Re slow updates, I am not sure what to expect, but Pharo's response times in general are of concern to me, and there are efforts getting under way to find out what is wrong (PackageInfo has been marked as a suspect a couple of times), what is realistic, and what can be done about it.  

Bill


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Reinout Heeck
Sent: Sunday, July 05, 2009 6:36 AM
To: [hidden email]
Subject: [Pharo-project] Loads of Pharo development process questions

I'm having trouble wrapping my head around how the Pharo development/ maintenance process is supposed to work.
Most of these seem to be missing info on the wiki which merely need fleshing out there, I would update the wiki pages myself if only I knew what to write...

Please bear in mind that I'm a newbie to Squeak, so some of these questions may be rather naive.
One of my motivations is to evaluate the future viability of Pharo as a vehicle for commercial applications and whether it warrants to lobby for resources being allocated to its development.



So here goes:

The wiki mentions 'kernel' and 'MiniImage' as not being in scope of Pharo.
What does that entail, does this mean that discussions about allowing nil in Sets are out of scope?



What is the status of the dev-image? As delivered the tools are broken (because context menus on the source pane raise an UHE that is induced by OB-Enhancements, issue 904), is it actively used by the main committers or is the core image the place to be?
Even more confusingly the above issue has been marked as fixed this night but if I do a World->System->Software Update the bug persists in my image.



When I looked into fixing it myself I got confused (because I looked at InstallScript.st), it seems the code comes from various repositories -- not a single Pharo respository/branch. Are these various repositories Pharo-specific and maintained over Pharo-Inbox or do I need to submit fixes to different places?
I guess as a Squeak newbie I need a high-level overview on the wiki on how code and fixes flow within Pharo, towards Squeak and towards the various tool maintainers (and back to Pharo of course, the upstream/ downstream story). What is the story when such tools and Pharo become incompatible and need to fork (or are they already forked)?



Is there a vision on long-term support?
Pharo wants to be agile and be free to change at will (IMO a good
decision) but if it is to be viable at my job there need to be versions that receive long-term support (the 'professional' aspect of Pharo).
At least bug fixes (and hopefully speed enhancements) need to be ported back to an 'LTS' release for quite a while (I'm thinking three to four years here) and on a consistent/reliable basis if Pharo is to be viable for the kind of projects I work on. Porting our projects (and project specific tool enhancements) forwards turns out to be very expensive (even in a 'conservative' environment like VisualWorks) so a mechanism needs to be in place to avoid needing too many porting moments over the lifetime of our projects (think 10+ years).


The process of updating the dev image is excruciatingly slow, is this normal?
The process seems to be CPU bound and takes ages for what I consider a small amount of changes - in its current state it would be unusable for the process we use at work, it will need to be brought back to well under a minute as opposed to the twenty (or so) minutes it took me yesterday. What is the background  and vision for the future on this issue?


Let me finish with a quick and easy newbie question:
where do I find the profiler tools for Pharo?



Thanks!

Reinout
-------


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Hi reinout


> Please bear in mind that I'm a newbie to Squeak, so some of these
> questions may be rather naive.

We welcome newbie like you :)

ok this is old.
this was by reference to the effort of pavel to create a minimal kernel.
We did not want to stress that the only effort of pharo was to have a  
small kernel.
Our effort is to have a modular, clean, tested smalltalk (which will  
probably have a small kernel).

> What is the status of the dev-image? As delivered the tools are broken
> (because context menus on the source pane raise an UHE that is induced
> by OB-Enhancements, issue 904), is it actively used by the main
> committers or is the core image the place to be?

The dev image is alpha like pharo.
Some of the OB code changes under the feet of david. He promised us to  
fix that.

> Even more confusingly the above issue has been marked as fixed this
> night but if I do a World->System->Software Update the bug persists in
> my image.

the update is only working for the core because OB is not part of  
pharo but an add-on.
And the add-on are not managed by the pharo stream of changes but by  
an update of
the projects in the pharo Universe (yes this is complex but we do not  
want that people can
say that we are hijacking squeak tools).


> When I looked into fixing it myself I got confused (because I looked
> at InstallScript.st), it seems the code comes from various
> repositories -- not a single Pharo respository/branch. Are these
> various repositories Pharo-specific and maintained over Pharo-Inbox or
> do I need to submit fixes to different places?


ok
for pharo code
we have
        Pharo (our components)
        PharoInbox (for people to post)
        PharoTaskForces (for effort related to pharo but not ready yet)
        PharoTreatedInbox (once we clean the inbox we do not delete but move  
to an archive).


> I guess as a Squeak newbie I need a high-level overview on the wiki on
> how code and fixes flow within Pharo, towards Squeak and towards the
> various tool maintainers (and back to Pharo of course, the upstream/
> downstream story). What is the story when such tools and Pharo become
> incompatible and need to fork (or are they already forked)?

Ok here it is:
        - when a fix for pharo arrives via the inbox
        - we treat it
                -> reject with reason
                -> integrate it in the pharo core stream
        - when a fix arrives for a tool we cannot do too much
        besides asking that the maintainer push it into his code
                -> this implies that some tools may fork.


> Is there a vision on long-term support?
You mean are they pharoers that you can pay to maintain and provide  
support?
Yes I imagine that you can pay john mcintosh, mike rueger and some  
others to
offer support.

> Pharo wants to be agile and be free to change at will (IMO a good
> decision) but if it is to be viable at my job there need to be
> versions that receive long-term support (the 'professional' aspect of
> Pharo).

Yes
Normally with Squeak, severe bugs and selected one where retrofitted
in old versions. This way seaside 3.8 users got important fixes when  
we did 3.9.

> At least bug fixes (and hopefully speed enhancements) need to be
> ported back to an 'LTS' release for quite a while (I'm thinking three
> to four years here) and on a consistent/reliable basis if Pharo is to
> be viable for the kind of projects I work on.

Since all the code is and will be available either people will have to  
do it
or they should pay for it. We will be able to do the critical ones only.


> Porting our projects  (and project specific tool enhancements)  
> forwards turns out to be very
> expensive (even in a 'conservative' environment like VisualWorks) so a
> mechanism needs to be in place to avoid needing too many porting
> moments over the lifetime of our projects (think 10+ years).


> The process of updating the dev image is excruciatingly slow, is this
> normal?

Yes the complete code of the package is sent, parsed, compared.....
We do not have bytecode and the packageInfo used by Monticello is
really slow since it simulates packages by scanning the complete  
systemOrganisation
all the time. I started to write a real package class with pointers to  
replace that.
But I do that on my spare time.

> The process seems to be CPU bound and takes ages for what I consider a
> small amount of changes - in its current state it would be unusable
> for the process we use at work, it will need to be brought back to
> well under a minute as opposed to the twenty (or so) minutes it took
> me yesterday. What is the background  and vision for the future on
> this issue?

The vision is bytecode loader + real package + probably using  
monticello 2
when it will be ready.

> Let me finish with a quick and easy newbie question:
> where do I find the profiler tools for Pharo?

I like this question. We need more of profiler and fixes.

MessageTally
SpaceTally
bench
millisecondsToRun:

Reinout. We know that the road will be long and we do not have engineers
but we will do it small steps at a time and see if we succeed.

stef




_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

> Please note that Pharo is not even yet in beta, so some of the snags  
> you mention will no doubt get fixed as the feature list finalizes.  
> As an open source project, there will be a certain amount of bias  
> toward what is important to the heavy contributors.  I hope those  
> will include organizations such as yours to push Pharo in the  
> direction of a stable performance-oriented environment on top of  
> which a mix of commercial and academic projects will flourish.

Yes, yes, yes :-)
This is why I showed up here, at the moment it is just me probing  
around in my private time but the motivation is to lobby for  
considering its use at my work.
I assume there will be loads of additions required before we can even  
use it in the small there and clearly such additions won't just appear  
unless we contribute heavily towards them ourselves. (Off the cuff I'm  
thinking of requirements like full SOAP(all versions!) with support  
tools, multiple windows, COM interfacing etc, etc, but I need to cut  
my teeth on the basics first before I want to start discussing such  
things here).

Baby steps will do just fine ;-)


Thanks!

Reinout
-------

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Hi Stef!

Thanks for the in-depth answers, some remarks/further questions:


> the update is only working for the core because OB is not part of
> pharo but an add-on.
> And the add-on are not managed by the pharo stream of changes but by
> an update of
> the projects in the pharo Universe (yes this is complex but we do not
> want that people can
> say that we are hijacking squeak tools).

The motivation is fair, but the current solution seems untenable...

I think we should consider cribbing from the model that Linux distros  
use:

Load (more precisely: use -- we might not be able to load it) a /
specific/ version of the Squeak tool, and then apply a separate (Pharo-
maintained) set of patches to make it Pharo compliant.
Define a process to specify how bug fixes are to be pushed upstream,  
and how enhancements are pulled downstream -- reassessing the Pharo  
patches each time a new version is pulled downstream.

This way the tool is never hijacked but Pharo has a clear story on how  
it remains stable/unbreakable.
If pushing bug fixes upstream is supported directly by tools we might  
even be able to make the argument that Pharo is helping the tool  
builders make better Squeak tools ;-)


>> Pharo wants to be agile and be free to change at will (IMO a good
>> decision) but if it is to be viable at my job there need to be
>> versions that receive long-term support (the 'professional' aspect of
>> Pharo).
>
> Yes
> Normally with Squeak, severe bugs and selected one where retrofitted
> in old versions. This way seaside 3.8 users got important fixes when
> we did 3.9.

It depends on how often Pharo wants to do a 'stable' release, if that  
is once every two years I don't see a problem with this model.
But if that happens more often we will need to maintain a lot of older  
releases to make them viable for commercial use, thus spreading our  
resources thin. Hence my desire to pick only select releases that are  
designated to receive 'long term support' so bugfix back-porting  
efforts are concentrated on only one or two older releases to keep  
those alive for say 3+ years.




[...for version management]
> The vision is bytecode loader

Pleas don't go there. Byte-code loading is a very nice-to-have in  
deployment scenarios (lazy plugin loading, live update delivery to  
deployed apps etc) but if there is one thing that using VW's Store has  
taught me it is that it definitely should not be part of a code  
repository that is used for development code version management.

In the case of VW+Store standardizing on using only source-loading has  
served us well, it is definitely fast enough.
Building our 'refactoring' image that holds all our projects (almost  
100 MB image -- just code) takes less than an hour which is quite  
alright in a scenario where one uses nightly builds and updates the  
resulting image during the day-time development cycle. I suggest to  
get source-loading/diffing et al up to speed first.




Cheers!

Reinout
-------


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Hi Reinout

> What is the status of the dev-image? As delivered the tools are broken  
> (because context menus on the source pane raise an UHE that is induced  
> by OB-Enhancements, issue 904), is it actively used by the main  
> committers or is the core image the place to be?
> Even more confusingly the above issue has been marked as fixed this  
> night but if I do a World->System->Software Update the bug persists in  
> my image.

You can get the fix by executing 'ScriptLoader loadOBAlpha' in a workspace.

See Stephane's answer why this fix is not provided by 'software update'.

David

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Yes
>
> I think we should consider cribbing from the model that Linux distros
> use:
>
> Load (more precisely: use -- we might not be able to load it) a /
> specific/ version of the Squeak tool, and then apply a separate  
> (Pharo-
> maintained) set of patches to make it Pharo compliant.

but only the developer of the tool can do that.

> Define a process to specify how bug fixes are to be pushed upstream,
> and how enhancements are pulled downstream -- reassessing the Pharo
> patches each time a new version is pulled downstream.
>
> This way the tool is never hijacked but Pharo has a clear story on how
> it remains stable/unbreakable.
> If pushing bug fixes upstream is supported directly by tools we might
> even be able to make the argument that Pharo is helping the tool
> builders make better Squeak tools ;-)

:)

>>>
>>
>> Yes
>> Normally with Squeak, severe bugs and selected one where retrofitted
>> in old versions. This way seaside 3.8 users got important fixes when
>> we did 3.9.
>
> It depends on how often Pharo wants to do a 'stable' release, if that
> is once every two years I don't see a problem with this model.

we could see that. It also depends on the pace of changes. because
Pharo 1.0 got a lot of deep changes like closures. I do not expect  
that huge changes in
the future.

> But if that happens more often we will need to maintain a lot of older
> releases to make them viable for commercial use, thus spreading our
> resources thin. Hence my desire to pick only select releases that are
> designated to receive 'long term support' so bugfix back-porting
> efforts are concentrated on only one or two older releases to keep
> those alive for say 3+ years.

Yes I see.

> [...for version management]
>> The vision is bytecode loader
>
> Pleas don't go there. Byte-code loading is a very nice-to-have in
> deployment scenarios (lazy plugin loading, live update delivery to
> deployed apps etc) but if there is one thing that using VW's Store has
> taught me it is that it definitely should not be part of a code
> repository that is used for development code version management.

This is in VW store is bad. I do not see why a package MCZ could not  
ship with
a byte-code synchronized version of the source. Then we could have a  
stripper
or a synchronizer that recompile the source code and produce new  
bytecode.

> In the case of VW+Store standardizing on using only source-loading has
> served us well, it is definitely fast enough.
> Building our 'refactoring' image that holds all our projects (almost
> 100 MB image -- just code) takes less than an hour which is quite
> alright in a scenario where one uses nightly builds and updates the
> resulting image during the day-time development cycle. I suggest to
> get source-loading/diffing et al up to speed first.

we will start by fixing packageInfo

stef

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Hi Reinout,

Just to say - I'm in pretty similar position as you, trying to use
Squeak and friends for my commercial work (which is still on VisualWorks
for now), that's why I really like the progress Pharoers did so far and
I'm supporting them as much as I can.

Which does not mean I stopped supporting Squeak, as I like any good
progress in any Smalltalk. We need to stand by the guys doing great work
on a tools which we "commercial" guys then use for a living :)

Best regards
Janko

Reinout Heeck pravi:

--
Janko Mivšek
AIDA/Web
Smalltalk Web Application Server
http://www.aidaweb.si

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

may be the future is that a small company offers pharo support.
But it will depend on you :)

Stef


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

On Jul 6, 2009, at 10:49 AM, David Röthlisberger wrote:
>
> You can get the fix by executing 'ScriptLoader loadOBAlpha' in a  
> workspace.
>

Yo,  thanks -- I'll start all this studying this week ;-)


R
-


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

No, quite the opposite:

The Pharo maintainers maintain the Pharo compliance patches for every  
tool we import, the tool developer should not be bothered with these  
-- should not even see them.

Only a subset of these patches -- the bugfixes created in the Pharo  
community -- should be pushed up to the tool developer.



This way both worlds could get the best of both worlds:

The tool developers still get bugfixes posted despite their original  
implementation possibly being/becoming incompatible with Pharo.

The Pharo community gets the freedom to become incompatible with  
Squeak without loosing the benefits of ongoing tool development on  
Squeak.


I am not saying this is the only solution, just describing what is a  
'field proven' model in the Linux community which seems to perfectly  
implement the stated goals of Pharo.




In my experience it 'poisons' the repository with packages that load  
fine as bytecode but fail to load as source.
Mostly in 'auto brain surgery' scenarios where the /source/ should  
contain more load ordering hints (but never attains that level because  
the binary version hides these bugs).

I reiterate that VW shows that source loading is fast enough -- even  
for rather large updates.

So in my opinion 'fast enough' source loading should be attained  
first, and bin-loading should be added later as an option (and  
probably not in the code repository but perhaps in a cache in front of  
such a repository, or maybe sideways as part of a build process).

>
> we will start by fixing packageInfo

:-) right!



R
-


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

>>
>
> No, quite the opposite:
>
> The Pharo maintainers maintain the Pharo compliance patches for every
> tool we import, the tool developer should not be bothered with these
> -- should not even see them.

I agree but it means that other pharoers should stand up and sign for  
that
because I'm full and will probably continue to be.

Ok so this is a good news.

> So in my opinion 'fast enough' source loading should be attained
> first, and bin-loading should be added later as an option (and
> probably not in the code repository but perhaps in a cache in front of
> such a repository, or maybe sideways as part of a build process).

yes this is also an alternative.


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

On Mon, Jul 6, 2009 at 12:18 AM, Reinout Heeck<[hidden email]> wrote:
> Load (more precisely: use -- we might not be able to load it) a /
> specific/ version of the Squeak tool, and then apply a separate (Pharo-
> maintained) set of patches to make it Pharo compliant.
> Define a process to specify how bug fixes are to be pushed upstream,
> and how enhancements are pulled downstream -- reassessing the Pharo
> patches each time a new version is pulled downstream.

This is something that works very well in Linux and I think we could
use it. What we need to make this happen:

1) a repository of patched packages that can be easily loaded inside user images
2) a repository of patches that can be applied to tools in order to
make them compatible with Pharo
3) a database which associates a package with a list of ordered
patches to apply on it
4) a tool to populate the first repository with automatically created
packages containing the patches. That tool should be able to apply
patches without even load the package in the image (because there
might be undefined references or overrides in non-existing classes...)


What we have:

1) any Monticello repository would do (e.g., SqueakSource)
2) any ftp/http directory would do (like we do with the current update
stream 'pharo.gforge.inria.fr')
3) it could be a directory per package (named with the name of the
package) containing all patches for that package (named with a number
indicating the order of patches)
4) that one should be written. Possible steps: unzipping the mcz,
applying the patch to the source, creating a new mcz, pushing that mcz
to the repository 1).

What do you think?

--
Damien Cassou
http://damiencassou.seasidehosting.st

"Lambdas are relegated to relative obscurity until Java makes them
popular by not having them." James Iry

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

would be interesting to give a try.
Now I'm concerned by our resources.
Stef

On Jul 7, 2009, at 10:50 AM, Damien Cassou wrote:


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project