[squeak-dev] etoys performance problems

>
> But thats not the end of it Etoys-wise. 'Cause it turns out an Etoys
> project file is gzipped twice. First the individual file in
> 'writeForExportWithSources:inDirectory:' and then the containing
> function writeForExportWithSources:inDirectory: as part of the .pr
> bundle. And this for a file that already has lots of it's contents
> compressed, seeing that most of the project file is taken up by jpegs.
> And this would ideally also be the case for sound files, but more on
> that later. The size decrease due to zipping in general is not more
> than 10%. So my question on this topic is: is there an argument for
> retaining the double zipping? And would there be a general need for a
> 'just save an uncompressed .pr file, instead of a bundle'
> option/patch?

> Sound format
>
> Another strategy to trimming down the image size is using a compressed
> sound format. At the moment we use wav for sound right? If I'm wrong,
> excuse me, but I haven't had time to investigate the matter to
> thourougly. Anyways, since a number of activities have quite a few
> samples in them, ranging from short utterings to long sentences, it
> would be very worth while if we could use a compressed format. From
> the mailinglist I read rumours about Ogg plugin and a gstreamer
> plugin... Is it already possible to use Ogg files in Etoys? If so how?
> I'd already be happy with general directions.

> He lists (crossposting to Etoys and Squeak dev due to relevance for both),
>
>  One can say that I am sort of taking over from Luke (Gorrie) as OLE
>  Nepal's Etoys performance optimizer (optimizator?). And I'd like to
>  give a status update, but I would especially like some advice on some
>  issues.
>
>  And the sooner the better. Cause we at OLE Nepal are in dire straits;
>  we've got major performance issues, some of which are Etoys related,
>  and some of which are Squeak related. And the reason for the speed is
>  that we're gonna have to have a working XO build in a few days cause
>  we have to train teachers, and in te beginning of April we're gonna
>  flash a build to the machines of the children of the pilot schools,
>  who will use it in the classrooms. So without further ado, except for
>  this redundant sentence:
>
>
>  project loading
>
>  To start with the known, I've been looking into the project loading
>  times, and have managed to cut them in half by removing the gzipping
>  on the project files and just using the .pr file in stead of the zip
>  which claims to be a .pr. Now this might of course not be the perfect
>  solution for all, I did uncover some practices on which I'd like
>  either the opinion of Squeak people in general or that of the Etoys
>  people in particular.
>
>  First, I noticed (together with Luke, who showed me the debugging
>  ropes) that in the method asUnzippedStream, on ReadWriteStream, the
>  method upToEndWithProgressBar uses '/' an awful lot to give the the
>  right approximation to the progress bar. It sucked about 20 percent of
>  the total loading time of an activity. So I changed
>  upToEndWithProgressBar for upToEnd. Which seems like a more reasonable
>  default 'cause I'd guess that usually one won't actually use the
>  progress bar functionality.
>
>  But thats not the end of it Etoys-wise. 'Cause it turns out an Etoys
>  project file is gzipped twice. First the individual file in
>  'writeForExportWithSources:inDirectory:' and then the containing
>  function writeForExportWithSources:inDirectory: as part of the .pr
>  bundle. And this for a file that already has lots of it's contents
>  compressed, seeing that most of the project file is taken up by jpegs.
>  And this would ideally also be the case for sound files, but more on
>  that later. The size decrease due to zipping in general is not more
>  than 10%. So my question on this topic is: is there an argument for
>  retaining the double zipping? And would there be a general need for a
>  'just save an uncompressed .pr file, instead of a bundle'
>  option/patch?
>
>
>  project deletion/memory growth
>
>  Our second issue is related to memory growth within the Etoys image
>  that's causing problems for the, let us say, memory-challenged XO.
>  First of all we've got a, say, root project, which enumerates the
>  activities which it reads from a few directories. In that root project
>  we've got a script running that deletes child projects. Or it should
>  anyway. The key method here is okToChange, which i guess is a bit of a
>  misnomer. That is it's got a bunch of code in it that should remove a
>  project and it's content from the image, but it doesn't. To make it
>  concrete: we've got this script running in the root project:
>
>  unloadActivitiesToFreeSpace
>         Project current children
>                 do: [:t1 | t1 okToChange] "or okToChangeSilently, but okToChange is
>  nice for debugging"
>
>  But after removing a project, executing 'Project allInstances' shows
>  that there is still a reference to the project and when one keeps an
>  eye on the memory usage (with an OSX utility), we see that no memory
>  is deallocated. So the image grows and grows.
>
>  So my concrete question is: how can one for once and for all delete a
>  project instance. I was hoping to at least find some generic
>  deleteInstance method in the image, but I couldn't find one (I'm a bit
>  of a Squeak newbie).
>
>  Then it seems to me that the image grows quite fast in general. Is
>  there some known garbage collection problem concerning Squeak or Etoys
>  that I should know about? And related; when I'm gonna try and trim
>  down the image size: any suggestions on where to start? Is there any
>  known fat waiting to be caved off?
>
>
>  Sound format
>
>  Another strategy to trimming down the image size is using a compressed
>  sound format. At the moment we use wav for sound right? If I'm wrong,
>  excuse me, but I haven't had time to investigate the matter to
>  thourougly. Anyways, since a number of activities have quite a few
>  samples in them, ranging from short utterings to long sentences, it
>  would be very worth while if we could use a compressed format. From
>  the mailinglist I read rumours about Ogg plugin and a gstreamer
>  plugin... Is it already possible to use Ogg files in Etoys? If so how?
>  I'd already be happy with general directions.
>
>
>  Well that's perhaps a bit more than quite enough questions for one
>  post. I'll save some for later. It goes without saying that any advice
>  on these topics would be greatly appreciated.
>
>  /Ties
>  _______________________________________________
>  Etoys mailing list
>  [hidden email]
>  http://lists.laptop.org/listinfo/etoys
>

> He lists (crossposting to Etoys and Squeak dev due to relevance for both),
>
> One can say that I am sort of taking over from Luke (Gorrie) as OLE
> Nepal's Etoys performance optimizer (optimizator?). And I'd like to
> give a status update, but I would especially like some advice on some
> issues.
>
> And the sooner the better. Cause we at OLE Nepal are in dire straits;
> we've got major performance issues, some of which are Etoys related,
> and some of which are Squeak related. And the reason for the speed is
> that we're gonna have to have a working XO build in a few days cause
> we have to train teachers, and in te beginning of April we're gonna
> flash a build to the machines of the children of the pilot schools,
> who will use it in the classrooms. So without further ado, except for
> this redundant sentence:
>
>
> project loading
>
> To start with the known, I've been looking into the project loading
> times, and have managed to cut them in half by removing the gzipping
> on the project files and just using the .pr file in stead of the zip
> which claims to be a .pr. Now this might of course not be the perfect
> solution for all, I did uncover some practices on which I'd like
> either the opinion of Squeak people in general or that of the Etoys
> people in particular.
>
> First, I noticed (together with Luke, who showed me the debugging
> ropes) that in the method asUnzippedStream, on ReadWriteStream, the
> method upToEndWithProgressBar uses '/' an awful lot to give the the
> right approximation to the progress bar. It sucked about 20 percent of
> the total loading time of an activity. So I changed
> upToEndWithProgressBar for upToEnd. Which seems like a more reasonable
> default 'cause I'd guess that usually one won't actually use the
> progress bar functionality.
>
> But thats not the end of it Etoys-wise. 'Cause it turns out an Etoys
> project file is gzipped twice. First the individual file in
> 'writeForExportWithSources:inDirectory:' and then the containing
> function writeForExportWithSources:inDirectory: as part of the .pr
> bundle. And this for a file that already has lots of it's contents
> compressed, seeing that most of the project file is taken up by jpegs.
> And this would ideally also be the case for sound files, but more on
> that later. The size decrease due to zipping in general is not more
> than 10%. So my question on this topic is: is there an argument for
> retaining the double zipping? And would there be a general need for a
> 'just save an uncompressed .pr file, instead of a bundle'
> option/patch?
>  

>
> project deletion/memory growth
>
> Our second issue is related to memory growth within the Etoys image
> that's causing problems for the, let us say, memory-challenged XO.
> First of all we've got a, say, root project, which enumerates the
> activities which it reads from a few directories. In that root project
> we've got a script running that deletes child projects. Or it should
> anyway. The key method here is okToChange, which i guess is a bit of a
> misnomer. That is it's got a bunch of code in it that should remove a
> project and it's content from the image, but it doesn't. To make it
> concrete: we've got this script running in the root project:
>
> unloadActivitiesToFreeSpace
> Project current children
> do: [:t1 | t1 okToChange] "or okToChangeSilently, but okToChange is
> nice for debugging"
>
> But after removing a project, executing 'Project allInstances' shows
> that there is still a reference to the project and when one keeps an
> eye on the memory usage (with an OSX utility), we see that no memory
> is deallocated. So the image grows and grows.
>
> So my concrete question is: how can one for once and for all delete a
> project instance. I was hoping to at least find some generic
> deleteInstance method in the image, but I couldn't find one (I'm a bit
> of a Squeak newbie).
>
> Then it seems to me that the image grows quite fast in general. Is
> there some known garbage collection problem concerning Squeak or Etoys
> that I should know about? And related; when I'm gonna try and trim
> down the image size: any suggestions on where to start? Is there any
> known fat waiting to be caved off?
>
>
> Sound format
>
> Another strategy to trimming down the image size is using a compressed
> sound format. At the moment we use wav for sound right? If I'm wrong,
> excuse me, but I haven't had time to investigate the matter to
> thourougly. Anyways, since a number of activities have quite a few
> samples in them, ranging from short utterings to long sentences, it
> would be very worth while if we could use a compressed format. From
> the mailinglist I read rumours about Ogg plugin and a gstreamer
> plugin... Is it already possible to use Ogg files in Etoys? If so how?
> I'd already be happy with general directions.
>  

> Sound format
>
> Another strategy to trimming down the image size is using a compressed
> sound format. At the moment we use wav for sound right? If I'm wrong,
> excuse me, but I haven't had time to investigate the matter to
> thourougly. Anyways, since a number of activities have quite a few
> samples in them, ranging from short utterings to long sentences, it
> would be very worth while if we could use a compressed format. From
> the mailinglist I read rumours about Ogg plugin and a gstreamer
> plugin... Is it already possible to use Ogg files in Etoys? If so how?
> I'd already be happy with general directions.
>  

>  See comment above about not zipping JPEGS/image data that is
>  included in the ZIP file.   We choose to store media (sound/video) in
>  a separate
>  media folder because it was pointless to unpack from a zip file on
>  need. However
>  this might not be possible when you want all the data in the zip file,
>  still you could
>  alter the code so that sound/video files aren't compressed by the zip
>  logic when
>  they are added to the zip file.

>  http://tinlizzie.org/olpc/OggPlugin/
>
>  I am currently working on a gstreamer plugin for OLPC via funding from
>  Viewpoints Research Institute,
>  and have a target completion date for the end of the month.  This
>  plugin will let you build
>  gstreamer pipeline chains and run them, right now you can build a
>  pipleline to decode  audio/video ogg files for
>  example and send the video to a X11 window and the audio to the
>  hardware sound player.
>
>  I also hope to have 'sinks' which let you import audio and video into
>  Squeak.

> On Wed, Mar 26, 2008 at 12:43 AM, Hilaire Fernandes <[hidden email]> wrote:
>> Hello Tie,
>>
>>  Your email is very interesting.
>>  And I am wondering: does jpeg file inserted in a Squeak image then
>>  saved in a .pr file are saved in the .pr file as jpeg data or just
>>  24bits RBG form data?
>>  I am enclined to think this is the second option, if so the size
>>  impact is hudge.
>
> Hmm, yes interesting question. I didn't investigate, but judging from
> the amount of pictures compared to the file size, I guessed they were
> stored compressed in one way or another. Perhaps somebody with
> knowledge could shed some light?

> Ties Stuij wrote:
>  > On Wed, Mar 26, 2008 at 12:43 AM, Hilaire Fernandes <[hidden email]> wrote:
>  >> Hello Tie,
>  >>
>  >>  Your email is very interesting.
>  >>  And I am wondering: does jpeg file inserted in a Squeak image then
>  >>  saved in a .pr file are saved in the .pr file as jpeg data or just
>  >>  24bits RBG form data?
>  >>  I am enclined to think this is the second option, if so the size
>  >>  impact is hudge.
>  >
>  > Hmm, yes interesting question. I didn't investigate, but judging from
>  > the amount of pictures compared to the file size, I guessed they were
>  > stored compressed in one way or another. Perhaps somebody with
>  > knowledge could shed some light?
>
>
> It's compressed. When you load a form via Form>>fromFileNamed: the
>  project's resource manager will remember the original bits (the jpeg
>  file). Only if the reference gets lost (the file deleted, the image
>  moved) it will use the uncompressed bits.

>
> As a concrete simple example, how can I examine the content of this
> project file http://swiki.ofset.org:8000/super/36 (produced with a 3.8
> image)?
>
> It use 3 graphics and I am really curious to know what inside the .pr file.
>
> The info I got from the .pr is:
> hilaire@tice:/tmp$ file ENGRENAGE4.001.pr
> ENGRENAGE4.001.pr: gzip compressed data, from FAT filesystem (MS-DOS, OS/2, NT)
>
> Once gunzipped, I can't examine the content...
>  

> Hilaire Fernandes schrieb:
>>
>> As a concrete simple example, how can I examine the content of this
>> project file http://swiki.ofset.org:8000/super/36 (produced with a  
>> 3.8
>> image)?
>>
>> It use 3 graphics and I am really curious to know what inside  
>> the .pr file.
>>
>> The info I got from the .pr is:
>> hilaire@tice:/tmp$ file ENGRENAGE4.001.pr
>> ENGRENAGE4.001.pr: gzip compressed data, from FAT filesystem (MS-
>> DOS, OS/2, NT)
>>
>> Once gunzipped, I can't examine the content...
>>
> Probably the file command is confused about this file.
> .pr files are just ZIP files, so you should be able to unzip the  
> file to
> see its contents.

> Ties Stuij wrote:
>> On Wed, Mar 26, 2008 at 12:43 AM, Hilaire Fernandes <[hidden email]
>> > wrote:
>>> Hello Tie,
>>>
>>> Your email is very interesting.
>>> And I am wondering: does jpeg file inserted in a Squeak image then
>>> saved in a .pr file are saved in the .pr file as jpeg data or just
>>> 24bits RBG form data?
>>> I am enclined to think this is the second option, if so the size
>>> impact is hudge.
>>
>> Hmm, yes interesting question. I didn't investigate, but judging from
>> the amount of pictures compared to the file size, I guessed they were
>> stored compressed in one way or another. Perhaps somebody with
>> knowledge could shed some light?
>
> It's compressed. When you load a form via Form>>fromFileNamed: the
> project's resource manager will remember the original bits (the jpeg
> file). Only if the reference gets lost (the file deleted, the image
> moved) it will use the uncompressed bits.

>>> With your project growing,  you may want to stop using .pr file and
>>> use external file to describe the activity (xml files) plus media  
>>> data
>>> (jpeg, ogg, etc.). I am betting you will see a hudge performence
>>> boost. But then you are cut from Etoys..
>>>
>>> I am currious to read Squeak friends advices.
>>
>> As am I,
>
> I would not recommend going down this path if you're new to Squeak and
> don't know what you are buying into. The .pr files work because half a
> dozen people spent a couple of years to make all of this stuff work.
> You'd be pretty much on your own recreating this effort so unless you
> have sufficient resources just try to deal with the inefficiencies and
> leave the architecture alone.

> As a concrete simple example, how can I examine the content of this
> project file http://swiki.ofset.org:8000/super/36 (produced with a 3.8
> image)?
>
> It use 3 graphics and I am really curious to know what inside  
> the .pr file.
>
> The info I got from the .pr is:
> hilaire@tice:/tmp$ file ENGRENAGE4.001.pr
> ENGRENAGE4.001.pr: gzip compressed data, from FAT filesystem (MS-
> DOS, OS/2, NT)
>
> Once gunzipped, I can't examine the content...

> On Wed, Mar 26, 2008 at 12:43 AM, Hilaire Fernandes  
> <[hidden email]> wrote:
>> Hello Tie,
>>
>> Your email is very interesting.
>> And I am wondering: does jpeg file inserted in a Squeak image then
>> saved in a .pr file are saved in the .pr file as jpeg data or just
>> 24bits RBG form data?
>> I am enclined to think this is the second option, if so the size
>> impact is hudge.
>
> Hmm, yes interesting question. I didn't investigate, but judging from
> the amount of pictures compared to the file size, I guessed they were
> stored compressed in one way or another. Perhaps somebody with
> knowledge could shed some light?

> project deletion/memory growth
>
> Our second issue is related to memory growth within the Etoys image
> that's causing problems for the, let us say, memory-challenged XO.
> First of all we've got a, say, root project, which enumerates the
> activities which it reads from a few directories. In that root project
> we've got a script running that deletes child projects. Or it should
> anyway. The key method here is okToChange, which i guess is a bit of a
> misnomer. That is it's got a bunch of code in it that should remove a
> project and it's content from the image, but it doesn't. To make it
> concrete: we've got this script running in the root project:
>
> unloadActivitiesToFreeSpace
> Project current children
> do: [:t1 | t1 okToChange] "or okToChangeSilently, but okToChange is
> nice for debugging"
>
> But after removing a project, executing 'Project allInstances' shows
> that there is still a reference to the project and when one keeps an
> eye on the memory usage (with an OSX utility), we see that no memory
> is deallocated. So the image grows and grows.