Pavel's ChangeLog week of 2017-01-01

> On 8 Jan 2018, at 10:30, Pavel Krivanek <[hidden email]> wrote:
>
> In memory sources file
> - I slightly improved the PR for integration of the sources file inside the object memory. Now it will be possible to extract them back to the disk from the menu so everything can work as before. I'm not sure if it will be integrated into Pharo 7 or we will wait for Pharo 8 but it seems to be prepared.
>
> File API and caching
> - because the in-memory sources file do not use deprecated MultiByteFileStream, I played with the new file API and discovered that ZnCharacterReadStream on File is about 10 times slower than the old MultiByteFileStream. That's because ZnCharacterReadStream does not use any cache. For better speed the ZnBufferedReadStream needs to be used like this:
>
> readStream := File openForReadFileNamed: 'file.txt'.
> zn := ZnCharacterReadStream on: readStream encoding: #UTF8.
> buffered := ZnBufferedReadStream on: zn.
> [ buffered atEnd ] whileFalse: [ buffered next: 20000 ].

Yes, that is correct.

> However the ZnBufferedReadStream is by design only a linear stream and the position cannot be changed and thus it cannot be used for *.sources files etc. We will need to prepare a positionable alternative.

This is great work, Pavel, I know Marcus wanted this for a loooooong time ;-)

Now, I thought (naively maybe) that in memory sources meant that each method object holds a single (compressed) string with its own individual source code. In such a system no positioning in a huge stream would (ever) be necessary, no ?

AFAIK, the need for positioning is only to get to the right source bit in this huge pool.

> Ring 2 integration
> - I was working on replacing of the old Ring with the new reimplementation. The old Ring is used on many sensitive places in the system. Mainly in tools however it is used by Monticello and Epicea too so to try to remove it is like to cut a branch below yourself on an undermined tree. The approach of the old and new Ring is in many senses very different and the full compatibility will not be supported. I partly provide compatibility API, partly adopt the code directly.
> - The real system and the model is mixed many times in the users of the old Ring. For example a model for a method is created but then when the package of the method is asked, it is already a real package, not its model. Such places need to be cleaned.
> - I already have an image that has no old Ring code at all but because of some memory leaks caused by the mixtures of the models and real system I was not able to bootstrap it successfully. More work is needed.
> - I discovered that the image has a serious memory leak caused by Iceberg/GT and Announcers but I still need to find more information about it.
>
> Smalltalk archeology
> - during the free days on the beginning of the year I looked at Smalltalk-78. I extracted the data from Lively Kernel so the original image can read from external files, not only the recent updated ones.
> - Then I looked at SqueakJS again and tried to reproduce Craig's experiments with Pharo on it. Generally it is about 100 times slower than native Pharo which makes it hard to use for real-life tasks. However it would be pity to do not provide at least basic attention to it because in some cases it may be really useful.
>
> Cheers,
> -- Pavel
>
>

In memory sources file

- I slightly improved the PR for integration of the sources file inside the object memory. Now it will be possible to extract them back to the disk from the menu so everything can work as before. I'm not sure if it will be integrated into Pharo 7 or we will wait for Pharo 8 but it seems to be prepared.

File API and caching

- because the in-memory sources file do not use deprecated MultiByteFileStream, I played with the new file API and discovered that ZnCharacterReadStream on File is about 10 times slower than the old MultiByteFileStream. That's because ZnCharacterReadStream does not use any cache. For better speed the ZnBufferedReadStream needs to be used like this:

readStream := File openForReadFileNamed: 'file.txt'.

zn := ZnCharacterReadStream on: readStream encoding: #UTF8.

buffered := ZnBufferedReadStream on: zn.

[ buffered atEnd ] whileFalse: [ buffered next: 20000 ].

However the ZnBufferedReadStream is by design only a linear stream and the position cannot be changed and thus it cannot be used for *.sources files etc. We will need to prepare a positionable alternative.

Ring 2 integration

- I was working on replacing of the old Ring with the new reimplementation. The old Ring is used on many sensitive places in the system. Mainly in tools however it is used by Monticello and Epicea too so to try to remove it is like to cut a branch below yourself on an undermined tree. The approach of the old and new Ring is in many senses very different and the full compatibility will not be supported. I partly provide compatibility API, partly adopt the code directly.

- The real system and the model is mixed many times in the users of the old Ring. For example a model for a method is created but then when the package of the method is asked, it is already a real package, not its model. Such places need to be cleaned. 

- I already have an image that has no old Ring code at all but because of some memory leaks caused by the mixtures of the models and real system I was not able to bootstrap it successfully. More work is needed. 

- I discovered that the image has a serious memory leak caused by Iceberg/GT and Announcers but I still need to find more information about it.

Smalltalk archeology

- during the free days on the beginning of the year I looked at Smalltalk-78. I extracted the data from Lively Kernel so the original image can read from external files, not only the recent updated ones.

- Then I looked at SqueakJS again and tried to reproduce Craig's experiments with Pharo on it. Generally it is about 100 times slower than native Pharo which makes it hard to use for real-life tasks. However it would be pity to do not provide at least basic attention to it because in some cases it may be really useful. 

Cheers,

-- Pavel

On Mon, Jan 8, 2018 at 6:30 AM, Pavel Krivanek <[hidden email]> wrote:

In memory sources file

- I slightly improved the PR for integration of the sources file inside the object memory. Now it will be possible to extract them back to the disk from the menu so everything can work as before. I'm not sure if it will be integrated into Pharo 7 or we will wait for Pharo 8 but it seems to be prepared.

File API and caching

- because the in-memory sources file do not use deprecated MultiByteFileStream, I played with the new file API and discovered that ZnCharacterReadStream on File is about 10 times slower than the old MultiByteFileStream. That's because ZnCharacterReadStream does not use any cache. For better speed the ZnBufferedReadStream needs to be used like this:

readStream := File openForReadFileNamed: 'file.txt'.

zn := ZnCharacterReadStream on: readStream encoding: #UTF8.

buffered := ZnBufferedReadStream on: zn.

[ buffered atEnd ] whileFalse: [ buffered next: 20000 ].

However the ZnBufferedReadStream is by design only a linear stream and the position cannot be changed and thus it cannot be used for *.sources files etc. We will need to prepare a positionable alternative.

 

Ring 2 integration

- I was working on replacing of the old Ring with the new reimplementation. The old Ring is used on many sensitive places in the system. Mainly in tools however it is used by Monticello and Epicea too so to try to remove it is like to cut a branch below yourself on an undermined tree. The approach of the old and new Ring is in many senses very different and the full compatibility will not be supported. I partly provide compatibility API, partly adopt the code directly.

Great news. I didn't look at Ring 2 code recently but I say your esug talk and I imagine it will fit well for the change model and UI tool.

What may need attention is serialization to text file. Right now, the Ring 1 object returned by asRingDefinition (+ some tweaks) is serialized and materialized via STON into the .ombu files. With Ring 2 serialized with STON, the file format will be different and in consequence the new epicea won't be able to open a .ombu file generated in Ring 1. Not sure if Pharo community find this acceptable. IMO it is.

- If not-acceptable, there could be some workaround. 

- If acceptable, that's the easiest path, and also it could be a good opportunity to improve the file format: It could be more compact and even have a better extension such as .epicea or something that comunicates that tthose files have code changes or sessions. 

cheers,

Martín

 

- The real system and the model is mixed many times in the users of the old Ring. For example a model for a method is created but then when the package of the method is asked, it is already a real package, not its model. Such places need to be cleaned. 

- I already have an image that has no old Ring code at all but because of some memory leaks caused by the mixtures of the models and real system I was not able to bootstrap it successfully. More work is needed. 

- I discovered that the image has a serious memory leak caused by Iceberg/GT and Announcers but I still need to find more information about it.

Smalltalk archeology

- during the free days on the beginning of the year I looked at Smalltalk-78. I extracted the data from Lively Kernel so the original image can read from external files, not only the recent updated ones.

- Then I looked at SqueakJS again and tried to reproduce Craig's experiments with Pharo on it. Generally it is about 100 times slower than native Pharo which makes it hard to use for real-life tasks. However it would be pity to do not provide at least basic attention to it because in some cases it may be really useful. 

Cheers,

-- Pavel

Hi Pavel

I was thinking that it would be nice to have a simple DB build in
Pharo (I thought that CDB could do the job) to manage source.
Because I like the idea that when the system crashes I only lose at
max the method I'm editing.
Esteban did once a try to connect a tokyo tyrant db now I would prefer
to avoid to have FFI in the middle.
To me we should **********stabilise*********** pharo for real.

Stef

On Mon, Jan 8, 2018 at 4:50 PM, Pavel Krivanek <[hidden email]> wrote:
>
>
> 2018-01-08 16:27 GMT+01:00 Martin Dias <[hidden email]>:
>>
>>
>>
>> On Mon, Jan 8, 2018 at 6:30 AM, Pavel Krivanek <[hidden email]>
>> wrote:
>>>
>>> In memory sources file
>>> - I slightly improved the PR for integration of the sources file inside
>>> the object memory. Now it will be possible to extract them back to the disk
>>> from the menu so everything can work as before. I'm not sure if it will be
>>> integrated into Pharo 7 or we will wait for Pharo 8 but it seems to be
>>> prepared.
>>>
>>> File API and caching
>>> - because the in-memory sources file do not use deprecated
>>> MultiByteFileStream, I played with the new file API and discovered that
>>> ZnCharacterReadStream on File is about 10 times slower than the old
>>> MultiByteFileStream. That's because ZnCharacterReadStream does not use any
>>> cache. For better speed the ZnBufferedReadStream needs to be used like this:
>>>
>>> readStream := File openForReadFileNamed: 'file.txt'.
>>> zn := ZnCharacterReadStream on: readStream encoding: #UTF8.
>>> buffered := ZnBufferedReadStream on: zn.
>>> [ buffered atEnd ] whileFalse: [ buffered next: 20000 ].
>>>
>>> However the ZnBufferedReadStream is by design only a linear stream and
>>> the position cannot be changed and thus it cannot be used for *.sources
>>> files etc. We will need to prepare a positionable alternative.
>>>
>>
>>>
>>> Ring 2 integration
>>> - I was working on replacing of the old Ring with the new
>>> reimplementation. The old Ring is used on many sensitive places in the
>>> system. Mainly in tools however it is used by Monticello and Epicea too so
>>> to try to remove it is like to cut a branch below yourself on an undermined
>>> tree. The approach of the old and new Ring is in many senses very different
>>> and the full compatibility will not be supported. I partly provide
>>> compatibility API, partly adopt the code directly.
>>
>>
>> Great news. I didn't look at Ring 2 code recently but I say your esug talk
>> and I imagine it will fit well for the change model and UI tool.
>>
>> What may need attention is serialization to text file. Right now, the Ring
>> 1 object returned by asRingDefinition (+ some tweaks) is serialized and
>> materialized via STON into the .ombu files. With Ring 2 serialized with
>> STON, the file format will be different and in consequence the new epicea
>> won't be able to open a .ombu file generated in Ring 1. Not sure if Pharo
>> community find this acceptable. IMO it is.
>> - If not-acceptable, there could be some workaround.
>> - If acceptable, that's the easiest path, and also it could be a good
>> opportunity to improve the file format: It could be more compact and even
>> have a better extension such as .epicea or something that comunicates that
>> tthose files have code changes or sessions.
>
>
> I think that the change of the format is not a a practical issue because
> probably no-one will need to read the old Ombu files from a different Pharo
> version.
> But a change of the format will be handy because every Ring2 model is a
> standalone environment and Ombu thus saves all information that it includes.
> So a simple method change creates a record with almost 400 lines. It must be
> of course much slower too.
>
> -- Pavel
>
>>
>> cheers,
>> Martín
>>
>>>
>>> - The real system and the model is mixed many times in the users of the
>>> old Ring. For example a model for a method is created but then when the
>>> package of the method is asked, it is already a real package, not its model.
>>> Such places need to be cleaned.
>>> - I already have an image that has no old Ring code at all but because of
>>> some memory leaks caused by the mixtures of the models and real system I was
>>> not able to bootstrap it successfully. More work is needed.
>>> - I discovered that the image has a serious memory leak caused by
>>> Iceberg/GT and Announcers but I still need to find more information about
>>> it.
>>>
>>> Smalltalk archeology
>>> - during the free days on the beginning of the year I looked at
>>> Smalltalk-78. I extracted the data from Lively Kernel so the original image
>>> can read from external files, not only the recent updated ones.
>>> - Then I looked at SqueakJS again and tried to reproduce Craig's
>>> experiments with Pharo on it. Generally it is about 100 times slower than
>>> native Pharo which makes it hard to use for real-life tasks. However it
>>> would be pity to do not provide at least basic attention to it because in
>>> some cases it may be really useful.
>>>
>>> Cheers,
>>> -- Pavel
>>>
>>>
>>
>