File/Stream changes: one Integer decoder/encoder to rule them all

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

File/Stream changes: one Integer decoder/encoder to rule them all

Sven Van Caekenberghe-2
Hi,

After the File and Stream changes in Pharo 7, a binary read, resp. write stream from/to a file is actually a ZnBuffered(Read|Write)Stream on a BinaryFileStream. You access these using #binary(Read|Write)Stream[Do:] sent to a FileReference.

As minimal streams the API of ZnBuffered(Read|Write)Stream was different from what existed before.

Specifically, a number of Integer decoding/encoding methods were missing. It is probably best to add those (an alternative would be a subclass).

When I looked at what was available, I thought I could improve upon the current situation. In fact, I think all existing ones can be written in terms of just one key method, one for reading and one for writing. The existing methods than become simple aliases, all while offering more functionality.

(Incidentally this would be an ideal use of a Trait, can we use them again/still ?)

The key method is #nextIntegerOfSize: numberOfBytes signed: signed bigEndian: bigEndian [put: value] and can be found in the latest version of Zinc-CharacterEncoding-Core with a comprehensive set of unit tests.

For example,

int32
  ^ self nextIntegerOfSize: 4 signed: true bigEndian: true

nextWord
  ^ self nextIntegerOfSize: 2 signed: false bigEndian: true

or non-aliases ones like

  binaryStream nextIntegerOfSize: 3 signed: true bigEndian: false

I think I nailed it, but I would love a second pair of eyes to check what I did.

The same functionality could be added to ByteArray in a slight variation (with offsets), where similar integer encoding/decoding methods exist.

Sven
Reply | Threaded
Open this post in threaded view
|

Re: File/Stream changes: one Integer decoder/encoder to rule them all

Stephane Ducasse-3
I should that I love twice the title of your email.
- yes if we can have one good instead of multiple not so good. This is
wonderful.
Now I'm not expert in that and brain dead after car, flight, rer,
metro, train to get back home


On Fri, Apr 20, 2018 at 5:24 PM, Sven Van Caekenberghe <[hidden email]> wrote:

> Hi,
>
> After the File and Stream changes in Pharo 7, a binary read, resp. write stream from/to a file is actually a ZnBuffered(Read|Write)Stream on a BinaryFileStream. You access these using #binary(Read|Write)Stream[Do:] sent to a FileReference.
>
> As minimal streams the API of ZnBuffered(Read|Write)Stream was different from what existed before.
>
> Specifically, a number of Integer decoding/encoding methods were missing. It is probably best to add those (an alternative would be a subclass).
>
> When I looked at what was available, I thought I could improve upon the current situation. In fact, I think all existing ones can be written in terms of just one key method, one for reading and one for writing. The existing methods than become simple aliases, all while offering more functionality.
>
> (Incidentally this would be an ideal use of a Trait, can we use them again/still ?)
>
> The key method is #nextIntegerOfSize: numberOfBytes signed: signed bigEndian: bigEndian [put: value] and can be found in the latest version of Zinc-CharacterEncoding-Core with a comprehensive set of unit tests.
>
> For example,
>
> int32
>   ^ self nextIntegerOfSize: 4 signed: true bigEndian: true
>
> nextWord
>   ^ self nextIntegerOfSize: 2 signed: false bigEndian: true
>
> or non-aliases ones like
>
>   binaryStream nextIntegerOfSize: 3 signed: true bigEndian: false
>
> I think I nailed it, but I would love a second pair of eyes to check what I did.
>
> The same functionality could be added to ByteArray in a slight variation (with offsets), where similar integer encoding/decoding methods exist.
>
> Sven

Reply | Threaded
Open this post in threaded view
|

Re: File/Stream changes: one Integer decoder/encoder to rule them all

Tudor Girba-2
In reply to this post by Sven Van Caekenberghe-2
Hi,

I am so looking forward to playing with this (as a user).

For various reasons, we are still in Pharo 6.1, but encoding/decoding was always a troublesome issue in Pharo.

Thanks a lot!

Cheers,
Doru


> On Apr 20, 2018, at 5:24 PM, Sven Van Caekenberghe <[hidden email]> wrote:
>
> Hi,
>
> After the File and Stream changes in Pharo 7, a binary read, resp. write stream from/to a file is actually a ZnBuffered(Read|Write)Stream on a BinaryFileStream. You access these using #binary(Read|Write)Stream[Do:] sent to a FileReference.
>
> As minimal streams the API of ZnBuffered(Read|Write)Stream was different from what existed before.
>
> Specifically, a number of Integer decoding/encoding methods were missing. It is probably best to add those (an alternative would be a subclass).
>
> When I looked at what was available, I thought I could improve upon the current situation. In fact, I think all existing ones can be written in terms of just one key method, one for reading and one for writing. The existing methods than become simple aliases, all while offering more functionality.
>
> (Incidentally this would be an ideal use of a Trait, can we use them again/still ?)
>
> The key method is #nextIntegerOfSize: numberOfBytes signed: signed bigEndian: bigEndian [put: value] and can be found in the latest version of Zinc-CharacterEncoding-Core with a comprehensive set of unit tests.
>
> For example,
>
> int32
>  ^ self nextIntegerOfSize: 4 signed: true bigEndian: true
>
> nextWord
>  ^ self nextIntegerOfSize: 2 signed: false bigEndian: true
>
> or non-aliases ones like
>
>  binaryStream nextIntegerOfSize: 3 signed: true bigEndian: false
>
> I think I nailed it, but I would love a second pair of eyes to check what I did.
>
> The same functionality could be added to ByteArray in a slight variation (with offsets), where similar integer encoding/decoding methods exist.
>
> Sven

--
www.tudorgirba.com
www.feenk.com

"From an abstract enough point of view, any two things are similar."





Reply | Threaded
Open this post in threaded view
|

Re: File/Stream changes: one Integer decoder/encoder to rule them all

Sven Van Caekenberghe-2


> On 22 Apr 2018, at 10:22, Tudor Girba <[hidden email]> wrote:
>
> Hi,
>
> I am so looking forward to playing with this (as a user).
>
> For various reasons, we are still in Pharo 6.1, but encoding/decoding was always a troublesome issue in Pharo.

Well it should not be, not even in older Pharo versions, as long as you use everything correctly.

But the latest changes are a bold step forward.

> Thanks a lot!
>
> Cheers,
> Doru
>
>
>> On Apr 20, 2018, at 5:24 PM, Sven Van Caekenberghe <[hidden email]> wrote:
>>
>> Hi,
>>
>> After the File and Stream changes in Pharo 7, a binary read, resp. write stream from/to a file is actually a ZnBuffered(Read|Write)Stream on a BinaryFileStream. You access these using #binary(Read|Write)Stream[Do:] sent to a FileReference.
>>
>> As minimal streams the API of ZnBuffered(Read|Write)Stream was different from what existed before.
>>
>> Specifically, a number of Integer decoding/encoding methods were missing. It is probably best to add those (an alternative would be a subclass).
>>
>> When I looked at what was available, I thought I could improve upon the current situation. In fact, I think all existing ones can be written in terms of just one key method, one for reading and one for writing. The existing methods than become simple aliases, all while offering more functionality.
>>
>> (Incidentally this would be an ideal use of a Trait, can we use them again/still ?)
>>
>> The key method is #nextIntegerOfSize: numberOfBytes signed: signed bigEndian: bigEndian [put: value] and can be found in the latest version of Zinc-CharacterEncoding-Core with a comprehensive set of unit tests.
>>
>> For example,
>>
>> int32
>> ^ self nextIntegerOfSize: 4 signed: true bigEndian: true
>>
>> nextWord
>> ^ self nextIntegerOfSize: 2 signed: false bigEndian: true
>>
>> or non-aliases ones like
>>
>> binaryStream nextIntegerOfSize: 3 signed: true bigEndian: false
>>
>> I think I nailed it, but I would love a second pair of eyes to check what I did.
>>
>> The same functionality could be added to ByteArray in a slight variation (with offsets), where similar integer encoding/decoding methods exist.
>>
>> Sven
>
> --
> www.tudorgirba.com
> www.feenk.com
>
> "From an abstract enough point of view, any two things are similar."
>
>
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: File/Stream changes: one Integer decoder/encoder to rule them all

Stephane Ducasse-3
Thanks a lot Sven. This is great to see all this positive energy to
improve this part.
From a core library stand point, streams were the last pieces that
needed to be revisited.
This is great to see this happening.

Stef

On Sun, Apr 22, 2018 at 11:04 AM, Sven Van Caekenberghe <[hidden email]> wrote:

>
>
>> On 22 Apr 2018, at 10:22, Tudor Girba <[hidden email]> wrote:
>>
>> Hi,
>>
>> I am so looking forward to playing with this (as a user).
>>
>> For various reasons, we are still in Pharo 6.1, but encoding/decoding was always a troublesome issue in Pharo.
>
> Well it should not be, not even in older Pharo versions, as long as you use everything correctly.
>
> But the latest changes are a bold step forward.
>
>> Thanks a lot!
>>
>> Cheers,
>> Doru
>>
>>
>>> On Apr 20, 2018, at 5:24 PM, Sven Van Caekenberghe <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> After the File and Stream changes in Pharo 7, a binary read, resp. write stream from/to a file is actually a ZnBuffered(Read|Write)Stream on a BinaryFileStream. You access these using #binary(Read|Write)Stream[Do:] sent to a FileReference.
>>>
>>> As minimal streams the API of ZnBuffered(Read|Write)Stream was different from what existed before.
>>>
>>> Specifically, a number of Integer decoding/encoding methods were missing. It is probably best to add those (an alternative would be a subclass).
>>>
>>> When I looked at what was available, I thought I could improve upon the current situation. In fact, I think all existing ones can be written in terms of just one key method, one for reading and one for writing. The existing methods than become simple aliases, all while offering more functionality.
>>>
>>> (Incidentally this would be an ideal use of a Trait, can we use them again/still ?)
>>>
>>> The key method is #nextIntegerOfSize: numberOfBytes signed: signed bigEndian: bigEndian [put: value] and can be found in the latest version of Zinc-CharacterEncoding-Core with a comprehensive set of unit tests.
>>>
>>> For example,
>>>
>>> int32
>>> ^ self nextIntegerOfSize: 4 signed: true bigEndian: true
>>>
>>> nextWord
>>> ^ self nextIntegerOfSize: 2 signed: false bigEndian: true
>>>
>>> or non-aliases ones like
>>>
>>> binaryStream nextIntegerOfSize: 3 signed: true bigEndian: false
>>>
>>> I think I nailed it, but I would love a second pair of eyes to check what I did.
>>>
>>> The same functionality could be added to ByteArray in a slight variation (with offsets), where similar integer encoding/decoding methods exist.
>>>
>>> Sven
>>
>> --
>> www.tudorgirba.com
>> www.feenk.com
>>
>> "From an abstract enough point of view, any two things are similar."
>>
>>
>>
>>
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: File/Stream changes: one Integer decoder/encoder to rule them all

Tudor Girba-2
That’s exactly my feeling as well.

Doru

> On Apr 22, 2018, at 6:41 PM, Stephane Ducasse <[hidden email]> wrote:
>
> Thanks a lot Sven. This is great to see all this positive energy to
> improve this part.
> From a core library stand point, streams were the last pieces that
> needed to be revisited.
> This is great to see this happening.
>
> Stef
>
> On Sun, Apr 22, 2018 at 11:04 AM, Sven Van Caekenberghe <[hidden email]> wrote:
>>
>>
>>> On 22 Apr 2018, at 10:22, Tudor Girba <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> I am so looking forward to playing with this (as a user).
>>>
>>> For various reasons, we are still in Pharo 6.1, but encoding/decoding was always a troublesome issue in Pharo.
>>
>> Well it should not be, not even in older Pharo versions, as long as you use everything correctly.
>>
>> But the latest changes are a bold step forward.
>>
>>> Thanks a lot!
>>>
>>> Cheers,
>>> Doru
>>>
>>>
>>>> On Apr 20, 2018, at 5:24 PM, Sven Van Caekenberghe <[hidden email]> wrote:
>>>>
>>>> Hi,
>>>>
>>>> After the File and Stream changes in Pharo 7, a binary read, resp. write stream from/to a file is actually a ZnBuffered(Read|Write)Stream on a BinaryFileStream. You access these using #binary(Read|Write)Stream[Do:] sent to a FileReference.
>>>>
>>>> As minimal streams the API of ZnBuffered(Read|Write)Stream was different from what existed before.
>>>>
>>>> Specifically, a number of Integer decoding/encoding methods were missing. It is probably best to add those (an alternative would be a subclass).
>>>>
>>>> When I looked at what was available, I thought I could improve upon the current situation. In fact, I think all existing ones can be written in terms of just one key method, one for reading and one for writing. The existing methods than become simple aliases, all while offering more functionality.
>>>>
>>>> (Incidentally this would be an ideal use of a Trait, can we use them again/still ?)
>>>>
>>>> The key method is #nextIntegerOfSize: numberOfBytes signed: signed bigEndian: bigEndian [put: value] and can be found in the latest version of Zinc-CharacterEncoding-Core with a comprehensive set of unit tests.
>>>>
>>>> For example,
>>>>
>>>> int32
>>>> ^ self nextIntegerOfSize: 4 signed: true bigEndian: true
>>>>
>>>> nextWord
>>>> ^ self nextIntegerOfSize: 2 signed: false bigEndian: true
>>>>
>>>> or non-aliases ones like
>>>>
>>>> binaryStream nextIntegerOfSize: 3 signed: true bigEndian: false
>>>>
>>>> I think I nailed it, but I would love a second pair of eyes to check what I did.
>>>>
>>>> The same functionality could be added to ByteArray in a slight variation (with offsets), where similar integer encoding/decoding methods exist.
>>>>
>>>> Sven
>>>
>>> --
>>> www.tudorgirba.com
>>> www.feenk.com
>>>
>>> "From an abstract enough point of view, any two things are similar."
>>>
>>>
>>>
>>>
>>>
>>
>>
>

--
www.tudorgirba.com
www.feenk.com

"There are no old things, there are only old ways of looking at them."





Reply | Threaded
Open this post in threaded view
|

Re: File/Stream changes: one Integer decoder/encoder to rule them all

Guillermo Polito
In reply to this post by Sven Van Caekenberghe-2


On Fri, Apr 20, 2018 at 5:24 PM, Sven Van Caekenberghe <[hidden email]> wrote:
Hi,

After the File and Stream changes in Pharo 7, a binary read, resp. write stream from/to a file is actually a ZnBuffered(Read|Write)Stream on a BinaryFileStream. You access these using #binary(Read|Write)Stream[Do:] sent to a FileReference.

As minimal streams the API of ZnBuffered(Read|Write)Stream was different from what existed before.

Specifically, a number of Integer decoding/encoding methods were missing. It is probably best to add those (an alternative would be a subclass).

When I looked at what was available, I thought I could improve upon the current situation. In fact, I think all existing ones can be written in terms of just one key method, one for reading and one for writing. The existing methods than become simple aliases, all while offering more functionality.

(Incidentally this would be an ideal use of a Trait, can we use them again/still ?)

The key method is #nextIntegerOfSize: numberOfBytes signed: signed bigEndian: bigEndian [put: value] and can be found in the latest version of Zinc-CharacterEncoding-Core with a comprehensive set of unit tests.

For example,

int32
  ^ self nextIntegerOfSize: 4 signed: true bigEndian: true

nextWord
  ^ self nextIntegerOfSize: 2 signed: false bigEndian: true

or non-aliases ones like

  binaryStream nextIntegerOfSize: 3 signed: true bigEndian: false

I think I nailed it, but I would love a second pair of eyes to check what I did.

Sorry I arrive late.

I'm thinking that this is only valid for binary streams, putting it in buffered stream overloads the API... We can leave with it but... Two thoughts that come to my mind about this:

 - first, when we are decoding like that from a stream, we are probably parsing a binary format, so we want in general to have utility methods to read different sizes. Maybe we could include a Reader class with methods such as
   * readSignedInt32
   * readUnsignedInt32
   * readSignedInt64
   * readUnsignedInt64
   * readFloat64

... and so on?

- Second, that Reader may be (or not) a new decorator alternative. ZnBinaryDecoder? Wrapping a binary (buffered or not) stream. And this would not overload the API ^^.

What do you think?


The same functionality could be added to ByteArray in a slight variation (with offsets), where similar integer encoding/decoding methods exist.

Sven



--

   

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - http://www.cnrs.fr


Web: http://guillep.github.io

Phone: +33 06 52 70 66 13

Reply | Threaded
Open this post in threaded view
|

Re: File/Stream changes: one Integer decoder/encoder to rule them all

Nicolas Cellier


2018-05-14 13:22 GMT+02:00 Guillermo Polito <[hidden email]>:


On Fri, Apr 20, 2018 at 5:24 PM, Sven Van Caekenberghe <[hidden email]> wrote:
Hi,

After the File and Stream changes in Pharo 7, a binary read, resp. write stream from/to a file is actually a ZnBuffered(Read|Write)Stream on a BinaryFileStream. You access these using #binary(Read|Write)Stream[Do:] sent to a FileReference.

As minimal streams the API of ZnBuffered(Read|Write)Stream was different from what existed before.

Specifically, a number of Integer decoding/encoding methods were missing. It is probably best to add those (an alternative would be a subclass).

When I looked at what was available, I thought I could improve upon the current situation. In fact, I think all existing ones can be written in terms of just one key method, one for reading and one for writing. The existing methods than become simple aliases, all while offering more functionality.

(Incidentally this would be an ideal use of a Trait, can we use them again/still ?)

The key method is #nextIntegerOfSize: numberOfBytes signed: signed bigEndian: bigEndian [put: value] and can be found in the latest version of Zinc-CharacterEncoding-Core with a comprehensive set of unit tests.

For example,

int32
  ^ self nextIntegerOfSize: 4 signed: true bigEndian: true

nextWord
  ^ self nextIntegerOfSize: 2 signed: false bigEndian: true

or non-aliases ones like

  binaryStream nextIntegerOfSize: 3 signed: true bigEndian: false

I think I nailed it, but I would love a second pair of eyes to check what I did.

Sorry I arrive late.

I'm thinking that this is only valid for binary streams, putting it in buffered stream overloads the API... We can leave with it but... Two thoughts that come to my mind about this:

 - first, when we are decoding like that from a stream, we are probably parsing a binary format, so we want in general to have utility methods to read different sizes. Maybe we could include a Reader class with methods such as
   * readSignedInt32
   * readUnsignedInt32
   * readSignedInt64
   * readUnsignedInt64
   * readFloat64

... and so on?

- Second, that Reader may be (or not) a new decorator alternative. ZnBinaryDecoder? Wrapping a binary (buffered or not) stream. And this would not overload the API ^^.

What do you think?

Hi Guillermo,
wouldn't a specialized wrapper work for adding this protocol where required?
For ByteArray, I didn't check recently in Pharo, but I know that it's quite a mess in Squeak because some methods are in Alien, other in FFI, other in base Squeak...
IMO all methods should be in core Squeak/Pharo because of general usage (like decoding binary data).




The same functionality could be added to ByteArray in a slight variation (with offsets), where similar integer encoding/decoding methods exist.

Sven



--

   

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - http://www.cnrs.fr


Web: http://guillep.github.io

Phone: +33 06 52 70 66 13


Reply | Threaded
Open this post in threaded view
|

Re: File/Stream changes: one Integer decoder/encoder to rule them all

Marcus Denker-4
>
>
> For ByteArray, I didn't check recently in Pharo, but I know that it's quite a mess in Squeak because some methods are in Alien, other in FFI, other in base Squeak...
> IMO all methods should be in core Squeak/Pharo because of general usage (like decoding binary data).
>
I want to move the methods from FFI to the base since years… I was told that this is impossible to ever happen because it is not important to do changes like that.

        Marcus
Reply | Threaded
Open this post in threaded view
|

Re: File/Stream changes: one Integer decoder/encoder to rule them all

Nicolas Cellier


2018-05-14 14:44 GMT+02:00 Marcus Denker <[hidden email]>:
>
>
> For ByteArray, I didn't check recently in Pharo, but I know that it's quite a mess in Squeak because some methods are in Alien, other in FFI, other in base Squeak...
> IMO all methods should be in core Squeak/Pharo because of general usage (like decoding binary data).
>
I want to move the methods from FFI to the base since years… I was told that this is impossible to ever happen because it is not important to do changes like that.
 
        Marcus

Hi Marcus,
we all know that those un-important and not-so-much-rewarding changes makes a big difference at the end.
like commenting classes, classifying methods, deprecating duplicates, reducing package inter-dependencies, etc...
I thought that the defenders of statu quo had two reasons in mind:
- the will to keep a FFI-less (locked-in) image
- the fear to loose backward compatibility
But's it the past, now you are free :)


Reply | Threaded
Open this post in threaded view
|

Re: File/Stream changes: one Integer decoder/encoder to rule them all

Marcus Denker-4


On 14 May 2018, at 15:09, Nicolas Cellier <[hidden email]> wrote:



2018-05-14 14:44 GMT+02:00 Marcus Denker <[hidden email]>:
>
>
> For ByteArray, I didn't check recently in Pharo, but I know that it's quite a mess in Squeak because some methods are in Alien, other in FFI, other in base Squeak...
> IMO all methods should be in core Squeak/Pharo because of general usage (like decoding binary data).
>
I want to move the methods from FFI to the base since years… I was told that this is impossible to ever happen because it is not important to do changes like that.
 
        Marcus

Hi Marcus,
we all know that those un-important and not-so-much-rewarding changes makes a big difference at the end.
like commenting classes, classifying methods, deprecating duplicates, reducing package inter-dependencies, etc...
I thought that the defenders of statu quo had two reasons in mind:
- the will to keep a FFI-less (locked-in) image
- the fear to loose backward compatibility

The idea that “only big important changes” are worth doing… you know, it’s “inventing the future” business, not “cleaning up”!

(Of course revolutionary changes while staying 100% backward compatible… because think about the Children!)

It would be funny if it would not be sad…

But's it the past, now you are free :)

Not really, because we share that package with Squeak. If I would be free to change it, it would have happened a loong time ago…

Marcus

Reply | Threaded
Open this post in threaded view
|

Re: File/Stream changes: one Integer decoder/encoder to rule them all

Nicolas Cellier


2018-05-14 15:15 GMT+02:00 Marcus Denker <[hidden email]>:


On 14 May 2018, at 15:09, Nicolas Cellier <[hidden email]> wrote:



2018-05-14 14:44 GMT+02:00 Marcus Denker <[hidden email]>:
>
>
> For ByteArray, I didn't check recently in Pharo, but I know that it's quite a mess in Squeak because some methods are in Alien, other in FFI, other in base Squeak...
> IMO all methods should be in core Squeak/Pharo because of general usage (like decoding binary data).
>
I want to move the methods from FFI to the base since years… I was told that this is impossible to ever happen because it is not important to do changes like that.
 
        Marcus

Hi Marcus,
we all know that those un-important and not-so-much-rewarding changes makes a big difference at the end.
like commenting classes, classifying methods, deprecating duplicates, reducing package inter-dependencies, etc...
I thought that the defenders of statu quo had two reasons in mind:
- the will to keep a FFI-less (locked-in) image
- the fear to loose backward compatibility

The idea that “only big important changes” are worth doing… you know, it’s “inventing the future” business, not “cleaning up”!

(Of course revolutionary changes while staying 100% backward compatible… because think about the Children!)

It would be funny if it would not be sad…

But's it the past, now you are free :)

Not really, because we share that package with Squeak. If I would be free to change it, it would have happened a loong time ago…

Marcus


OK, coordinated changes must be performed then, but it's a bit more difficult...
It can be a two step process:
1) separate the methods to be re-integrated in core into a separate package
2) move this methods in core, or use the separate package for old images (backward compatibility)
It has to be performed both in FFI and Alien.