Smalltalk › Gemtalk › GLASS

[Glass] Problems managing files from Pharo and GemStone

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

6 messages Options

Mariano Martinez Peck

[Glass] Problems managing files from Pharo and GemStone

Hi guys. I don't understand why I have just faced this problem now and not before...but... I am writing a text file (sixx serialization) in GemStone this way:

FileStream forceNewFileNamed: 'test' do: [:str | str nextPutAll: 'whateverRootObject' sixxString ].

And then in Pharo I read like:

SixxShapeChangeReadStream on: (FileStream oldFileNamed: 'test')

For some files, the above works OK. But for other files, while reading in Pharo I get a: "UTF8Error: Invalid UTF8 input detected"

The problem is that FileStream oldFileNamed: in Pharo answers a MultiByteFileStream, which uses a UTF8TextConverter. Hence, in UTF8TextConverter >> nextFromStream: I get an error because 'unicode' instVar is nil.

Of course, if in Pharo I use directly a StandardFileStream oldFileNamed: (to avoid using MultiByteFileStream) I can indeed read those files.

So...how should I make this to work in GemStone and Pharo? I know Johan ported the text converters and MultiByteBinaryOrTextStream. But I don't know how to get a similar MultiByteFileStream for GemStone.

Thanks in advance,

--
Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Mariano Martinez Peck

Re: [Glass] Problems managing files from Pharo and GemStone

Ok. It seems that in GemStone GsFile only accepts 0-255 range so I need to pre-encode strings. Something like this:

myfile := GsFile openWrite: self filename.

myfile nextPutAll: 'myRootObject' sixxString encodeAsUTF8.

myfile close.

is this the way that it should be done in GemStone?

Thanks,

On Tue, Apr 15, 2014 at 12:27 PM, Mariano Martinez Peck <[hidden email]> wrote:

Hi guys. I don't understand why I have just faced this problem now and not before...but... I am writing a text file (sixx serialization) in GemStone this way:

FileStream forceNewFileNamed: 'test' do: [:str | str nextPutAll: 'whateverRootObject' sixxString ].

And then in Pharo I read like:

SixxShapeChangeReadStream on: (FileStream oldFileNamed: 'test')

For some files, the above works OK. But for other files, while reading in Pharo I get a: "UTF8Error: Invalid UTF8 input detected"

The problem is that FileStream oldFileNamed: in Pharo answers a MultiByteFileStream, which uses a UTF8TextConverter. Hence, in UTF8TextConverter >> nextFromStream: I get an error because 'unicode' instVar is nil.

Of course, if in Pharo I use directly a StandardFileStream oldFileNamed: (to avoid using MultiByteFileStream) I can indeed read those files.

So...how should I make this to work in GemStone and Pharo? I know Johan ported the text converters and MultiByteBinaryOrTextStream. But I don't know how to get a similar MultiByteFileStream for GemStone.

Thanks in advance,

--
Mariano
http://marianopeck.wordpress.com

--
Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Richard Sargent

Re: [Glass] Problems managing files from Pharo and GemStone

Administrator

Mariano Martinez Peck wrote

Ok. It seems that in GemStone GsFile only accepts 0-255 range so I need to
pre-encode strings. ...

is this the way that it should be done in GemStone?

That is correct. GsFile provides no automatic character encoding.

johnmci

Re: [Glass] Problems managing files from Pharo and GemStone

I'm curious here on os-x file name/paths require normalization with specialization

https://developer.apple.com/library/mac/qa/qa1173/_index.html

I'd expect if you are targeting os-x you'd need to know that a UTF8 string might not work as you expect.

On Tue, Apr 15, 2014 at 9:37 AM, Richard Sargent <[hidden email]> wrote:

Mariano Martinez Peck wrote

> Ok. It seems that in GemStone GsFile only accepts 0-255 range so I need to

> pre-encode strings. ...

>
> is this the way that it should be done in GemStone?

That is correct. GsFile provides no automatic character encoding.

--
View this message in context: http://forum.world.st/Glass-Problems-managing-files-from-Pharo-and-GemStone-tp4754791p4754805.html
Sent from the GLASS mailing list archive at Nabble.com.
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

--
===========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd. Twitter: squeaker68882
===========================================================================

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Mariano Martinez Peck

Re: [Glass] Problems managing files from Pharo and GemStone

In reply to this post by Richard Sargent

On Tue, Apr 15, 2014 at 1:37 PM, Richard Sargent <[hidden email]> wrote:

Mariano Martinez Peck wrote

> Ok. It seems that in GemStone GsFile only accepts 0-255 range so I need to

> pre-encode strings. ...

>
> is this the way that it should be done in GemStone?

That is correct. GsFile provides no automatic character encoding.

mmmm I wonder which is the better way to handle the real stream API.. I mean... if each time I do a #nextPutAll: with a string I need to send #encodeAsUTF8, that is not very fun. I wonder if Johan's text converters or MultiByteBinaryOrTextStream could be used. Ideally we should have a stream API that allow us to transparently encode with certain encoding, right?

Anyway....is it OK to assume FileStream will be able to properly decode a UTF stored? Say I wrote a file like this:

myfile := GsFile openWrite: self filename.

myfile nextPutAll: 'myRootObject' sixxString encodeAsUTF8.

myfile close.

(or any other file created by Pharo encoded with UTF8 as well)

(FileStream oldFileNamed: self filename)

will the api of FileStream know how to read this? I ask because I found methods like #contentsAsUtf8 which made me doubt.

Thanks in advance,

--
View this message in context: http://forum.world.st/Glass-Problems-managing-files-from-Pharo-and-GemStone-tp4754791p4754805.html
Sent from the GLASS mailing list archive at Nabble.com.
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

--
Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Richard Sargent

Re: [Glass] Problems managing files from Pharo and GemStone

Administrator

Mariano Martinez Peck wrote

...
I wonder which is the better way to handle the real stream API.. I
mean... if each time I do a #nextPutAll: with a string I need to send
#encodeAsUTF8, that is not very fun.

I agree. It's not fun. And it seems to me that we will need encoded stream support in the GemStone product. Community contributions in this area would be welcome. :-)

I wonder if Johan's text converters or MultiByteBinaryOrTextStream could be used. Ideally we should have a stream API that allow us to transparently encode with certain encoding, right?

Anyway....is it OK to assume FileStream will be able to properly decode a UTF stored? Say I wrote a file like this:

myfile := GsFile openWrite: self filename.
myfile nextPutAll: 'myRootObject' sixxString encodeAsUTF8.
myfile close.

(or any other file created by Pharo encoded with UTF8 as well)

(FileStream oldFileNamed: self filename)

will the api of FileStream know how to read this? I ask because I found
methods like #contentsAsUtf8 which made me doubt.

I don't know the answer to these. GS has a FileStreamPortable in the base.
(I don't know whether the FileStream you refer to is a GLASS artefact or in Pharo.)

At the present time, it remains "hard work" for you. I suggest a Stream wrapper around GsFile to provide a subset API for the things you need, so you don't have to add UTF-8 conversions everywhere in your code. e.g. a Utf8ReadStream or perhaps a Utf8ReadWriteStream class. It doesn't have to expose the entire GsFile API.