[Glass] Problems managing files from Pharo and GemStone

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[Glass] Problems managing files from Pharo and GemStone

Mariano Martinez Peck
Hi guys. I don't understand why I have just faced this problem now and not before...but... I am writing a text file (sixx serialization) in GemStone this way:

FileStream forceNewFileNamed: 'test' do: [:str | str nextPutAll: 'whateverRootObject' sixxString ].

And then in Pharo I read like:

SixxShapeChangeReadStream on: (FileStream oldFileNamed: 'test')

For some files, the above works OK. But for other files, while reading in Pharo I get a: "UTF8Error: Invalid UTF8 input detected"

The problem is that FileStream oldFileNamed: in Pharo answers a MultiByteFileStream, which uses a UTF8TextConverter. Hence, in UTF8TextConverter >> nextFromStream:  I get an error because 'unicode' instVar is nil.

Of course, if in Pharo I use directly a StandardFileStream oldFileNamed: (to avoid using MultiByteFileStream) I can indeed read those files. 

So...how should I make this to work in GemStone and Pharo? I know Johan ported the text converters and MultiByteBinaryOrTextStream. But I don't know how to get a similar MultiByteFileStream for GemStone.

Thanks in advance,

--
Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Problems managing files from Pharo and GemStone

Mariano Martinez Peck
Ok. It seems that in GemStone GsFile only accepts 0-255 range so I need to pre-encode strings. Something like this:

myfile := GsFile openWrite: self filename.
myfile nextPutAll: 'myRootObject' sixxString encodeAsUTF8.
myfile close.

is this the way that it should be done in GemStone?

Thanks,


On Tue, Apr 15, 2014 at 12:27 PM, Mariano Martinez Peck <[hidden email]> wrote:
Hi guys. I don't understand why I have just faced this problem now and not before...but... I am writing a text file (sixx serialization) in GemStone this way:

FileStream forceNewFileNamed: 'test' do: [:str | str nextPutAll: 'whateverRootObject' sixxString ].

And then in Pharo I read like:

SixxShapeChangeReadStream on: (FileStream oldFileNamed: 'test')

For some files, the above works OK. But for other files, while reading in Pharo I get a: "UTF8Error: Invalid UTF8 input detected"

The problem is that FileStream oldFileNamed: in Pharo answers a MultiByteFileStream, which uses a UTF8TextConverter. Hence, in UTF8TextConverter >> nextFromStream:  I get an error because 'unicode' instVar is nil.

Of course, if in Pharo I use directly a StandardFileStream oldFileNamed: (to avoid using MultiByteFileStream) I can indeed read those files. 

So...how should I make this to work in GemStone and Pharo? I know Johan ported the text converters and MultiByteBinaryOrTextStream. But I don't know how to get a similar MultiByteFileStream for GemStone.

Thanks in advance,

--
Mariano
http://marianopeck.wordpress.com



--
Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Problems managing files from Pharo and GemStone

Richard Sargent
Administrator
Mariano Martinez Peck wrote
Ok. It seems that in GemStone GsFile only accepts 0-255 range so I need to
pre-encode strings. ...

is this the way that it should be done in GemStone?
That is correct. GsFile provides no automatic character encoding.
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Problems managing files from Pharo and GemStone

johnmci
I'm curious here on os-x file name/paths require normalization with specialization 

I'd expect if you are targeting os-x you'd need to know that a UTF8 string might not work as you expect.


On Tue, Apr 15, 2014 at 9:37 AM, Richard Sargent <[hidden email]> wrote:
Mariano Martinez Peck wrote
> Ok. It seems that in GemStone GsFile only accepts 0-255 range so I need to
> pre-encode strings. ...
>
> is this the way that it should be done in GemStone?

That is correct. GsFile provides no automatic character encoding.




--
View this message in context: http://forum.world.st/Glass-Problems-managing-files-from-Pharo-and-GemStone-tp4754791p4754805.html
Sent from the GLASS mailing list archive at Nabble.com.
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass



--
===========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd. Twitter: squeaker68882
===========================================================================


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Problems managing files from Pharo and GemStone

Mariano Martinez Peck
In reply to this post by Richard Sargent



On Tue, Apr 15, 2014 at 1:37 PM, Richard Sargent <[hidden email]> wrote:
Mariano Martinez Peck wrote
> Ok. It seems that in GemStone GsFile only accepts 0-255 range so I need to
> pre-encode strings. ...
>
> is this the way that it should be done in GemStone?

That is correct. GsFile provides no automatic character encoding.


mmmm I wonder which is the better way to handle the real stream API.. I mean... if each time I do a #nextPutAll: with a string I need to send #encodeAsUTF8, that is not very fun. I wonder if Johan's text converters or MultiByteBinaryOrTextStream could be used. Ideally we should have a stream API that allow us to transparently encode with certain encoding, right?
 
Anyway....is it OK to assume FileStream will be able to properly decode a UTF stored? Say I wrote a file like this:

myfile := GsFile openWrite: self filename.
myfile nextPutAll: 'myRootObject' sixxString encodeAsUTF8.
myfile close.

(or any other file created by Pharo encoded with UTF8 as well)

(FileStream oldFileNamed: self filename) 

will the api of FileStream know how to read this? I ask because I found methods like #contentsAsUtf8 which made me doubt.

Thanks in advance,




--
View this message in context: http://forum.world.st/Glass-Problems-managing-files-from-Pharo-and-GemStone-tp4754791p4754805.html
Sent from the GLASS mailing list archive at Nabble.com.
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass



--
Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Problems managing files from Pharo and GemStone

Richard Sargent
Administrator
Mariano Martinez Peck wrote
...
I wonder which is the better way to handle the real stream API.. I
mean... if each time I do a #nextPutAll: with a string I need to send
#encodeAsUTF8, that is not very fun.
I agree. It's not fun. And it seems to me that we will need encoded stream support in the GemStone product. Community contributions in this area would be welcome. :-)


I wonder if Johan's text converters or MultiByteBinaryOrTextStream could be used. Ideally we should have a stream API that allow us to transparently encode with certain encoding, right?

Anyway....is it OK to assume FileStream will be able to properly decode a UTF stored? Say I wrote a file like this:

myfile := GsFile openWrite: self filename.
myfile nextPutAll: 'myRootObject' sixxString encodeAsUTF8.
myfile close.

(or any other file created by Pharo encoded with UTF8 as well)

(FileStream oldFileNamed: self filename)

will the api of FileStream know how to read this? I ask because I found
methods like #contentsAsUtf8 which made me doubt.
I don't know the answer to these. GS has a FileStreamPortable in the base.
(I don't know whether the FileStream you refer to is a GLASS artefact or in Pharo.)


At the present time, it remains "hard work" for you. I suggest a Stream wrapper around GsFile to provide a subset API for the things you need, so you don't have to add UTF-8 conversions everywhere in your code. e.g. a Utf8ReadStream or perhaps a Utf8ReadWriteStream class. It doesn't have to expose the entire GsFile API.