Lost in stream

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Lost in stream

HilaireFernandes
Hi,

I have issue when moving away from the deprecated File API to save PNG
image:

This fromer code to save file (xml or PNG file):

"    |streamOnDisk|
     [streamOnDisk := MultiByteFileStream forceNewFileNamed: (self
absolutePath: filename).
     streamOnDisk nextPutAll: stream contents] ensure:
         [streamOnDisk close]"

and the newer one:

     (location  asFileReference / filename) binaryWriteStreamDo: [
:fileStream |
         fileStream nextPutAll: stream contents]

The new one produce both wrong XML file and PNG file. The file looks
short cuted at the end.

The XML document contents itself a chunk of binary data (a bitmap
dropped in the drgeo sketch), and the fragility seems to come from there.

I already met fragilities with some file when loading older drgeo
document, but I have not fully tracked the issue.

What is wrong?

The enclosed archive contains both files saved with the former and newer
methods, both XML and PNG documents.

Thanks

Hilaire

--
Dr. Geo
http://drgeo.eu


OutputComparing.zip (471K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Lost in stream

HilaireFernandes
I investigated a bit more the other issue, and it looks like XML
entities are not correctly decoded when read from the newer File API.
Somehow it is related to stream I guess.

For example, drgeo document starting like:

<?xml version="1.0"?>
<drgenius><drgeo name="Triangle homoth&#xE9;tie">

are correctly decoded with DrGeo 17.07, but not with the newer one.

When debugging, it looks like there is a jump at the end of the entity,
the $; character is not returned by the (self next) in the SAX Driver.

May be I used the wrong stream when opening the doc, but I don't know,
the stream was open as:

'triangle-scale.fgeo' asFileReference readStream

Hilaire


Le 15/06/2018 à 18:31, Hilaire a écrit :
> I already met fragilities with some file when loading older drgeo
> document, but I have not fully tracked the issue.

--
Dr. Geo
http://drgeo.eu



Reply | Threaded
Open this post in threaded view
|

Re: Lost in stream

Sven Van Caekenberghe-2
I don't see anything wrong with Toto-Wrong.png.

As for the XML part, the following seemed to work:

xmldoc := (FileLocator desktop / 'Toto-ok.fgeo') readStreamDo: [ :in | XMLDOMParser parse: in ].

(FileLocator desktop / 'drgeo.xml') writeStreamDo: [ :out | xmldoc writeXMLOn: (XMLWriter on: out) ].

I know this is the DOM parser, not the SAX parser, but as far as streams are concerned, XML* seems to work OK.

> On 15 Jun 2018, at 19:17, Hilaire <[hidden email]> wrote:
>
> I investigated a bit more the other issue, and it looks like XML entities are not correctly decoded when read from the newer File API. Somehow it is related to stream I guess.
>
> For example, drgeo document starting like:
>
> <?xml version="1.0"?>
> <drgenius><drgeo name="Triangle homoth&#xE9;tie">
>
> are correctly decoded with DrGeo 17.07, but not with the newer one.
>
> When debugging, it looks like there is a jump at the end of the entity, the $; character is not returned by the (self next) in the SAX Driver.
>
> May be I used the wrong stream when opening the doc, but I don't know, the stream was open as:
>
> 'triangle-scale.fgeo' asFileReference readStream
>
> Hilaire
>
>
> Le 15/06/2018 à 18:31, Hilaire a écrit :
>> I already met fragilities with some file when loading older drgeo document, but I have not fully tracked the issue.
>
> --
> Dr. Geo
> http://drgeo.eu
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Lost in stream

HilaireFernandes
Le 15/06/2018 à 20:40, Sven Van Caekenberghe a écrit :
> I don't see anything wrong with Toto-Wrong.png.
When loading like this I have space low detection:
PNGReadWriter formFromStream: 'Toto-Wrong.png' asFileReference
binaryReadStream


> As for the XML part, the following seemed to work:
>
> xmldoc := (FileLocator desktop / 'Toto-ok.fgeo') readStreamDo: [ :in | XMLDOMParser parse: in ].
>
> (FileLocator desktop / 'drgeo.xml') writeStreamDo: [ :out | xmldoc writeXMLOn: (XMLWriter on: out) ].
>
> I know this is the DOM parser, not the SAX parser, but as far as streams are concerned, XML* seems to work OK.

I am using the DOM parser, but apparently it uses SAX
I don't have writeXMLOn: method. I am using an old version of the XML
package, is it? Its newer version breaks the API.


--
Dr. Geo
http://drgeo.eu



Reply | Threaded
Open this post in threaded view
|

Re: Lost in stream

HilaireFernandes
In reply to this post by HilaireFernandes
Sometime you just need a good sleep, which I was fortunate to have.

My newer code append contents to existing file, so it lead to corrupted
files at the end. Ensuring it is deleted first solved the problem:

(location  asFileReference / filename) ensureDelete binaryWriteStreamDo:
[ :fileStream |
        fileStream nextPutAll: stream contents]

There is still the issue of XML entity I described in an previous email,
the impact is minor for Dr. Geo though. Do other have issues with that?

Hilaire


Le 15/06/2018 à 18:31, Hilaire a écrit :

> This fromer code to save file (xml or PNG file):
>
> " |streamOnDisk|
>     [streamOnDisk := MultiByteFileStream forceNewFileNamed: (self
> absolutePath: filename).
>     streamOnDisk nextPutAll: stream contents] ensure:
>         [streamOnDisk close]"
>
> and the newer one:
>
>     (location  asFileReference / filename) binaryWriteStreamDo: [
> :fileStream |
>         fileStream nextPutAll: stream contents]
>
> The new one produce both wrong XML file and PNG file. The file looks
> short cuted at the end.

--
Dr. Geo
http://drgeo.eu



Reply | Threaded
Open this post in threaded view
|

Re: Lost in stream

monty-3
Print-evaluate this:
'<?xml version="1.0"?>
<drgenius><drgeo name="Triangle homoth&#xE9;tie"/></drgenius>' parseXML

and you should get (with any recent XMLParser lib):
<?xml version="1.0"?><drgenius><drgeo name="Triangle homothétie"/></drgenius>

If you don't want entity replacement, set #replacesContentEntityReferences: to false. Also consider using XMLParser's built-in file reading support: #parseFileNamed:/#onFileNamed:. They work cross platform (Squeak, GS), and handle character decoding.

___
montyos.wordpress.com


> Sent: Saturday, June 16, 2018 at 4:27 AM
> From: Hilaire <[hidden email]>
> To: [hidden email]
> Subject: Re: [Pharo-users] Lost in stream
>
> Sometime you just need a good sleep, which I was fortunate to have.
>
> My newer code append contents to existing file, so it lead to corrupted
> files at the end. Ensuring it is deleted first solved the problem:
>
> (location  asFileReference / filename) ensureDelete binaryWriteStreamDo:
> [ :fileStream |
>         fileStream nextPutAll: stream contents]
>
> There is still the issue of XML entity I described in an previous email,
> the impact is minor for Dr. Geo though. Do other have issues with that?
>
> Hilaire
>
>
> Le 15/06/2018 à 18:31, Hilaire a écrit :
> > This fromer code to save file (xml or PNG file):
> >
> > " |streamOnDisk|
> >     [streamOnDisk := MultiByteFileStream forceNewFileNamed: (self
> > absolutePath: filename).
> >     streamOnDisk nextPutAll: stream contents] ensure:
> >         [streamOnDisk close]"
> >
> > and the newer one:
> >
> >     (location  asFileReference / filename) binaryWriteStreamDo: [
> > :fileStream |
> >         fileStream nextPutAll: stream contents]
> >
> > The new one produce both wrong XML file and PNG file. The file looks
> > short cuted at the end.
>
> --
> Dr. Geo
> http://drgeo.eu
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Lost in stream

monty-3
In reply to this post by HilaireFernandes
Are you developing in Squeak? I ask because the XML parser in the default Squeak image is very old. You can download the latest Pharo XMLParser lib from the SqueakMap. It's the "XMLParser" project. Other libs I maintain are also available as SM projects, like "XMLParser-XPath".

If you're doing DOM parsing, you can save a parsed DOM document to a file using a message like #printToFileNamed: (see the XMLNode "printing" category for more).

___
montyos.wordpress.com


> Sent: Saturday, June 16, 2018 at 4:27 AM
> From: Hilaire <[hidden email]>
> To: [hidden email]
> Subject: Re: [Pharo-users] Lost in stream
>
> Sometime you just need a good sleep, which I was fortunate to have.
>
> My newer code append contents to existing file, so it lead to corrupted
> files at the end. Ensuring it is deleted first solved the problem:
>
> (location  asFileReference / filename) ensureDelete binaryWriteStreamDo:
> [ :fileStream |
>         fileStream nextPutAll: stream contents]
>
> There is still the issue of XML entity I described in an previous email,
> the impact is minor for Dr. Geo though. Do other have issues with that?
>
> Hilaire
>
>
> Le 15/06/2018 à 18:31, Hilaire a écrit :
> > This fromer code to save file (xml or PNG file):
> >
> > " |streamOnDisk|
> >     [streamOnDisk := MultiByteFileStream forceNewFileNamed: (self
> > absolutePath: filename).
> >     streamOnDisk nextPutAll: stream contents] ensure:
> >         [streamOnDisk close]"
> >
> > and the newer one:
> >
> >     (location  asFileReference / filename) binaryWriteStreamDo: [
> > :fileStream |
> >         fileStream nextPutAll: stream contents]
> >
> > The new one produce both wrong XML file and PNG file. The file looks
> > short cuted at the end.
>
> --
> Dr. Geo
> http://drgeo.eu
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Lost in stream

Sven Van Caekenberghe-2
In reply to this post by monty-3


> On 18 Jun 2018, at 02:18, monty <[hidden email]> wrote:
>
> Also consider using XMLParser's built-in file reading support: #parseFileNamed:/#onFileNamed:. They work cross platform (Squeak, GS), and handle character decoding.

Monty, do these (already) work with all the latest changes in Pharo 7, I mean the deprecation of FileStream and subclasses as well as [RW|MultiByte]BinaryOrTextStream for FileReference, File and Zn streams ?

If not, we should help you.

Sven
Reply | Threaded
Open this post in threaded view
|

Re: Lost in stream

monty-3
They still use (binary) StandardFileStreams on Pharo and Squeak. But since it's done through dynamically chosen file stream factory classes (XMLFileReadStreamFactory and XMLFileWriteStreamFactory), it's easy add support for other stream classes. (The GemStone compat .mcz adds factories for GsFile read/write factories, for example).

#preferredImplementation selects which implementation to use in the hierarchy when there's more than one supported (#isSupportedImplementation).
___
montyos.wordpress.com


> Sent: Monday, June 18, 2018 at 1:34 AM
> From: "Sven Van Caekenberghe" <[hidden email]>
> To: "Any question about pharo is welcome" <[hidden email]>
> Subject: Re: [Pharo-users] Lost in stream
>
>
>
> > On 18 Jun 2018, at 02:18, monty <[hidden email]> wrote:
> >
> > Also consider using XMLParser's built-in file reading support: #parseFileNamed:/#onFileNamed:. They work cross platform (Squeak, GS), and handle character decoding.
>
> Monty, do these (already) work with all the latest changes in Pharo 7, I mean the deprecation of FileStream and subclasses as well as [RW|MultiByte]BinaryOrTextStream for FileReference, File and Zn streams ?
>
> If not, we should help you.
>
> Sven
>

Reply | Threaded
Open this post in threaded view
|

Re: Lost in stream

HilaireFernandes
In reply to this post by monty-3
No with Pharo, and pretty old XML parser version. But still this one was
always capable to parse entity


Le 18/06/2018 à 02:43, monty a écrit :
> Are you developing in Squeak? I ask because the XML parser in the default Squeak image is very old. You can download the latest Pharo XMLParser lib from the SqueakMap. It's the "XMLParser" project. Other libs I maintain are also available as SM projects, like "XMLParser-XPath".
>
> If you're doing DOM parsing, you can save a parsed DOM document to a file using a message like #printToFileNamed: (see the XMLNode "printing" category for more).

--
Dr. Geo
http://drgeo.eu



Reply | Threaded
Open this post in threaded view
|

Re: Lost in stream

monty-3
https://ci.inria.fr/pharo-contribution/job/XMLParser/

You should be able to upgrade regardless of your Pharo version.
___
montyos.wordpress.com


> Sent: Wednesday, June 20, 2018 at 8:39 AM
> From: Hilaire <[hidden email]>
> To: [hidden email]
> Subject: Re: [Pharo-users] Lost in stream
>
> No with Pharo, and pretty old XML parser version. But still this one was
> always capable to parse entity
>
>
> Le 18/06/2018 à 02:43, monty a écrit :
> > Are you developing in Squeak? I ask because the XML parser in the default Squeak image is very old. You can download the latest Pharo XMLParser lib from the SqueakMap. It's the "XMLParser" project. Other libs I maintain are also available as SM projects, like "XMLParser-XPath".
> >
> > If you're doing DOM parsing, you can save a parsed DOM document to a file using a message like #printToFileNamed: (see the XMLNode "printing" category for more).
>
> --
> Dr. Geo
> http://drgeo.eu
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Lost in stream

HilaireFernandes
I should, but there were changes in the API impacting its use in Dr.
Geo, and I have to concentrate on other matter first. Pavel did a port
on Github[1], but I failed to track on-line the changes to the code
related to Dr. Geo's use of XML. I will need time, or helps. I don't use
GitLab, so I have little knowledge about. Dr. Geo uses Launchpad[2]
since years, it is for me far more useful and practical, even more with
Tonel file representation.

Hilaire

[1] https://github.com/pavel-krivanek/DrGeo

[2] http://dev.drgeo.eu


Le 21/06/2018 à 00:45, monty a écrit :
> https://ci.inria.fr/pharo-contribution/job/XMLParser/
>
> You should be able to upgrade regardless of your Pharo version.

--
Dr. Geo
http://drgeo.eu



Reply | Threaded
Open this post in threaded view
|

Re: Lost in stream

HilaireFernandes
Recorded https://bugs.launchpad.net/drgeo/+bug/1777991


Le 21/06/2018 à 08:11, Hilaire a écrit :

> I should, but there were changes in the API impacting its use in Dr.
> Geo, and I have to concentrate on other matter first. Pavel did a port
> on Github[1], but I failed to track on-line the changes to the code
> related to Dr. Geo's use of XML. I will need time, or helps. I don't
> use GitLab, so I have little knowledge about. Dr. Geo uses
> Launchpad[2] since years, it is for me far more useful and practical,
> even more with Tonel file representation.
>
> Hilaire
>
> [1] https://github.com/pavel-krivanek/DrGeo
>
> [2] http://dev.drgeo.eu

--
Dr. Geo
http://drgeo.eu



Reply | Threaded
Open this post in threaded view
|

Re: Lost in stream

Sven Van Caekenberghe-2
In reply to this post by monty-3
Hi Monty,

> On 18 Jun 2018, at 23:43, monty <[hidden email]> wrote:
>
> They still use (binary) StandardFileStreams on Pharo and Squeak. But since it's done through dynamically chosen file stream factory classes (XMLFileReadStreamFactory and XMLFileWriteStreamFactory), it's easy add support for other stream classes. (The GemStone compat .mcz adds factories for GsFile read/write factories, for example).
>
> #preferredImplementation selects which implementation to use in the hierarchy when there's more than one supported (#isSupportedImplementation).
> ___
> montyos.wordpress.com
>
>
>> Sent: Monday, June 18, 2018 at 1:34 AM
>> From: "Sven Van Caekenberghe" <[hidden email]>
>> To: "Any question about pharo is welcome" <[hidden email]>
>> Subject: Re: [Pharo-users] Lost in stream
>>
>>
>>
>>> On 18 Jun 2018, at 02:18, monty <[hidden email]> wrote:
>>>
>>> Also consider using XMLParser's built-in file reading support: #parseFileNamed:/#onFileNamed:. They work cross platform (Squeak, GS), and handle character decoding.
>>
>> Monty, do these (already) work with all the latest changes in Pharo 7, I mean the deprecation of FileStream and subclasses as well as [RW|MultiByte]BinaryOrTextStream for FileReference, File and Zn streams ?
>>
>> If not, we should help you.
>>
>> Sven
>>
>

Here is an attempt at an implementation (untested):

Change

XMLStandardFileStream[Read|Write]StreamFactory class>>isSupportedImplementation
        ^ (XMLClassFinder hasClassNamed: #StandardFileStream)
                and: [ (XMLClassFinder classNamed: #StandardFileStream) isDeprecated not ]

New

subclasses XMLPharoFile[Read|Write]StreamFactory

XMLPharoFile[Read|Write]StreamFactory class>>#isSupportedImplementation
        ^ (XMLClassFinder hasClassNamed: #File)
                and: [ (XMLClassFinder classNamed: #FileStream) isDeprecated ]

XMLPharoFileReadStreamFactory class>>basicOnPath: aPathString
        ^ aPathString asFileReference binaryReadStream

XMLPharoFileWriteStreamFactory class>>basicOnPath: aPathString
        ^ aPathString asFileName ensureDelete binaryWriteStream

File and FileStream have existed in Pharo for quite a while, it is only after FileStream and subclasses where deprecated that we can speak of a real switch.

You also have XMLFileSystemFileHandle that could be used directly, I guess. But you pass on paths as strings, so if you want to keep that, the above should work AFAIKT.

Regards,

Sven