Stdio>>#standardIOStreamNamed:forWrite: bug?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Stdio>>#standardIOStreamNamed:forWrite: bug?

Alistair Grant
Hi Damien, Vincent and Guille,

Just a disclaimer: despite having done some work on this, I don't
consider myself an expert.  If someone else has a better explanation,
please jump in.


Damien wrote on discord:
> is there a notion of standard input/output on windows ?

Yes, console applications can use stdio.  PharoConsole.exe when run from
a command prompt has stdio.  (Note that cygwin terminals are different
and don't support stdio (since Pharo uses the native windows functions)).


> looks like Alistair added the possibility to open files by number,

Holger Freyther deserves the credit here.  My clumsy git merging left
his name out as the author (sorry Holger!).


> that's probably what was missing at the time Stdio was introduced

I'm not sure that opening files by fd would be useful in this particular
case.

There are three ways to access files on Windows:

file descriptor - posix compatibility
FILE* - posix compatibility
HANDLE - windows native

The VM uses native operations for file i/o on Windows, e.g.
Read/WriteConsole() for stdio (including pipe redirection of stdio).

Mixing file descriptors and FILE* is possible if you're careful.  I
don't know how to mix fd / FILE* and HANDLEs.  It's probably better to
stick with using native windows functions most of the time.  We included
these functions on Windows since they exist, and there are probably some
use cases, but you need to know what you're doing.

File class>>stdioHandles calls primitiveFileStdioHandles() which
eventually opens the stdio streams (if possible) using the Windows
native GetStdHandle() function.


> I suspect File needs a nameless variant (I'm not certain how pipes
> work in unix but I doubt echo foo | cat ever touches the filesystem)

That's my understanding (it doesn't touch the file system).


> hmm but the short-term bug is that the TTY logic in Stdio >>
> #standardIOStreamNamed:forWrite: fails when all 3 standard streams are
> redirected

OK, my understanding of what you are saying is that if all 3 io streams
are redirected the method incorrectly determines that a console isn't
present and so falls back to calling #createStdioFileFor:.

More below...


> I also don't understand the assumption there

Vincent Blondeau added / modified these methods, so he may be able to
provide more information.

I think the assumption is that if any of the stdio files are connected
to a TTY. then it is possible to open all the stdio files.  If none are
present, then (on Windows) create regular files to act as a substitute.

The problem is that it only tests for a TTY, when what we really want is
a TTY or a file or a pipe.


Assuming I've understood correctly, maybe try modifying (sorry for the
loss of formatting, only the last line is changed):


File class>>stdioDescriptorIsATTY
^ (0 to: 2)
anySatisfy: [ :fdNum |
| res |
res := self fileDescriptorType: fdNum.
res between: 1 and: 3 ]


We'd need to rename the method, since it is misleading, but it looks
like it better represents the intended use.



Cheers,
Alistair

Reply | Threaded
Open this post in threaded view
|

Re: Stdio>>#standardIOStreamNamed:forWrite: bug?

Guillermo Polito
Hi,

I don't know the windows situation at all, but I'm wondering.

([ File stdioDescriptorIsATTY not ]
on: PrimitiveFailed
do: [ :ex | "HACK kept for retrocompatibility" Smalltalk os isWin32 ])
ifTrue: [ ^ self createStdioFileFor: moniker ].

the first thing I would do, just trying to keep the same behaviour, is to invert the test... Something like

Smalltalk os isWin32 ifTrue: [
[ File stdioDescriptorIsATTY not ]
on: PrimitiveFailed
do: [ :ex | false ])
ifTrue: [ ^ self createStdioFileFor: moniker ]. ]

Like that, this would only penalise windows in the worst case. Then a first question arises in my head: why isWin32 and not isWindows? Is there a reason behind that? Wouldn't that prevent correct behaviour in windows 64 bits?

My next impression is that testing for "a TTY or a file or a pipe" seems buggy... We should put a more intention revealing selector like #isInvalidStdioHandle?
A more high level test will allow us to do any check we want in the background, and it allows better to understand why we check for TTY or File or Pipe or la mar en coche :)

On Thu, Aug 2, 2018 at 8:54 PM Alistair Grant <[hidden email]> wrote:
File class>>stdioHandles calls primitiveFileStdioHandles() which
eventually opens the stdio streams (if possible) using the Windows
native GetStdHandle() function.

What does #stdioHandles return to the image in the case of a non-console windows VM? I mean, when "it is not possible"?

From the docs, https://docs.microsoft.com/en-us/windows/console/getstdhandle says we may have an INVALID_HANDLE_VALUE or NULL in case it fails.
Is the file plugin is masking the error? If so, thats why we need such strange workarounds...

 - If it is an invalid handle, why not just testing for "isInvalidHandle" or so? We can have that on the image side and then dispatch to the correct thing in the file plugin maybe?

 - Maybe the #stdioHandles primitive should just fail in the case we don't have correct handles? that would allow us to do a much more elegant approach on the image side...

> I suspect File needs a nameless variant (I'm not certain how pipes
> work in unix but I doubt echo foo | cat ever touches the filesystem)

That's my understanding (it doesn't touch the file system).


Well... I see this is just a aesthetic issue and I don't see how this is related to the above more serious issue...
Reply | Threaded
Open this post in threaded view
|

Re: Stdio>>#standardIOStreamNamed:forWrite: bug?

Alistair Grant
Hi Guille,

On Fri, Aug 03, 2018 at 10:34:22AM +0200, Guillermo Polito wrote:

> Hi,
>
> I don't know the windows situation at all, but I'm wondering.
>
> ([ File stdioDescriptorIsATTY not ]
> on: PrimitiveFailed
> do: [ :ex | "HACK kept for retrocompatibility" Smalltalk os isWin32 ])
> ifTrue: [ ^ self createStdioFileFor: moniker ].
>
> the first thing I would do, just trying to keep the same behaviour, is to
> invert the test... Something like
>
> Smalltalk os isWin32 ifTrue: [
> [ File stdioDescriptorIsATTY not ]
> on: PrimitiveFailed
> do: [ :ex | false ])
> ifTrue: [ ^ self createStdioFileFor: moniker ]. ]
>
> Like that, this would only penalise windows in the worst case.

True.  Actually I think it would be safe to remove it completely.  Unix
platforms should always have stdio, and other platforms should act
appropriately.  But more below...


> Then a first question arises in my head: why isWin32 and not
> isWindows? Is there a reason behind that? Wouldn't that prevent
> correct behaviour in windows 64 bits?

You're right, it should be #isWindows.


> My next impression is that testing for "a TTY or a file or a pipe"
> seems buggy... We should put a more intention revealing selector like
> #isInvalidStdioHandle?
> A more high level test will allow us to do any check we want in the
> background, and it allows better to understand why we check for TTY or
> File or Pipe or la mar en coche :)

I made the change so that Damien could test that it gives the behaviour
he expects, and did say that the selector will need to be renamed.


> What does #stdioHandles return to the image in the case of a
> non-console windows VM? I mean, when "it is not possible"?

> From the docs,
> https://docs.microsoft.com/en-us/windows/console/getstdhandle says we
> may have an INVALID_HANDLE_VALUE or NULL in case it fails.
> Is the file plugin is masking the error? If so, thats why we need such
> strange workarounds...

We'd still want the ability to test what type of device stdio is
connected to.  But I agree the primitive should be doing the correct
thing.


> - If it is an invalid handle, why not just testing for
> "isInvalidHandle" or so? We can have that on the image side and then
> dispatch to the correct thing in the file plugin maybe?

> - Maybe the #stdioHandles primitive should just fail in the case we
> don't have correct handles? that would allow us to do a much more
> elegant approach on the image side...

I don't think the error handling has been changed in a long time, but
just looking quickly at the source code, I think you're correct that it
isn't returning the correct values.  It should be returning nil for
those handles that aren't valid.

I don't have my Windows VM handy at the moment, is anyone able to
quickly confirm?  (Start pharo with Pharo.exe (not PharoConsole.exe) and
supply the result from:

File stdioHandles


I've been asking lots of questions lately and not providing much input
(I was hoping to have File Attributes integrated by now, but MacOS file
encoding and my inability to get the VM to compile on a Mac VM have
frustrated me), so I'm more than happy to tidy up the stdio primitives.

I'll need to check that it doesn't break anything on Squeak as well
(I don't think any of the changes discussed should cause problems).


Cheers,
Alistair

Reply | Threaded
Open this post in threaded view
|

Re: Stdio>>#standardIOStreamNamed:forWrite: bug?

Alistair Grant
On Fri, 3 Aug 2018 at 11:24, Alistair Grant <[hidden email]> wrote:

>
> Hi Guille,
>
> On Fri, Aug 03, 2018 at 10:34:22AM +0200, Guillermo Polito wrote:
> > Hi,
> >
> > I don't know the windows situation at all, but I'm wondering.
> >
> > ([ File stdioDescriptorIsATTY not ]
> > on: PrimitiveFailed
> > do: [ :ex | "HACK kept for retrocompatibility" Smalltalk os isWin32 ])
> > ifTrue: [ ^ self createStdioFileFor: moniker ].
> >
> > the first thing I would do, just trying to keep the same behaviour, is to
> > invert the test... Something like
> >
> > Smalltalk os isWin32 ifTrue: [
> > [ File stdioDescriptorIsATTY not ]
> > on: PrimitiveFailed
> > do: [ :ex | false ])
> > ifTrue: [ ^ self createStdioFileFor: moniker ]. ]
> >
> > Like that, this would only penalise windows in the worst case.
>
> True.  Actually I think it would be safe to remove it completely.  Unix
> platforms should always have stdio, and other platforms should act
> appropriately.  But more below...
>
>
> > Then a first question arises in my head: why isWin32 and not
> > isWindows? Is there a reason behind that? Wouldn't that prevent
> > correct behaviour in windows 64 bits?
>
> You're right, it should be #isWindows.
>
>
> > My next impression is that testing for "a TTY or a file or a pipe"
> > seems buggy... We should put a more intention revealing selector like
> > #isInvalidStdioHandle?
> > A more high level test will allow us to do any check we want in the
> > background, and it allows better to understand why we check for TTY or
> > File or Pipe or la mar en coche :)
>
> I made the change so that Damien could test that it gives the behaviour
> he expects, and did say that the selector will need to be renamed.
>
>
> > What does #stdioHandles return to the image in the case of a
> > non-console windows VM? I mean, when "it is not possible"?
>
> > From the docs,
> > https://docs.microsoft.com/en-us/windows/console/getstdhandle says we
> > may have an INVALID_HANDLE_VALUE or NULL in case it fails.
> > Is the file plugin is masking the error? If so, thats why we need such
> > strange workarounds...
>
> We'd still want the ability to test what type of device stdio is
> connected to.  But I agree the primitive should be doing the correct
> thing.
>
>
> > - If it is an invalid handle, why not just testing for
> > "isInvalidHandle" or so? We can have that on the image side and then
> > dispatch to the correct thing in the file plugin maybe?
>
> > - Maybe the #stdioHandles primitive should just fail in the case we
> > don't have correct handles? that would allow us to do a much more
> > elegant approach on the image side...
>
> I don't think the error handling has been changed in a long time, but
> just looking quickly at the source code, I think you're correct that it
> isn't returning the correct values.  It should be returning nil for
> those handles that aren't valid.
>
> I don't have my Windows VM handy at the moment, is anyone able to
> quickly confirm?  (Start pharo with Pharo.exe (not PharoConsole.exe) and
> supply the result from:
>
> File stdioHandles
>
>
> I've been asking lots of questions lately and not providing much input
> (I was hoping to have File Attributes integrated by now, but MacOS file
> encoding and my inability to get the VM to compile on a Mac VM have
> frustrated me), so I'm more than happy to tidy up the stdio primitives.
>
> I'll need to check that it doesn't break anything on Squeak as well
> (I don't think any of the changes discussed should cause problems).

Tracking issue: https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/274

Reply | Threaded
Open this post in threaded view
|

Re: Stdio>>#standardIOStreamNamed:forWrite: bug?

Alistair Grant
On Sat, 4 Aug 2018 at 15:43, Alistair Grant <[hidden email]> wrote:

>
> >
> > I don't think the error handling has been changed in a long time, but
> > just looking quickly at the source code, I think you're correct that it
> > isn't returning the correct values.  It should be returning nil for
> > those handles that aren't valid.
> >
> > I don't have my Windows VM handy at the moment, is anyone able to
> > quickly confirm?  (Start pharo with Pharo.exe (not PharoConsole.exe) and
> > supply the result from:
> >
> > File stdioHandles

This does indeed fail to return nil even though stdio is not
available.  Instead a sqFile handle with null HANDLE* is returned.
But the image can't test this as the format of sqFile is platform
dependent and is private to the VM.

> > I've been asking lots of questions lately and not providing much input
> > (I was hoping to have File Attributes integrated by now, but MacOS file
> > encoding and my inability to get the VM to compile on a Mac VM have
> > frustrated me), so I'm more than happy to tidy up the stdio primitives.
> >
> > I'll need to check that it doesn't break anything on Squeak as well
> > (I don't think any of the changes discussed should cause problems).
>
> Tracking issue: https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/274

The issue above is for the VM changes.

The issue for image changes is:
https://pharo.fogbugz.com/f/cases/22296/Stdio-file-creation-fixes


Cheers,
Alistair

Reply | Threaded
Open this post in threaded view
|

Re: Stdio>>#standardIOStreamNamed:forWrite: bug?

Sean P. DeNigris
Administrator
Alistair Grant wrote
>> Tracking issue:
>> https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/274
> The issue above is for the VM changes.
>
> The issue for image changes is:
> https://pharo.fogbugz.com/f/cases/22296/Stdio-file-creation-fixes

Thanks for looking into this, Alistair! Few are brave enough to dive into
these dark corners of the system ;-)



-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Stdio>>#standardIOStreamNamed:forWrite: bug?

Guillermo Polito
Damien,

Could you check that at least the proposed image-side changes fix your problem for *nixes?

On Sat, Aug 4, 2018 at 7:13 PM Sean P. DeNigris <[hidden email]> wrote:
Alistair Grant wrote
>> Tracking issue:
>> https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/274
> The issue above is for the VM changes.
>
> The issue for image changes is:
> https://pharo.fogbugz.com/f/cases/22296/Stdio-file-creation-fixes

Thanks for looking into this, Alistair! Few are brave enough to dive into
these dark corners of the system ;-)



-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html



--

   

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - http://www.cnrs.fr


Web: http://guillep.github.io

Phone: +33 06 52 70 66 13

Reply | Threaded
Open this post in threaded view
|

Re: Stdio>>#standardIOStreamNamed:forWrite: bug?

Damien Pollet
Yes, they do, at least on my macOS.

I have some doubt about #stdioIsAvailable though… before there was a possible value of 4 (cygwin terminal) for the file descriptor type, which is now out of the tested range, is that intentional

On Mon, 6 Aug 2018 at 12:00, Guillermo Polito <[hidden email]> wrote:
Damien,

Could you check that at least the proposed image-side changes fix your problem for *nixes?

On Sat, Aug 4, 2018 at 7:13 PM Sean P. DeNigris <[hidden email]> wrote:
Alistair Grant wrote
>> Tracking issue:
>> https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/274
> The issue above is for the VM changes.
>
> The issue for image changes is:
> https://pharo.fogbugz.com/f/cases/22296/Stdio-file-creation-fixes

Thanks for looking into this, Alistair! Few are brave enough to dive into
these dark corners of the system ;-)



-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html



--

   

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - http://www.cnrs.fr


Web: http://guillep.github.io

Phone: +33 06 52 70 66 13



--
Damien Pollet
type less, do more [ | ] http://people.untyped.org/damien.pollet
Reply | Threaded
Open this post in threaded view
|

Re: Stdio>>#standardIOStreamNamed:forWrite: bug?

Alistair Grant
On Mon, 6 Aug 2018 at 12:18, Damien Pollet <[hidden email]> wrote:
>
> Yes, they do, at least on my macOS.
>
> I have some doubt about #stdioIsAvailable though… before there was a possible value of 4 (cygwin terminal) for the file descriptor type, which is now out of the tested range, is that intentional

Yep.  Pharo can't write to a cygwin terminal since it uses the Windows
Read/WriteConsole() functions.

cygwin terminals are actually written to through pipes, and we can get
the pipe information, and thus determine it is a cygwin terminal.  But
that's it for now.

Cheers,
Alistair



> On Mon, 6 Aug 2018 at 12:00, Guillermo Polito <[hidden email]> wrote:
>>
>> Damien,
>>
>> Could you check that at least the proposed image-side changes fix your problem for *nixes?
>>
>> On Sat, Aug 4, 2018 at 7:13 PM Sean P. DeNigris <[hidden email]> wrote:
>>>
>>> Alistair Grant wrote
>>> >> Tracking issue:
>>> >> https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/274
>>> > The issue above is for the VM changes.
>>> >
>>> > The issue for image changes is:
>>> > https://pharo.fogbugz.com/f/cases/22296/Stdio-file-creation-fixes
>>>
>>> Thanks for looking into this, Alistair! Few are brave enough to dive into
>>> these dark corners of the system ;-)
>>>
>>>
>>>
>>> -----
>>> Cheers,
>>> Sean
>>> --
>>> Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
>>>
>>
>>
>> --
>>
>>
>>
>> Guille Polito
>>
>> Research Engineer
>>
>> Centre de Recherche en Informatique, Signal et Automatique de Lille
>>
>> CRIStAL - UMR 9189
>>
>> French National Center for Scientific Research - http://www.cnrs.fr
>>
>>
>> Web: http://guillep.github.io
>>
>> Phone: +33 06 52 70 66 13
>
>
>
> --
> Damien Pollet
> type less, do more [ | ] http://people.untyped.org/damien.pollet