Using a Unix filedescriptor in a FileStream?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Using a Unix filedescriptor in a FileStream?

Holger Freyther
Hi,

I explored to use the Linux inotify API using UFFI. The inotify_init/inotify_init1 routine will me a Unix filedescriptor and I would like to

        a.) monitor it for being readable
        b.) read from it with a stream


For a.) I have found the AioEventHandler and think I will be able to call >>#descriptor: directly and then can use it (still figuring out the API, probably just wait for the >>#changed: call).

For b.) I thought I could use AttachableFileStream but that required a "fileId" but not a filedescriptor. Is there a way I can read from my fd using the standard stream API (otherwise I can try to use UFFI for read)


regards
        holger
Reply | Threaded
Open this post in threaded view
|

Re: Using a Unix filedescriptor in a FileStream?

Mariano Martinez Peck
Hi Holger,

I took some the AttacheableFileStream from OSProcess and I changed it a bit for OSSubprocess. If you load OSSubprocess into Pharo 5.0 [1] you can try the following:

| reader |
reader := OSSAttachableFileStream name:'myStream' attachTo: aFileID writable: false.
reader setNonBlocking "optional"

Then you can read via #upToEnd (or any of StandardFileStream, as OSSAttachableFileStream is a subclass). I hope with this you can solve b).
I still didn't understand why do you mean with a). What do you mean by "monitor it from being readable" ?

I think you could dig a bit in OSSPipe, OSSAttachableFileStream and their usage. All classes have class comments, all methods are also documented, and there is quite some documentation in [1]. 

Let me know how it goes.

Chers,



On Tue, Sep 20, 2016 at 3:26 PM, Holger Freyther <[hidden email]> wrote:
Hi,

I explored to use the Linux inotify API using UFFI. The inotify_init/inotify_init1 routine will me a Unix filedescriptor and I would like to

        a.) monitor it for being readable
        b.) read from it with a stream


For a.) I have found the AioEventHandler and think I will be able to call >>#descriptor: directly and then can use it (still figuring out the API, probably just wait for the >>#changed: call).

For b.) I thought I could use AttachableFileStream but that required a "fileId" but not a filedescriptor. Is there a way I can read from my fd using the standard stream API (otherwise I can try to use UFFI for read)


regards
        holger



--
Reply | Threaded
Open this post in threaded view
|

Re: Using a Unix filedescriptor in a FileStream?

Holger Freyther

> On 20 Sep 2016, at 20:38, Mariano Martinez Peck <[hidden email]> wrote:
>
> Hi Holger,


Good Morning,

thank you for your reply.

> | reader |
> reader := OSSAttachableFileStream name:'myStream' attachTo: aFileID writable: false.
> reader setNonBlocking "optional"

the only issue is that i have a "int fd" and not a SQFile. In >>#name:attachToCFile:writable: you create a SQFile* out of a FILE* (OSSCFile) but judging the comment it doesn't work. The 32/64 bit issue can be solved by using FFIExternalStructure to model SQFile but the question if the VM was built with large file support on GNU/Linux is a tricky one.

So maybe we create another primitive to convert a FILE* to a SQFile* (and have it manage the lifetime of that memory?)? And maybe another primitive to do the same for a Socket?


> I still didn't understand why do you mean with a). What do you mean by "monitor it from being readable" ?

Imagine you want to exit the image in case the file /exit changes. You will charge inotify to watch this filepath and if the fd becomes readable you already know the answer, you don't have to read the event. But true if I have a Socket or FileStream I can do blocking read on it as well.


> I think you could dig a bit in OSSPipe, OSSAttachableFileStream and their usage. All classes have class comments, all methods are also documented, and there is quite some documentation in [1].

Will look again but I didn't see anything obvious. E.g. primCreatePipe seems to already return two SQFile* ("fileId")?

cheers
        holger
Reply | Threaded
Open this post in threaded view
|

Re: Using a Unix filedescriptor in a FileStream?

Mariano Martinez Peck


On Wed, Sep 21, 2016 at 3:55 AM, Holger Freyther <[hidden email]> wrote:

> On 20 Sep 2016, at 20:38, Mariano Martinez Peck <[hidden email]> wrote:
>
> Hi Holger,


Good Morning,

thank you for your reply.


Good morning. 
 
> | reader |
> reader := OSSAttachableFileStream name:'myStream' attachTo: aFileID writable: false.
> reader setNonBlocking "optional"

the only issue is that i have a "int fd" and not a SQFile. In >>#name:attachToCFile:writable: you create a SQFile* out of a FILE* (OSSCFile) but judging the comment it doesn't work.


Thanks! That was the method I was trying to remember and I was failing! hahahaha.  

 
The 32/64 bit issue can be solved by using FFIExternalStructure to model SQFile but the question if the VM was built with large file support on GNU/Linux is a tricky one.


Yes exactly. The basic scenario was working, but having it working in all the different flavors of OS and VMs was tricky as the comment says. My conclusion in this case was that it's not worth trying to do this from image side via UFFI and that instead we should have a VM primitive. 

 
So maybe we create another primitive to convert a FILE* to a SQFile* (and have it manage the lifetime of that memory?)? And maybe another primitive to do the same for a Socket?


Exactly. I have been wanting this a couple of times while doing OSSubprocess. 
 

> I still didn't understand why do you mean with a). What do you mean by "monitor it from being readable" ?

Imagine you want to exit the image in case the file /exit changes. You will charge inotify to watch this filepath and if the fd becomes readable you already know the answer, you don't have to read the event. But true if I have a Socket or FileStream I can do blocking read on it as well.

OK, now I understand. But if you do blocking read, wouldn't be locking the rest of the image by that read?  Of course, you can do busy waiting with none blocking reads.... but for that, inotify may be more performant?
 


> I think you could dig a bit in OSSPipe, OSSAttachableFileStream and their usage. All classes have class comments, all methods are also documented, and there is quite some documentation in [1].

Will look again but I didn't see anything obvious. E.g. primCreatePipe seems to already return two SQFile* ("fileId")?


Yes, exactly. I remember now. And as I said, I also wanted to be able to work at fd or FILE* level and I failed. 

Cheers, 


--
Reply | Threaded
Open this post in threaded view
|

Re: Using a Unix filedescriptor in a FileStream?

Holger Freyther

> On 21 Sep 2016, at 15:09, Mariano Martinez Peck <[hidden email]> wrote:
>
>
>
> Exactly. I have been wanting this a couple of times while doing OSSubprocess.


https://github.com/pharo-project/pharo-vm/pull/108. Would be nice if you could review it and give it a try. It adds two primitive (one to work on fd one to work on FILE).

I probably also want to do:

        sqFile->isStdioStream = isatty(fileno(file))


> Yes, exactly. I remember now. And as I said, I also wanted to be able to work at fd or FILE* level and I failed.

Could you give the above a try and then I try to get it into the Opensmalltalk VM.

holger


Reply | Threaded
Open this post in threaded view
|

Re: Using a Unix filedescriptor in a FileStream?

Mariano Martinez Peck


On Thu, Sep 22, 2016 at 12:12 PM, Holger Freyther <[hidden email]> wrote:

> On 21 Sep 2016, at 15:09, Mariano Martinez Peck <[hidden email]> wrote:
>
>
>
> Exactly. I have been wanting this a couple of times while doing OSSubprocess.


https://github.com/pharo-project/pharo-vm/pull/108. Would be nice if you could review it and give it a try. It adds two primitive (one to work on fd one to work on FILE).

I probably also want to do:

        sqFile->isStdioStream = isatty(fileno(file))


> Yes, exactly. I remember now. And as I said, I also wanted to be able to work at fd or FILE* level and I failed.

Could you give the above a try and then I try to get it into the Opensmalltalk VM.

Hi Holger,

I just reviewed the PR and it looks really good. Please, allow me some time to get updated to the VM compiling instructions and I will give it a try. Probably, I will be testing it by doing my original (unused now) #name:attachToCFile:writable:  use the new primitive and running the OSSubprocess tests. Does this make sense to you?

Thanks a LOT for going deep and fix it!

 


--
Reply | Threaded
Open this post in threaded view
|

Re: Using a Unix filedescriptor in a FileStream?

Mariano Martinez Peck
Hi Holger,

OK, I made a new VM with your PR after I reviewed it. Good news? It works!!!  If you want to check OSSubprocess, I am using now the primCreatePipe from OSProcess because that would answer me directly the SQFiles of the pipe. Originally (before ending up doing that), I was trying to make the pipes myself via FFI ( pipe() ) but I came to the problem we discussed earlier (remember the #name:attachTo:writable:).

So now I can go back to my original solution. The code is in #makePipeWithReadBlocking: and now it looks like this:

| pipePointer returnValue fileDescriptors pipe fileIDsArray fileDescriptor1 fileDescriptor2 |
pipePointer := ExternalAddress allocate: 8.
[
returnValue := self primitivePipe: pipePointer.
(returnValue = -1) ifTrue: [ self perror: 'pipe()' ]. 
fileIDsArray := Array new: 2.
fileDescriptor1 := pipePointer nbUInt32AtOffset: 0.
fileDescriptor2 := pipePointer nbUInt32AtOffset: 4.
fileIDsArray at: 1 put: (self primitiveFileOpenUseFileDescriptor: fileDescriptor1 writeFlag: false).
fileIDsArray at: 2 put: (self primitiveFileOpenUseFileDescriptor: fileDescriptor2 writeFlag: true).
pipe := OSSPipe newWith: fileIDsArray readBlocking: aBoolean. 
] ensure:[
pipePointer free.
].
^ pipe


I just run all OSSubprocess tests and they all worked! (tested in Pharo 5.0). 

I guess I will commit this on the dev branch and hopefully when this is integrated into the VM I can merge that for my next OSSubprocess release.

Thank you very much Holger




On Thu, Sep 22, 2016 at 12:35 PM, Mariano Martinez Peck <[hidden email]> wrote:


On Thu, Sep 22, 2016 at 12:12 PM, Holger Freyther <[hidden email]> wrote:

> On 21 Sep 2016, at 15:09, Mariano Martinez Peck <[hidden email]> wrote:
>
>
>
> Exactly. I have been wanting this a couple of times while doing OSSubprocess.


https://github.com/pharo-project/pharo-vm/pull/108. Would be nice if you could review it and give it a try. It adds two primitive (one to work on fd one to work on FILE).

I probably also want to do:

        sqFile->isStdioStream = isatty(fileno(file))


> Yes, exactly. I remember now. And as I said, I also wanted to be able to work at fd or FILE* level and I failed.

Could you give the above a try and then I try to get it into the Opensmalltalk VM.

Hi Holger,

I just reviewed the PR and it looks really good. Please, allow me some time to get updated to the VM compiling instructions and I will give it a try. Probably, I will be testing it by doing my original (unused now) #name:attachToCFile:writable:  use the new primitive and running the OSSubprocess tests. Does this make sense to you?

Thanks a LOT for going deep and fix it!

 


--



--

On Thu, Sep 22, 2016 at 12:35 PM, Mariano Martinez Peck <[hidden email]> wrote:


On Thu, Sep 22, 2016 at 12:12 PM, Holger Freyther <[hidden email]> wrote:

> On 21 Sep 2016, at 15:09, Mariano Martinez Peck <[hidden email]> wrote:
>
>
>
> Exactly. I have been wanting this a couple of times while doing OSSubprocess.


https://github.com/pharo-project/pharo-vm/pull/108. Would be nice if you could review it and give it a try. It adds two primitive (one to work on fd one to work on FILE).

I probably also want to do:

        sqFile->isStdioStream = isatty(fileno(file))


> Yes, exactly. I remember now. And as I said, I also wanted to be able to work at fd or FILE* level and I failed.

Could you give the above a try and then I try to get it into the Opensmalltalk VM.

Hi Holger,

I just reviewed the PR and it looks really good. Please, allow me some time to get updated to the VM compiling instructions and I will give it a try. Probably, I will be testing it by doing my original (unused now) #name:attachToCFile:writable:  use the new primitive and running the OSSubprocess tests. Does this make sense to you?

Thanks a LOT for going deep and fix it!

 


--



--
Reply | Threaded
Open this post in threaded view
|

Re: Using a Unix filedescriptor in a FileStream?

Holger Freyther

> On 22 Sep 2016, at 23:33, Mariano Martinez Peck <[hidden email]> wrote:
>
> Hi Holger,

Hey!

>
> I just run all OSSubprocess tests and they all worked! (tested in Pharo 5.0).
>
> I guess I will commit this on the dev branch and hopefully when this is integrated into the VM I can merge that for my next OSSubprocess release.

cool and thank you for trying it so quickly. I thought it make sense to mark non files the same as isStdioStream but that triggers a funny case:

                do {
                        clearerr(file);
                        if (fread(dst, 1, 1, file) == 1) {
                                bytesRead += 1;
                                if (dst[bytesRead-1] == '\n'
                                 || dst[bytesRead-1] == '\r')
                                        break;
                        }
                } while (bytesRead <= 0 && ferror(file) && errno == EINTR);

which means 0 or 1 char is read with >>#primRead:into:startingAt:count: and in case of inotify the event is lost (partially read and the rest discarded). I will have to start a discussion why primRead should be line buffered at all.

Anyway. My good news is that:

"an INotify"
| arr fileID |
self init: 8r00004000.
self prim_add_watch: '/tmp' flags: 16r00000100.
arr := ByteArray new: 4096.
fileID := StandardFileStream new primFdOpen: fd writable: false.
StandardFileStream new
        primRead: fileID into: arr startingAt: 1 count: 4096.
arr

has read a file notification event.. :)
Reply | Threaded
Open this post in threaded view
|

Re: Using a Unix filedescriptor in a FileStream?

Ronie Salgado
In reply to this post by Holger Freyther
Hi Holger,

Sorry for not noticing this thread before. Some months ago, I did some experiments myself using inotify and epoll, with the UFFI. I was creating a file system monitoring API similar in design to OSWindow. I just did some minor changes for making this usable along with making a configure script. I left all of the code and some scripts for loading the FileSystemMonitor API at: http://smalltalkhub.com/#!/~ronsaldo/OSCalls .

For testing the monitoring API, try the following in a playground:

monitor := FileSystemMonitor for: '.' do: [ :event |
    Transcript show: event; cr.
].

If you do this, open a Transcript and you start browsing some methods in Nautilus, you will start receiving some events where Pharo is accessing the .changes file. For the destroying the monitor and stop receiving the events, you can do the following:

monitor destroy.

Currently the events given by this API are only holding a copy of the raw inotify event in rawEvent and a copy of the raw path in rawPath. Further work is required for translating the events into a more platform independent interface, and for supporting OS X and Windows with this API. However, this is already something that could be useful to you.

Best regards,
Ronie

2016-09-20 15:26 GMT-03:00 Holger Freyther <[hidden email]>:
Hi,

I explored to use the Linux inotify API using UFFI. The inotify_init/inotify_init1 routine will me a Unix filedescriptor and I would like to

        a.) monitor it for being readable
        b.) read from it with a stream


For a.) I have found the AioEventHandler and think I will be able to call >>#descriptor: directly and then can use it (still figuring out the API, probably just wait for the >>#changed: call).

For b.) I thought I could use AttachableFileStream but that required a "fileId" but not a filedescriptor. Is there a way I can read from my fd using the standard stream API (otherwise I can try to use UFFI for read)


regards
        holger

Reply | Threaded
Open this post in threaded view
|

Re: Using a Unix filedescriptor in a FileStream?

Holger Freyther

> On 23 Sep 2016, at 02:44, Ronie Salgado <[hidden email]> wrote:
>
> Hi Holger,

Hi!


> Currently the events given by this API are only holding a copy of the raw inotify event in rawEvent and a copy of the raw path in rawPath. Further work is required for translating the events into a more platform independent interface, and for supporting OS X and Windows with this API. However, this is already something that could be useful to you.

if I see it correctly you start a poll loop (wait 100ms then call epoll wait)? Would you be interested to try to use AioEventHandler with the primitive I proposed?

regards
        holger