Smalltalk › Squeak › Squeak - Dev

Re: [Pharo-dev] From a mooc user: method source with it' take so long in Pharo 5

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

3 messages Options

Eliot Miranda-2

Re: [Pharo-dev] From a mooc user: method source with it' take so long in Pharo 5

Hi Henrik,

cc'ing squeak-dev cuz this is relevant to both communities (if and until Pharo takes the unwise decision to double the size of the image by keeping sources in the image).

On May 22, 2016, at 5:01 AM, Henrik Nergaard <[hidden email]> wrote:

There is a method in SourceFileArray #localProcessReadStreamAtFileIndex:atPosition:ifPresent:ifAbsent: which uses a ProccessLocalVariable called ProccessAndSessionLocalSourcesFileArray (see #localProcessReadOnlyCopy). Changing the last line in #readStreamAt:ifPresent:ifAbsent: to use this local process one makes the time to run this snippet:

[ 1 systemNavigation browseMethodsWithSourceString: 'Morph' matchCase: false ] timeProfile

From ~ 10 seconds to ~ 3 seconds (Windows).

However I cannot see that the file handles created by this processLocalVariable to ever be closed, so I suspect those are leaked? In that case there might be the need to implement some “clean up” mechanism for ProccessLocalVariables before they are changed/nilled when a process changes.

Another approach could be to not use a ProccessLocalVariabe at all, but extend the SourceFilesArray class to also hold one read only handle for each of its files, and use these in readStreamAtFileIndex:::… . I guess that it is also necessary then to have semaphore protecting the two last lines such that setting the position in the stream and reading from it cannot be changed by other threads.

I like this approach, but it has a fatal flaw. If one is debugging file access, then as the debugger causes the source of methods to be fetched as the debugger displays them, so the file pointer in the read-only file the access of which one is trying to debug will change under one's feet, and the results will be completely wrong.

An approach which uses a special copy for the debugger doesn't work; one can't debug access to that file for the same reason.

An approach that /should/ work is for the debugger to install its own copy around file-access. This should be recursive. So around every file access in the debugger there would need to be something like

SourceFiles substituteFreshReadOnlyCopiesDuring: [...file access...]

which would remember the current read-only files, evaluate the block using ensure: and have the ensure: block reinstall the previous file.

Does this make sense?

readStreamAtFileIndex: index atPosition: position ifPresent: presentBlock ifAbsent: absentBlock

               | stream result|

               stream := self readOnlyFileAt: index.

               stream ifNil: [ ^ absentBlock value ].        "sources file not available"

               position > stream size ifTrue: [ ^ absentBlock value ].

               readSema critical: [

                              stream position: position.

                              result := presentBlock value: stream

               ].

               ^ result

Best regards,

Henrik

From: Pharo-dev [[hidden email]] On Behalf Of Nicolai Hess
Sent: Thursday, May 19, 2016 9:31 AM
To: Pharo Development List <[hidden email]>
Subject: Re: [Pharo-dev] From a mooc user: method source with it' take so long in Pharo 5

Squeak caches the opened readonly file(handle). It does not have to reopen the file on every call for reading (readonly).

2016-05-18 19:12 GMT+02:00 stepharo <[hidden email]>:

I am wondering why does the search 'method source with it' take so long in Pharo 5? On my PC, When I select the text 'menu' and search for all 'method source with it', in Squeak 5 it takes 3 seconds. In Pharo 5 it takes 21 seconds.

EstebanLM

Re: [Pharo-dev] From a mooc user: method source with it' take so long in Pharo 5

On 22 May 2016, at 17:26, Eliot Miranda <[hidden email]> wrote:

Hi Henrik,

cc'ing squeak-dev cuz this is relevant to both communities (if and until Pharo takes the unwise decision to double the size of the image by keeping sources in the image).

this will not happen :)

yes, we have played with the idea, but at the end we have some good reasons to not do it… but by the way, it was not “doubling” because what we were keeping was a compressed version.

What we’ll probably do is to change the file format to a real hash database for sources (like CDB, or something) and to epicea for changes (because epicea can store full refactor histories, etc…)… but this format migration is still very green, we need first to extract the functionality and decouple it (because now read/write is scattered all over the image).

On May 22, 2016, at 5:01 AM, Henrik Nergaard <[hidden email]> wrote:

There is a method in SourceFileArray #localProcessReadStreamAtFileIndex:atPosition:ifPresent:ifAbsent: which uses a ProccessLocalVariable called ProccessAndSessionLocalSourcesFileArray (see #localProcessReadOnlyCopy). Changing the last line in #readStreamAt:ifPresent:ifAbsent: to use this local process one makes the time to run this snippet:

[ 1 systemNavigation browseMethodsWithSourceString: 'Morph' matchCase: false ] timeProfile

From ~ 10 seconds to ~ 3 seconds (Windows).

However I cannot see that the file handles created by this processLocalVariable to ever be closed, so I suspect those are leaked? In that case there might be the need to implement some “clean up” mechanism for ProccessLocalVariables before they are changed/nilled when a process changes.

Another approach could be to not use a ProccessLocalVariabe at all, but extend the SourceFilesArray class to also hold one read only handle for each of its files, and use these in readStreamAtFileIndex:::… . I guess that it is also necessary then to have semaphore protecting the two last lines such that setting the position in the stream and reading from it cannot be changed by other threads.

I like this approach, but it has a fatal flaw. If one is debugging file access, then as the debugger causes the source of methods to be fetched as the debugger displays them, so the file pointer in the read-only file the access of which one is trying to debug will change under one's feet, and the results will be completely wrong.

An approach which uses a special copy for the debugger doesn't work; one can't debug access to that file for the same reason.

An approach that /should/ work is for the debugger to install its own copy around file-access. This should be recursive. So around every file access in the debugger there would need to be something like

SourceFiles substituteFreshReadOnlyCopiesDuring: [...file access...]

which would remember the current read-only files, evaluate the block using ensure: and have the ensure: block reinstall the previous file.

Does this make sense?

readStreamAtFileIndex: index atPosition: position ifPresent: presentBlock ifAbsent: absentBlock

               | stream result|
               stream := self readOnlyFileAt: index.
               stream ifNil: [ ^ absentBlock value ].        "sources file not available"

               position > stream size ifTrue: [ ^ absentBlock value ].

               readSema critical: [
                              stream position: position.
                              result := presentBlock value: stream
               ].

               ^ result

Best regards,
Henrik

From: Pharo-dev [[hidden email]] On Behalf Of Nicolai Hess
Sent: Thursday, May 19, 2016 9:31 AM
To: Pharo Development List <[hidden email]>
Subject: Re: [Pharo-dev] From a mooc user: method source with it' take so long in Pharo 5

Squeak caches the opened readonly file(handle). It does not have to reopen the file on every call for reading (readonly).

2016-05-18 19:12 GMT+02:00 stepharo <[hidden email]>:
I am wondering why does the search 'method source with it' take so long in Pharo 5? On my PC, When I select the text 'menu' and search for all 'method source with it', in Squeak 5 it takes 3 seconds. In Pharo 5 it takes 21 seconds.

tinchodias

Re: [Pharo-dev] From a mooc user: method source with it' take so long in Pharo 5

In reply to this post by Eliot Miranda-2

Hi all,

On Sun, May 22, 2016 at 12:26 PM, Eliot Miranda <[hidden email]> wrote:

Hi Henrik,

cc'ing squeak-dev cuz this is relevant to both communities (if and until Pharo takes the unwise decision to double the size of the image by keeping sources in the image).

On May 22, 2016, at 5:01 AM, Henrik Nergaard <[hidden email]> wrote:

There is a method in SourceFileArray #localProcessReadStreamAtFileIndex:atPosition:ifPresent:ifAbsent: which uses a ProccessLocalVariable called ProccessAndSessionLocalSourcesFileArray (see #localProcessReadOnlyCopy). Changing the last line in #readStreamAt:ifPresent:ifAbsent: to use this local process one makes the time to run this snippet:

[ 1 systemNavigation browseMethodsWithSourceString: 'Morph' matchCase: false ] timeProfile

From ~ 10 seconds to ~ 3 seconds (Windows).

However I cannot see that the file handles created by this processLocalVariable to ever be closed, so I suspect those are leaked? In that case there might be the need to implement some “clean up” mechanism for ProccessLocalVariables before they are changed/nilled when a process changes.

Another approach could be to not use a ProccessLocalVariabe at all, but extend the SourceFilesArray class to also hold one read only handle for each of its files, and use these in readStreamAtFileIndex:::… . I guess that it is also necessary then to have semaphore protecting the two last lines such that setting the position in the stream and reading from it cannot be changed by other threads.

I like this approach, but it has a fatal flaw. If one is debugging file access, then as the debugger causes the source of methods to be fetched as the debugger displays them, so the file pointer in the read-only file the access of which one is trying to debug will change under one's feet, and the results will be completely wrong.

An approach which uses a special copy for the debugger doesn't work; one can't debug access to that file for the same reason.

An approach that /should/ work is for the debugger to install its own copy around file-access. This should be recursive. So around every file access in the debugger there would need to be something like

SourceFiles substituteFreshReadOnlyCopiesDuring: [...file access...]

which would remember the current read-only files, evaluate the block using ensure: and have the ensure: block reinstall the previous file.

Does this make sense?

Interesting discussion. We could give a new try on the "local process read-only copy".

It was me who started with it last year in a Pharo sprint, but disabled it due to some errors... and didn't spend more time on it afterwards.

it was reported here: https://pharo.fogbugz.com/f/cases/15782

the reason to disable it: https://pharo.fogbugz.com/f/cases/15781/

none of them is very clear as documentation of the problem.

readStreamAtFileIndex: index atPosition: position ifPresent: presentBlock ifAbsent: absentBlock

               | stream result|

               stream := self readOnlyFileAt: index.

               stream ifNil: [ ^ absentBlock value ].        "sources file not available"

               position > stream size ifTrue: [ ^ absentBlock value ].

               readSema critical: [

                              stream position: position.

                              result := presentBlock value: stream

               ].

               ^ result

Best regards,

Henrik

From: Pharo-dev [[hidden email]] On Behalf Of Nicolai Hess
Sent: Thursday, May 19, 2016 9:31 AM
To: Pharo Development List <[hidden email]>
Subject: Re: [Pharo-dev] From a mooc user: method source with it' take so long in Pharo 5

Squeak caches the opened readonly file(handle). It does not have to reopen the file on every call for reading (readonly).

2016-05-18 19:12 GMT+02:00 stepharo <[hidden email]>:

I am wondering why does the search 'method source with it' take so long in Pharo 5? On my PC, When I select the text 'menu' and search for all 'method source with it', in Squeak 5 it takes 3 seconds. In Pharo 5 it takes 21 seconds.