Git based repositories (package-cache) problem when moving extent to another stone

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

On 10/26/2015 03:03 PM, Mariano Martinez Peck via Glass wrote:
> Dale,
>
> I am finding problems with git-based repositories. Basically, let's
> say I load some stuff from git under the stone called 3.2.9. Then I
> copy the extent and I restore such a extent in another stone, called
> 'otherStone'.
What scripts steps do you use when you"copy and restore in another
stone" ....
>
> I now try to update code in my 'otherStone' and I get an exception
> while loading Seaside from metacello. The problem ends up being that
> in fact it is trying to get the stuff from here:
>
> '/opt/gemstoneAdditions/GsDevKit_home/server/stones/gs_329/logs/github-cache/GsDevKit/Seaside31/v3.1.4.2-gs/GsDevKit-Seaside31-55f1bac/repository'
Ah okay ... I've been working with local clones of all of the projects
(the mode that I think everyone should follow) ... the problem you are
seeing has to do with using github:// repositories ... git repositories
use filetree:// and the expectation is that you are moving extents
within the same $GS_HOME environment ... nonetheless this should work ...
>
> All those files have a different OS owner and file permissiones etc etc.
>
> Of course, if I remove
> /opt/gemstoneAdditions/GsDevKit_home/server/stones/gs_329/logs/github-cache
>   it suddenly starts to work in 'otherStone'.
This seems to be a bug in MCGitHubRepository ... specifically
MCGitHubRepository class>>resetCacheDirectoryIfInvalid will reset the
CacheDirectory if it's no longer "valid" but it's test for "valid" is
that the directory exists ... it should also worry about permissions ...
I guess ... alternatively I could catch an error on access and reset the
CacheDirectory and try again ... I don't know if there is platform
independent protocol in Metacello for doing permission testing ... no
quick fix there, I think ....

In the mean time, you can clear the cache directory manually:

   MCGitHubRepository cacheDirectory: nil.

If you are using $GS_HOME/bin/newExtent or $GS_HOME/bin/createStone with
the `-t` option (which you should be), then newExtent ends up running
the `rebuildServerTode` script:

   $GS_HOME/bin/todeIt $stoneName "script --script=rebuildServerTode"

which itself runs $GS_HOME/sys/local/client/tode-scripts/rebuildSys ...
whose job is to do things like make sure that the freshly copied extent
is adapted correctly to it's new environment so I think that adding this
line to the script would be correct:

   eval `MCGitHubRepository cacheDirectory: nil`

In fact, if you don't get there first, I'll submit a Metacello bug
report, and then add this patch (along with a reference to the Metacello
bug) to rebuildSys. I'm not planning a drop for a day or so, so if you
want me to do the patch sooner, I can slipstream the patch  into master
and then create the bug and comment the patch with my next regular Early
Access drop ...

I might be inclined to think that patch actually belongs there ... when
you move an extent ... all of the disk caches should be cleared ....

Dale

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Submitted a Metacello issue[1]

[1] https://github.com/dalehenrich/metacello-work/issues/375

Dale,

I am finding problems with git-based repositories. Basically, let's say I load some stuff from git under the stone called 3.2.9. Then I copy the extent and I restore such a extent in another stone, called 'otherStone'.

I now try to update code in my 'otherStone' and I get an exception while loading Seaside from metacello. The problem ends up being that in fact it is trying to get the stuff from here:

'/opt/gemstoneAdditions/GsDevKit_home/server/stones/gs_329/logs/github-cache/GsDevKit/Seaside31/v3.1.4.2-gs/GsDevKit-Seaside31-55f1bac/repository'

All those files have a different OS owner and file permissiones etc etc.

Of course, if I remove /opt/gemstoneAdditions/GsDevKit_home/server/stones/gs_329/logs/github-cache   it suddenly starts to work in 'otherStone'.  Before removing package-cache I tried flushing all MC classes, but nothing. Same problem. Only removing package-cache did the trick.

So let me ask..is there an easy way I can force the re-location of pacakge-cache when I move a stone?

Thanks in advance,

--

Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

I'm headed home

If the MCGitHubRepository class>>resetCacheDirectoryIfInvalid  works for a deleted directory (and you have found that that case works), then it should work if th cache is explicitly cleared ...

HOWEVER, I recall that resetCacheDirectoryIfInvalid is triggered during image startup, so perhaps there are additional initialization steps that run at image startup ... perhaps the directory inst vars are nuked I didn't look closely ... anyway take a look down that path and see if you can find the additional inits that might be run at image startup (also be suspicious of SessionTemps being used, since SessionTemps don't survive across logins) ...

Dale

On Mon, Oct 26, 2015 at 7:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

On 10/26/2015 03:03 PM, Mariano Martinez Peck via Glass wrote:

Dale,

I am finding problems with git-based repositories. Basically, let's say I load some stuff from git under the stone called 3.2.9. Then I copy the extent and I restore such a extent in another stone, called 'otherStone'.

What scripts steps do you use when you"copy and restore in another stone" ....

I do a snapshot with tODE then a cp of the extent. 

 

I now try to update code in my 'otherStone' and I get an exception while loading Seaside from metacello. The problem ends up being that in fact it is trying to get the stuff from here:

'/opt/gemstoneAdditions/GsDevKit_home/server/stones/gs_329/logs/github-cache/GsDevKit/Seaside31/v3.1.4.2-gs/GsDevKit-Seaside31-55f1bac/repository'

Ah okay ... I've been working with local clones of all of the projects (the mode that I think everyone should follow) ... the problem you are seeing has to do with using github:// repositories ... git repositories use filetree:// and the expectation is that you are moving extents within the same $GS_HOME environment ... nonetheless this should work ...

Yes, the problem I found is when using github repos, not filetree (at least the error I had), and yes, using same $GS_HOME.

 

All those files have a different OS owner and file permissiones etc etc.

Of course, if I remove /opt/gemstoneAdditions/GsDevKit_home/server/stones/gs_329/logs/github-cache   it suddenly starts to work in 'otherStone'.

This seems to be a bug in MCGitHubRepository ... specifically MCGitHubRepository class>>resetCacheDirectoryIfInvalid will reset the CacheDirectory if it's no longer "valid" but it's test for "valid" is that the directory exists ... it should also worry about permissions ... I guess ... alternatively I could catch an error on access and reset the CacheDirectory and try again ... I don't know if there is platform independent protocol in Metacello for doing permission testing ... no quick fix there, I think ....

Yes, that's exactly the point. In my case, the 'another' stone runs with a different OS user and hence there is a permission issue at file system level.

 

In the mean time, you can clear the cache directory manually:

  MCGitHubRepository cacheDirectory: nil.

If you are using $GS_HOME/bin/newExtent or $GS_HOME/bin/createStone with the `-t` option (which you should be), then newExtent ends up running the `rebuildServerTode` script:

  $GS_HOME/bin/todeIt $stoneName "script --script=rebuildServerTode"

which itself runs $GS_HOME/sys/local/client/tode-scripts/rebuildSys ... whose job is to do things like make sure that the freshly copied extent is adapted correctly to it's new environment so I think that adding this line to the script would be correct:

  eval `MCGitHubRepository cacheDirectory: nil`

Thanks. I just tried it but it seems it is not yet enough. Look at the exception I get:

Inspect MetacelloEnsureFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer: @31 line 39a GoferRepositoryError occurred (error 2710), a ImproperOperation occurred (error 2085), Expected nil to be a Boolean./

--------------------

.              -> a GoferRepositoryError occurred (error 2710), a ImproperOperation occurred (error 2085), Expected nil to be a Boolean.

..             -> aMetacelloEnsureFetchingMCSpecLoader(explicit load : 1.6 [ConfigurationOfIAM])

(class)@       -> GoferRepositoryError

(oop)@         -> 1767077889

currGsHandler@ -> nil

gsArgs@        -> nil

gsDetails@     -> 'a ImproperOperation occurred (error 2085), Expected nil to be a Boolean.'

gsNumber@      -> 2710

gsReason@      -> nil

gsResumable@   -> false

gsStack@       -> nil

gsTrappable@   -> true

messageText@   -> 'a GoferRepositoryError occurred (error 2710), a ImproperOperation occurred (error 2085), Expected nil to be a Boolean.'

repository@    -> aMCGitHubRepository(github://GsDevKit/Seaside31:v3.1.4.2-gs/repository)

tag@           -> nil

And when I inspect the 'repository' I get this:

Inspect MetacelloEnsureFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer: @31 line 39a GoferRepositoryError occurred (error 2710), a ImproperOperation occurred (error 2085), Expected nil to be a Boolean./aMCGitHubRepository(github://GsDevKit/Seaside31:v3.1.4.2-gs/repository)/

--------------------

.                      -> aMCGitHubRepository(github://GsDevKit/Seaside31:v3.1.4.2-gs/repository)

..                     -> a GoferRepositoryError occurred (error 2710), a ImproperOperation occurred (error 2085), Expected nil to be a Boolean.

(class)@               -> MCGitHubRepository

(oop)@                 -> 1226322689

(committed)@           -> true

cache@                 -> nil

creationTemplate@      -> nil

directory@             -> aServerFileDirectory

projectPath@           -> 'GsDevKit/Seaside31'

projectVersion@        -> 'v3.1.4.2-gs'

projectVersionPattern@ -> '3.1.?'

readonly@              -> true

repoPath@              -> 'repository'

repositoryProperties@  -> aDictionary( 'packageExtension'->'.package', 'propertyFileExtension'->'.json')

storeDiffs@            -> nil

And as you can see the problematic is that 'directory' instVar.  if I send #pathName to it, I get:

'/opt/gemstoneAdditions/GsDevKit_home/server/stones/gs_329/logs/github-cache/GsDevKit/Seaside31/v3.1.4.2-gs/GsDevKit-Seaside31-55f1bac/repository'

I tried sending #flushCache to the repository instance, but not luck..the 'directory' instVar still there...

Note that #flushCache of github class DOES nil the 'directory' instVar. However, this is funny:

MCGitHubRepository cacheDirectory: nil.

MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ].

MCRepositoryGroup default repositoriesDo: [:each | (each projectVersion = 'v3.1.4.1-gs') ifTrue: [Halt halt] ].

And then it halts... 'each' continues to have again the wrong value in 'directory'. Shouldn't this be nil?    Of course I made sure the #flushCache was really sent to the repository. 

I don't get it. 

 

In fact, if you don't get there first, I'll submit a Metacello bug report, and then add this patch (along with a reference to the Metacello bug) to rebuildSys. I'm not planning a drop for a day or so, so if you want me to do the patch sooner, I can slipstream the patch  into master and then create the bug and comment the patch with my next regular Early Access drop ...

I might be inclined to think that patch actually belongs there ... when you move an extent ... all of the disk caches should be cleared ....

Dale

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

--

Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Ah ... good analysis see if the following change to FileDirectory>>exists gives a better error:

exists
  | result |
  result := GsFile _exists: self pathString onClient: self onClient.
  result ifNil: [ self error: 'Disk erro: ' , GsFile lastErrorString ].
  ^ result

If so, I've got it queued up as a change on the glassdb/glass dev branch ...

Dale

Dale, 

As you know, the original error was:   a ImproperOperation occurred (error 2085), Expected nil to be a Boolean

The problem is the code like this "MCFileTreeFileUtils current directoryExists: " . The real problem is that #directoryExists: is like this:

---------

MCFileTreeFileDirectoryUtils class >> directoryExists: aDirectory

 ^ aDirectory exists.

------ 

However, if the directory EXISTS but you do not have permissions, you get nil, hence the "Expected nil to be a Boolean"

Of course, an example of that code is 

---------

MCGitHubRepository class >> resetCacheDirectoryIfInvalid

  "use class var to survive upgrade when MCGitHubRepository moved to subclass of MCGitBasedRepository"

  "Reset if invalid"

  CacheDirectory notNil

    and: [ 

      (MCFileTreeFileUtils current directoryExists: CacheDirectory)

        ifFalse: [ CacheDirectory := nil ] ]

----------

But again, this is only ONE of the senders. There are others of above kind of code (sending #directoryExists: and then sending an ifFalse: or isTrue:)

If the answering nil when not correct permisions WOULD be general/portable, then I think a solution could be to create MCFileTreeFileDirectoryUtils class >> directoryExistsAndValid:  that does:

---------

directoryExistsAndValid: aDirectory

  ^ aDirectory exists ifNil: [false] 

--------

Anyway, now the problem is clear. Maybe you can come with a better approach?

Cheers, 

On Mon, Oct 26, 2015 at 10:13 PM, Mariano Martinez Peck <[hidden email]> wrote:

Dale,

Do not continue loosing time until my next email. I found the problem/solution but I don't have the time to explain and detail until tomorrow.

Thanks!

On Mon, Oct 26, 2015 at 9:22 PM, Dale Henrichs <[hidden email]> wrote:

I'm headed home

If the MCGitHubRepository class>>resetCacheDirectoryIfInvalid  works for a deleted directory (and you have found that that case works), then it should work if th cache is explicitly cleared ...

HOWEVER, I recall that resetCacheDirectoryIfInvalid is triggered during image startup, so perhaps there are additional initialization steps that run at image startup ... perhaps the directory inst vars are nuked I didn't look closely ... anyway take a look down that path and see if you can find the additional inits that might be run at image startup (also be suspicious of SessionTemps being used, since SessionTemps don't survive across logins) ...

Dale

On 10/26/2015 05:09 PM, Mariano Martinez Peck wrote:

On Mon, Oct 26, 2015 at 7:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

On 10/26/2015 03:03 PM, Mariano Martinez Peck via Glass wrote:

Dale,

I am finding problems with git-based repositories. Basically, let's say I load some stuff from git under the stone called 3.2.9. Then I copy the extent and I restore such a extent in another stone, called 'otherStone'.

What scripts steps do you use when you"copy and restore in another stone" ....

I do a snapshot with tODE then a cp of the extent. 

 

I now try to update code in my 'otherStone' and I get an exception while loading Seaside from metacello. The problem ends up being that in fact it is trying to get the stuff from here:

'/opt/gemstoneAdditions/GsDevKit_home/server/stones/gs_329/logs/github-cache/GsDevKit/Seaside31/v3.1.4.2-gs/GsDevKit-Seaside31-55f1bac/repository'

Ah okay ... I've been working with local clones of all of the projects (the mode that I think everyone should follow) ... the problem you are seeing has to do with using github:// repositories ... git repositories use filetree:// and the expectation is that you are moving extents within the same $GS_HOME environment ... nonetheless this should work ...

Yes, the problem I found is when using github repos, not filetree (at least the error I had), and yes, using same $GS_HOME.

 

All those files have a different OS owner and file permissiones etc etc.

Of course, if I remove /opt/gemstoneAdditions/GsDevKit_home/server/stones/gs_329/logs/github-cache   it suddenly starts to work in 'otherStone'.

This seems to be a bug in MCGitHubRepository ... specifically MCGitHubRepository class>>resetCacheDirectoryIfInvalid will reset the CacheDirectory if it's no longer "valid" but it's test for "valid" is that the directory exists ... it should also worry about permissions ... I guess ... alternatively I could catch an error on access and reset the CacheDirectory and try again ... I don't know if there is platform independent protocol in Metacello for doing permission testing ... no quick fix there, I think ....

Yes, that's exactly the point. In my case, the 'another' stone runs with a different OS user and hence there is a permission issue at file system level.

 

In the mean time, you can clear the cache directory manually:

  MCGitHubRepository cacheDirectory: nil.

If you are using $GS_HOME/bin/newExtent or $GS_HOME/bin/createStone with the `-t` option (which you should be), then newExtent ends up running the `rebuildServerTode` script:

  $GS_HOME/bin/todeIt $stoneName "script --script=rebuildServerTode"

which itself runs $GS_HOME/sys/local/client/tode-scripts/rebuildSys ... whose job is to do things like make sure that the freshly copied extent is adapted correctly to it's new environment so I think that adding this line to the script would be correct:

  eval `MCGitHubRepository cacheDirectory: nil`

Thanks. I just tried it but it seems it is not yet enough. Look at the exception I get:

Inspect MetacelloEnsureFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer: @31 line 39a GoferRepositoryError occurred (error 2710), a ImproperOperation occurred (error 2085), Expected nil to be a Boolean./

--------------------

.              -> a GoferRepositoryError occurred (error 2710), a ImproperOperation occurred (error 2085), Expected nil to be a Boolean.

..             -> aMetacelloEnsureFetchingMCSpecLoader(explicit load : 1.6 [ConfigurationOfIAM])

(class)@       -> GoferRepositoryError

(oop)@         -> 1767077889

currGsHandler@ -> nil

gsArgs@        -> nil

gsDetails@     -> 'a ImproperOperation occurred (error 2085), Expected nil to be a Boolean.'

gsNumber@      -> 2710

gsReason@      -> nil

gsResumable@   -> false

gsStack@       -> nil

gsTrappable@   -> true

messageText@   -> 'a GoferRepositoryError occurred (error 2710), a ImproperOperation occurred (error 2085), Expected nil to be a Boolean.'

repository@    -> aMCGitHubRepository(github://GsDevKit/Seaside31:v3.1.4.2-gs/repository)

tag@           -> nil

And when I inspect the 'repository' I get this:

Inspect MetacelloEnsureFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer: @31 line 39a GoferRepositoryError occurred (error 2710), a ImproperOperation occurred (error 2085), Expected nil to be a Boolean./aMCGitHubRepository(github://GsDevKit/Seaside31:v3.1.4.2-gs/repository)/

--------------------

.                      -> aMCGitHubRepository(github://GsDevKit/Seaside31:v3.1.4.2-gs/repository)

..                     -> a GoferRepositoryError occurred (error 2710), a ImproperOperation occurred (error 2085), Expected nil to be a Boolean.

(class)@               -> MCGitHubRepository

(oop)@                 -> 1226322689

(committed)@           -> true

cache@                 -> nil

creationTemplate@      -> nil

directory@             -> aServerFileDirectory

projectPath@           -> 'GsDevKit/Seaside31'

projectVersion@        -> 'v3.1.4.2-gs'

projectVersionPattern@ -> '3.1.?'

readonly@              -> true

repoPath@              -> 'repository'

repositoryProperties@  -> aDictionary( 'packageExtension'->'.package', 'propertyFileExtension'->'.json')

storeDiffs@            -> nil

And as you can see the problematic is that 'directory' instVar.  if I send #pathName to it, I get:

'/opt/gemstoneAdditions/GsDevKit_home/server/stones/gs_329/logs/github-cache/GsDevKit/Seaside31/v3.1.4.2-gs/GsDevKit-Seaside31-55f1bac/repository'

I tried sending #flushCache to the repository instance, but not luck..the 'directory' instVar still there...

Note that #flushCache of github class DOES nil the 'directory' instVar. However, this is funny:

MCGitHubRepository cacheDirectory: nil.

MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ].

MCRepositoryGroup default repositoriesDo: [:each | (each projectVersion = 'v3.1.4.1-gs') ifTrue: [Halt halt] ].

And then it halts... 'each' continues to have again the wrong value in 'directory'. Shouldn't this be nil?    Of course I made sure the #flushCache was really sent to the repository. 

I don't get it. 

 

In fact, if you don't get there first, I'll submit a Metacello bug report, and then add this patch (along with a reference to the Metacello bug) to rebuildSys. I'm not planning a drop for a day or so, so if you want me to do the patch sooner, I can slipstream the patch  into master and then create the bug and comment the patch with my next regular Early Access drop ...

I might be inclined to think that patch actually belongs there ... when you move an extent ... all of the disk caches should be cleared ....

Dale

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

--

Mariano
http://marianopeck.wordpress.com

--

Mariano
http://marianopeck.wordpress.com

--

Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Here's the bug report[1]

[1] https://github.com/GsDevKit/GsDevKit/issues/79

Ah ... good analysis see if the following change to FileDirectory>>exists gives a better error:

exists
  | result |
  result := GsFile _exists: self pathString onClient: self onClient.
  result ifNil: [ self error: 'Disk erro: ' , GsFile lastErrorString ].
  ^ result

If so, I've got it queued up as a change on the glassdb/glass dev branch ...

Dale

On 10/26/2015 06:47 PM, Mariano Martinez Peck wrote:

Dale, 

As you know, the original error was:   a ImproperOperation occurred (error 2085), Expected nil to be a Boolean

The problem is the code like this "MCFileTreeFileUtils current directoryExists: " . The real problem is that #directoryExists: is like this:

---------

MCFileTreeFileDirectoryUtils class >> directoryExists: aDirectory

 ^ aDirectory exists.

------ 

However, if the directory EXISTS but you do not have permissions, you get nil, hence the "Expected nil to be a Boolean"

Of course, an example of that code is 

---------

MCGitHubRepository class >> resetCacheDirectoryIfInvalid

  "use class var to survive upgrade when MCGitHubRepository moved to subclass of MCGitBasedRepository"

  "Reset if invalid"

  CacheDirectory notNil

    and: [ 

      (MCFileTreeFileUtils current directoryExists: CacheDirectory)

        ifFalse: [ CacheDirectory := nil ] ]

----------

But again, this is only ONE of the senders. There are others of above kind of code (sending #directoryExists: and then sending an ifFalse: or isTrue:)

If the answering nil when not correct permisions WOULD be general/portable, then I think a solution could be to create MCFileTreeFileDirectoryUtils class >> directoryExistsAndValid:  that does:

---------

directoryExistsAndValid: aDirectory

  ^ aDirectory exists ifNil: [false] 

--------

Anyway, now the problem is clear. Maybe you can come with a better approach?

Cheers, 

On Mon, Oct 26, 2015 at 10:13 PM, Mariano Martinez Peck <[hidden email]> wrote:

Dale,

Do not continue loosing time until my next email. I found the problem/solution but I don't have the time to explain and detail until tomorrow.

Thanks!

On Mon, Oct 26, 2015 at 9:22 PM, Dale Henrichs <[hidden email]> wrote:

I'm headed home

If the MCGitHubRepository class>>resetCacheDirectoryIfInvalid  works for a deleted directory (and you have found that that case works), then it should work if th cache is explicitly cleared ...

HOWEVER, I recall that resetCacheDirectoryIfInvalid is triggered during image startup, so perhaps there are additional initialization steps that run at image startup ... perhaps the directory inst vars are nuked I didn't look closely ... anyway take a look down that path and see if you can find the additional inits that might be run at image startup (also be suspicious of SessionTemps being used, since SessionTemps don't survive across logins) ...

Dale

On 10/26/2015 05:09 PM, Mariano Martinez Peck wrote:

On Mon, Oct 26, 2015 at 7:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

On 10/26/2015 03:03 PM, Mariano Martinez Peck via Glass wrote:

Dale,

I am finding problems with git-based repositories. Basically, let's say I load some stuff from git under the stone called 3.2.9. Then I copy the extent and I restore such a extent in another stone, called 'otherStone'.

What scripts steps do you use when you"copy and restore in another stone" ....

I do a snapshot with tODE then a cp of the extent. 

 

I now try to update code in my 'otherStone' and I get an exception while loading Seaside from metacello. The problem ends up being that in fact it is trying to get the stuff from here:

'/opt/gemstoneAdditions/GsDevKit_home/server/stones/gs_329/logs/github-cache/GsDevKit/Seaside31/v3.1.4.2-gs/GsDevKit-Seaside31-55f1bac/repository'

Ah okay ... I've been working with local clones of all of the projects (the mode that I think everyone should follow) ... the problem you are seeing has to do with using github:// repositories ... git repositories use filetree:// and the expectation is that you are moving extents within the same $GS_HOME environment ... nonetheless this should work ...

Yes, the problem I found is when using github repos, not filetree (at least the error I had), and yes, using same $GS_HOME.

 

All those files have a different OS owner and file permissiones etc etc.

Of course, if I remove /opt/gemstoneAdditions/GsDevKit_home/server/stones/gs_329/logs/github-cache   it suddenly starts to work in 'otherStone'.

This seems to be a bug in MCGitHubRepository ... specifically MCGitHubRepository class>>resetCacheDirectoryIfInvalid will reset the CacheDirectory if it's no longer "valid" but it's test for "valid" is that the directory exists ... it should also worry about permissions ... I guess ... alternatively I could catch an error on access and reset the CacheDirectory and try again ... I don't know if there is platform independent protocol in Metacello for doing permission testing ... no quick fix there, I think ....

Yes, that's exactly the point. In my case, the 'another' stone runs with a different OS user and hence there is a permission issue at file system level.

 

In the mean time, you can clear the cache directory manually:

  MCGitHubRepository cacheDirectory: nil.

If you are using $GS_HOME/bin/newExtent or $GS_HOME/bin/createStone with the `-t` option (which you should be), then newExtent ends up running the `rebuildServerTode` script:

  $GS_HOME/bin/todeIt $stoneName "script --script=rebuildServerTode"

which itself runs $GS_HOME/sys/local/client/tode-scripts/rebuildSys ... whose job is to do things like make sure that the freshly copied extent is adapted correctly to it's new environment so I think that adding this line to the script would be correct:

  eval `MCGitHubRepository cacheDirectory: nil`

Thanks. I just tried it but it seems it is not yet enough. Look at the exception I get:

Inspect MetacelloEnsureFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer: @31 line 39a GoferRepositoryError occurred (error 2710), a ImproperOperation occurred (error 2085), Expected nil to be a Boolean./

--------------------

.              -> a GoferRepositoryError occurred (error 2710), a ImproperOperation occurred (error 2085), Expected nil to be a Boolean.

..             -> aMetacelloEnsureFetchingMCSpecLoader(explicit load : 1.6 [ConfigurationOfIAM])

(class)@       -> GoferRepositoryError

(oop)@         -> 1767077889

currGsHandler@ -> nil

gsArgs@        -> nil

gsDetails@     -> 'a ImproperOperation occurred (error 2085), Expected nil to be a Boolean.'

gsNumber@      -> 2710

gsReason@      -> nil

gsResumable@   -> false

gsStack@       -> nil

gsTrappable@   -> true

messageText@   -> 'a GoferRepositoryError occurred (error 2710), a ImproperOperation occurred (error 2085), Expected nil to be a Boolean.'

repository@    -> aMCGitHubRepository(github://GsDevKit/Seaside31:v3.1.4.2-gs/repository)

tag@           -> nil

And when I inspect the 'repository' I get this:

Inspect MetacelloEnsureFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer: @31 line 39a GoferRepositoryError occurred (error 2710), a ImproperOperation occurred (error 2085), Expected nil to be a Boolean./aMCGitHubRepository(github://GsDevKit/Seaside31:v3.1.4.2-gs/repository)/

--------------------

.                      -> aMCGitHubRepository(github://GsDevKit/Seaside31:v3.1.4.2-gs/repository)

..                     -> a GoferRepositoryError occurred (error 2710), a ImproperOperation occurred (error 2085), Expected nil to be a Boolean.

(class)@               -> MCGitHubRepository

(oop)@                 -> 1226322689

(committed)@           -> true

cache@                 -> nil

creationTemplate@      -> nil

directory@             -> aServerFileDirectory

projectPath@           -> 'GsDevKit/Seaside31'

projectVersion@        -> 'v3.1.4.2-gs'

projectVersionPattern@ -> '3.1.?'

readonly@              -> true

repoPath@              -> 'repository'

repositoryProperties@  -> aDictionary( 'packageExtension'->'.package', 'propertyFileExtension'->'.json')

storeDiffs@            -> nil

And as you can see the problematic is that 'directory' instVar.  if I send #pathName to it, I get:

'/opt/gemstoneAdditions/GsDevKit_home/server/stones/gs_329/logs/github-cache/GsDevKit/Seaside31/v3.1.4.2-gs/GsDevKit-Seaside31-55f1bac/repository'

I tried sending #flushCache to the repository instance, but not luck..the 'directory' instVar still there...

Note that #flushCache of github class DOES nil the 'directory' instVar. However, this is funny:

MCGitHubRepository cacheDirectory: nil.

MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ].

MCRepositoryGroup default repositoriesDo: [:each | (each projectVersion = 'v3.1.4.1-gs') ifTrue: [Halt halt] ].

And then it halts... 'each' continues to have again the wrong value in 'directory'. Shouldn't this be nil?    Of course I made sure the #flushCache was really sent to the repository. 

I don't get it. 

 

In fact, if you don't get there first, I'll submit a Metacello bug report, and then add this patch (along with a reference to the Metacello bug) to rebuildSys. I'm not planning a drop for a day or so, so if you want me to do the patch sooner, I can slipstream the patch  into master and then create the bug and comment the patch with my next regular Early Access drop ...

I might be inclined to think that patch actually belongs there ... when you move an extent ... all of the disk caches should be cleared ....

Dale

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

--

Mariano
http://marianopeck.wordpress.com

--

Mariano
http://marianopeck.wordpress.com

--

Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Dale, 

While that more robust #exists does help in the sense that the user gets a much better error message, I think it's not enough.  So now I get this message:

...RETRY->BaselineOfSeaside3

...RETRY->BaselineOfSeaside3

gofer repository error: 'GoferRepositoryError: UserDefinedError: Disk error:  for path: /opt/gemstoneAdditions/GsDevKit_home/server/stones/gs_329/logs/github-cache/GsDevKit/Seaside31/v3.1.4.2-gs/GsDevKit-Seaside31-55f1bac/repository'...ignoring

...FAILED->BaselineOfSeaside3

Right, but aren't we trying to figure out why the load works when you remove the offending directory but does not work when you explicitly set the cache directory to nil:

  MCGitHubRepository cacheDirectory: nil

... when an extent is copied to a different stone directory there are potentially a number of places that need to be fixed ... basically anywhere that is going to look for things on disk ... the Smalltalk code that gets executed by $GS_HOME/bin/newExtent currently cleans up the tODE directory dependencies and it should also reset the github-cache directory (whether or not errors are going to be thrown because of permission errors)...

The $GS_HOME/sys/{default|local}/client/tode-scripts/rebuildSys script is intended to clean up these directory/stone dependencies for tODE and GLASS and you are expected to override with application-specific clean up code ....

Soooo is the above error happening before or after you have cleared MCGitHubRepository class>>cacheDirectory?

I am under the impression that:

  MCGitHubRepository cacheDirectory: nil

isn't working for you ... am I missing something?

But my code still fail to load. There are a couple of places that should be changed. Search for the string "MCFileTreeFileUtils current directoryExists:"

And you will find 

--------------------

  MCGitHubRepository class>>resetCacheDirectoryIfInvalid

 MCGitBasedNetworkRepository>>directory

 MCGitBasedNetworkRepository class>>resetCacheDirectoryIfInvalid

MCFileTreeRepository>>flushCache

I think those guys should handle the error and on error, the flush or reset anyway. To avoid this (adding error handling in all those places), I suggested adding:

MCFileTreeFileDirectoryUtils class >> directoryExistsAndValid: aDirectory

  ^ aDirectory exists on: Error do: [  false ] 

And so, the above 4 methods should actually send #directoryExistsAndValid:  rather than #directoryExists

I'm not sure that I want to try to bullet-proof this chunk of code ... I am worried about what happens if you don't have write permission in the first place or the disk is full or ???? there are too many possible error conditions that will either continue to fail or may mask the original problem if we just handle the errors ...

Besides it seems to me that the problem here is that MCGitBasedNetworkRepository class>>resetCacheDirectoryIfInvalid (which sets the cacheDirectory nil) won't work because if I understand things correctly `MCGitHubRepository cacheDirectory: nil` doesn't work ... or am I missing something here ....

Dale

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

On Tue, Oct 27, 2015 at 3:22 PM, Dale Henrichs <[hidden email]> wrote:

On 10/27/2015 10:50 AM, Mariano Martinez Peck wrote:

Dale, 

While that more robust #exists does help in the sense that the user gets a much better error message, I think it's not enough.  So now I get this message:

...RETRY->BaselineOfSeaside3

...RETRY->BaselineOfSeaside3

gofer repository error: 'GoferRepositoryError: UserDefinedError: Disk error:  for path: /opt/gemstoneAdditions/GsDevKit_home/server/stones/gs_329/logs/github-cache/GsDevKit/Seaside31/v3.1.4.2-gs/GsDevKit-Seaside31-55f1bac/repository'...ignoring

...FAILED->BaselineOfSeaside3

Right, but aren't we trying to figure out why the load works when you remove the offending directory

Yes, my load works when we remove the offending directory. 

 

but does not work when you explicitly set the cache directory to nil:

  MCGitHubRepository cacheDirectory: nil

Yes, this does not fix the problem. What I found out is that besides that, I could do:

MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ].

That WOULD have fixed my problem. However #flushCache fails too because of the same error. So we are not even able to #flushCahe with current code.

Ah this is the piece that I was missing ... I had misunderstood that this did not work either ... and I thought you were trying to track down why this did not clear the cache ... but I understand now that when you run this code, you get a permissions related error and so you "cannot flush the caches"

  

... when an extent is copied to a different stone directory there are potentially a number of places that need to be fixed ... basically anywhere that is going to look for things on disk ... the Smalltalk code that gets executed by $GS_HOME/bin/newExtent currently cleans up the tODE directory dependencies and it should also reset the github-cache directory (whether or not errors are going to be thrown because of permission errors)...

The $GS_HOME/sys/{default|local}/client/tode-scripts/rebuildSys script is intended to clean up these directory/stone dependencies for tODE and GLASS and you are expected to override with application-specific clean up code ....

Soooo is the above error happening before or after you have cleared MCGitHubRepository class>>cacheDirectory?

After. It has no effect for me running such code.

 

I am under the impression that:

  MCGitHubRepository cacheDirectory: nil

isn't working for you ... am I missing something?

You are not missing anything. That is not working for me and I don't understand why it would. I am under the impression that I would also need:

MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ].

No?

 

But my code still fail to load. There are a couple of places that should be changed. Search for the string "MCFileTreeFileUtils current directoryExists:"

And you will find 

--------------------

  MCGitHubRepository class>>resetCacheDirectoryIfInvalid

 MCGitBasedNetworkRepository>>directory

 MCGitBasedNetworkRepository class>>resetCacheDirectoryIfInvalid

MCFileTreeRepository>>flushCache

I think those guys should handle the error and on error, the flush or reset anyway. To avoid this (adding error handling in all those places), I suggested adding:

MCFileTreeFileDirectoryUtils class >> directoryExistsAndValid: aDirectory

  ^ aDirectory exists on: Error do: [  false ] 

And so, the above 4 methods should actually send #directoryExistsAndValid:  rather than #directoryExists

I'm not sure that I want to try to bullet-proof this chunk of code ... I am worried about what happens if you don't have write permission in the first place or the disk is full or ???? there are too many possible error conditions that will either continue to fail or may mask the original problem if we just handle the errors ...

Besides it seems to me that the problem here is that MCGitBasedNetworkRepository class>>resetCacheDirectoryIfInvalid (which sets the cacheDirectory nil) won't work because if I understand things correctly `MCGitHubRepository cacheDirectory: nil` doesn't work ... or am I missing something here ....

Problem is that even if I do  `MCGitHubRepository cacheDirectory: nil`  the MCGitHubRepository instances still have the offending directory as part of the instvar 'directory' and so the MC load fails. That's why I think I need this code:

MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ].

Which effectively clears the 'directory' from the MCGitHubRepository instances. But..guess what? I cannot flush, because:

MCGitBasedNetworkRepository >> flushCache

  "the directory acts like a cache since we download the directory from a git-based repository (github, bitbucket, etc.)"

  super flushCache.

  self class flushDownloadCache.

  directory := nil

that method calls super and so this is executed:

MCFileTreeRepository >> flushCache

  "force properties to be reread ... if the directory exists, otherwise let nature

   take it's course"

  super flushCache.

  directory

    ifNotNil: [ 

      (MCFileTreeFileUtils current directoryExists: directory)   ->>> this will throw error now

        ifTrue: [ 

          repositoryProperties := nil.

          self repositoryProperties ] ]

So...to conclude, THIS piece of code does seem to workaround my problem:

MCGitHubRepository cacheDirectory: nil.

System commit. 

MCGitBasedNetworkRepository allSubInstances do: [ :rep | 

      " cannot send #directory: becaue it really expects a directory instance. 

      By puttin a nil in 'directory' we are able to run #flushCache. See MCFileTreeRepository >> flushCache.  for more details  "

      rep instVarNamed: 'directory' put: nil.

      rep flushCache ].

System commit. 

I didn't realize the bit about "would work except for error" ...

No I have a good picture and I do think that MCFileTreeRepository >> flushCache can and should be fixed[1] ... we should be able to unconditionally clear the cache in the fact of an error. Here's my proposed fix to MCGitBasedNetworkRepository >> flushCache from the bug report[1]:

flushCache
  "the directory acts like a cache since we download the directory from a git-based repository (github, bitbucket, etc.)"

  [super flushCache] on: Error do: [:ignored | "perhaps dump something to Transcript?" ].
  self class flushDownloadCache.
  directory := nil

Can you test this and let me know if it solves your problem? ... if so I'll push out a new version of Metacello with the fix ... I've already patched GsDevKit_home[2] to include the flushCache logic in rebuildSys ....

I'll be interested to hear about any other issues you might have with copied extents ... typically there aren't a lot of places where the GLASS1 code relies on disk ... tODE does and I think I've got that covered in rebuildSys ... obviously the github repos also rely opn disk caches ... but I don't think that there are any others ....

Thanks for straightening me out!

Dale

[1] https://github.com/dalehenrich/metacello-work/issues/376
[2] https://github.com/GsDevKit/GsDevKit_home/commit/29ec9a6d0067e409aa09604c8baa1ba36b6bf998



_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

On Tue, Oct 27, 2015 at 4:39 PM, Dale Henrichs <[hidden email]> wrote:

On 10/27/2015 11:56 AM, Mariano Martinez Peck wrote:

On Tue, Oct 27, 2015 at 3:22 PM, Dale Henrichs <[hidden email]> wrote:

On 10/27/2015 10:50 AM, Mariano Martinez Peck wrote:

Dale, 

While that more robust #exists does help in the sense that the user gets a much better error message, I think it's not enough.  So now I get this message:

...RETRY->BaselineOfSeaside3

...RETRY->BaselineOfSeaside3

gofer repository error: 'GoferRepositoryError: UserDefinedError: Disk error:  for path: /opt/gemstoneAdditions/GsDevKit_home/server/stones/gs_329/logs/github-cache/GsDevKit/Seaside31/v3.1.4.2-gs/GsDevKit-Seaside31-55f1bac/repository'...ignoring

...FAILED->BaselineOfSeaside3

Right, but aren't we trying to figure out why the load works when you remove the offending directory

Yes, my load works when we remove the offending directory. 

 

but does not work when you explicitly set the cache directory to nil:

  MCGitHubRepository cacheDirectory: nil

Yes, this does not fix the problem. What I found out is that besides that, I could do:

MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ].

That WOULD have fixed my problem. However #flushCache fails too because of the same error. So we are not even able to #flushCahe with current code.

Ah this is the piece that I was missing ... I had misunderstood that this did not work either ... and I thought you were trying to track down why this did not clear the cache ... but I understand now that when you run this code, you get a permissions related error and so you "cannot flush the caches"

Yes. So there were 2 things. First, the  ` MCGitHubRepository cacheDirectory: nil` as not enough as I also needed to #flushCache instances.

Second, was that I was not even able to flush caches because of permissions related errors.

  

... when an extent is copied to a different stone directory there are potentially a number of places that need to be fixed ... basically anywhere that is going to look for things on disk ... the Smalltalk code that gets executed by $GS_HOME/bin/newExtent currently cleans up the tODE directory dependencies and it should also reset the github-cache directory (whether or not errors are going to be thrown because of permission errors)...

The $GS_HOME/sys/{default|local}/client/tode-scripts/rebuildSys script is intended to clean up these directory/stone dependencies for tODE and GLASS and you are expected to override with application-specific clean up code ....

Soooo is the above error happening before or after you have cleared MCGitHubRepository class>>cacheDirectory?

After. It has no effect for me running such code.

 

I am under the impression that:

  MCGitHubRepository cacheDirectory: nil

isn't working for you ... am I missing something?

You are not missing anything. That is not working for me and I don't understand why it would. I am under the impression that I would also need:

MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ].

No?

 

But my code still fail to load. There are a couple of places that should be changed. Search for the string "MCFileTreeFileUtils current directoryExists:"

And you will find 

--------------------

  MCGitHubRepository class>>resetCacheDirectoryIfInvalid

 MCGitBasedNetworkRepository>>directory

 MCGitBasedNetworkRepository class>>resetCacheDirectoryIfInvalid

MCFileTreeRepository>>flushCache

I think those guys should handle the error and on error, the flush or reset anyway. To avoid this (adding error handling in all those places), I suggested adding:

MCFileTreeFileDirectoryUtils class >> directoryExistsAndValid: aDirectory

  ^ aDirectory exists on: Error do: [  false ] 

And so, the above 4 methods should actually send #directoryExistsAndValid:  rather than #directoryExists

I'm not sure that I want to try to bullet-proof this chunk of code ... I am worried about what happens if you don't have write permission in the first place or the disk is full or ???? there are too many possible error conditions that will either continue to fail or may mask the original problem if we just handle the errors ...

Besides it seems to me that the problem here is that MCGitBasedNetworkRepository class>>resetCacheDirectoryIfInvalid (which sets the cacheDirectory nil) won't work because if I understand things correctly `MCGitHubRepository cacheDirectory: nil` doesn't work ... or am I missing something here ....

Problem is that even if I do  `MCGitHubRepository cacheDirectory: nil`  the MCGitHubRepository instances still have the offending directory as part of the instvar 'directory' and so the MC load fails. That's why I think I need this code:

MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ].

Which effectively clears the 'directory' from the MCGitHubRepository instances. But..guess what? I cannot flush, because:

MCGitBasedNetworkRepository >> flushCache

  "the directory acts like a cache since we download the directory from a git-based repository (github, bitbucket, etc.)"

  super flushCache.

  self class flushDownloadCache.

  directory := nil

that method calls super and so this is executed:

MCFileTreeRepository >> flushCache

  "force properties to be reread ... if the directory exists, otherwise let nature

   take it's course"

  super flushCache.

  directory

    ifNotNil: [ 

      (MCFileTreeFileUtils current directoryExists: directory)   ->>> this will throw error now

        ifTrue: [ 

          repositoryProperties := nil.

          self repositoryProperties ] ]

So...to conclude, THIS piece of code does seem to workaround my problem:

MCGitHubRepository cacheDirectory: nil.

System commit. 

MCGitBasedNetworkRepository allSubInstances do: [ :rep | 

      " cannot send #directory: becaue it really expects a directory instance. 

      By puttin a nil in 'directory' we are able to run #flushCache. See MCFileTreeRepository >> flushCache.  for more details  "

      rep instVarNamed: 'directory' put: nil.

      rep flushCache ].

System commit. 

I didn't realize the bit about "would work except for error" ...

No I have a good picture and I do think that MCFileTreeRepository >> flushCache can and should be fixed[1] ... we should be able to unconditionally clear the cache in the fact of an error. Here's my proposed fix to MCGitBasedNetworkRepository >> flushCache from the bug report[1]:

flushCache
  "the directory acts like a cache since we download the directory from a git-based repository (github, bitbucket, etc.)"

  [super flushCache] on: Error do: [:ignored | "perhaps dump something to Transcript?" ].
  self class flushDownloadCache.
  directory := nil

Yes, I would dump the error into the transcript.

Okay here's a modified patch:

flushCache
  "the directory acts like a cache since we download the directory from a git-based repository (github, bitbucket, etc.)"

  [ super flushCache ]
    on: Error
    do: [ :ex |
      Transcript
        cr;
        show:
            'Error for: ' , self description printString , ' during flushCache: '
                , ex description printString ].
  self class flushDownloadCache.
  directory := nil


Could you test this one as well? ... There is still some exposure to permission errors for the superclass (MCFileTreeRepository), but in this case we're not dealing with a cache but the repo itself ... which is more serious and not really repairable by an automatic method ... there is a `project rehome` command that let's you point a different fileTree repo and that should be the solution for this particular case ...

I'll start the process of pushing this fix to Metacello ...

Dale

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Okay ... I'll ship it! ... thx

Yes, it does work too. 

On Tue, Oct 27, 2015 at 6:48 PM, Dale Henrichs <[hidden email]> wrote:

On 10/27/2015 01:35 PM, Mariano Martinez Peck wrote:

On Tue, Oct 27, 2015 at 4:39 PM, Dale Henrichs <[hidden email]> wrote:

On 10/27/2015 11:56 AM, Mariano Martinez Peck wrote:

On Tue, Oct 27, 2015 at 3:22 PM, Dale Henrichs <[hidden email]> wrote:

On 10/27/2015 10:50 AM, Mariano Martinez Peck wrote:

Dale, 

While that more robust #exists does help in the sense that the user gets a much better error message, I think it's not enough.  So now I get this message:

...RETRY->BaselineOfSeaside3

...RETRY->BaselineOfSeaside3

gofer repository error: 'GoferRepositoryError: UserDefinedError: Disk error:  for path: /opt/gemstoneAdditions/GsDevKit_home/server/stones/gs_329/logs/github-cache/GsDevKit/Seaside31/v3.1.4.2-gs/GsDevKit-Seaside31-55f1bac/repository'...ignoring

...FAILED->BaselineOfSeaside3

Right, but aren't we trying to figure out why the load works when you remove the offending directory

Yes, my load works when we remove the offending directory. 

 

but does not work when you explicitly set the cache directory to nil:

  MCGitHubRepository cacheDirectory: nil

Yes, this does not fix the problem. What I found out is that besides that, I could do:

MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ].

That WOULD have fixed my problem. However #flushCache fails too because of the same error. So we are not even able to #flushCahe with current code.

Ah this is the piece that I was missing ... I had misunderstood that this did not work either ... and I thought you were trying to track down why this did not clear the cache ... but I understand now that when you run this code, you get a permissions related error and so you "cannot flush the caches"

Yes. So there were 2 things. First, the  ` MCGitHubRepository cacheDirectory: nil` as not enough as I also needed to #flushCache instances.

Second, was that I was not even able to flush caches because of permissions related errors.

  

... when an extent is copied to a different stone directory there are potentially a number of places that need to be fixed ... basically anywhere that is going to look for things on disk ... the Smalltalk code that gets executed by $GS_HOME/bin/newExtent currently cleans up the tODE directory dependencies and it should also reset the github-cache directory (whether or not errors are going to be thrown because of permission errors)...

The $GS_HOME/sys/{default|local}/client/tode-scripts/rebuildSys script is intended to clean up these directory/stone dependencies for tODE and GLASS and you are expected to override with application-specific clean up code ....

Soooo is the above error happening before or after you have cleared MCGitHubRepository class>>cacheDirectory?

After. It has no effect for me running such code.

 

I am under the impression that:

  MCGitHubRepository cacheDirectory: nil

isn't working for you ... am I missing something?

You are not missing anything. That is not working for me and I don't understand why it would. I am under the impression that I would also need:

MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ].

No?

 

But my code still fail to load. There are a couple of places that should be changed. Search for the string "MCFileTreeFileUtils current directoryExists:"

And you will find 

--------------------

  MCGitHubRepository class>>resetCacheDirectoryIfInvalid

 MCGitBasedNetworkRepository>>directory

 MCGitBasedNetworkRepository class>>resetCacheDirectoryIfInvalid

MCFileTreeRepository>>flushCache

I think those guys should handle the error and on error, the flush or reset anyway. To avoid this (adding error handling in all those places), I suggested adding:

MCFileTreeFileDirectoryUtils class >> directoryExistsAndValid: aDirectory

  ^ aDirectory exists on: Error do: [  false ] 

And so, the above 4 methods should actually send #directoryExistsAndValid:  rather than #directoryExists

I'm not sure that I want to try to bullet-proof this chunk of code ... I am worried about what happens if you don't have write permission in the first place or the disk is full or ???? there are too many possible error conditions that will either continue to fail or may mask the original problem if we just handle the errors ...

Besides it seems to me that the problem here is that MCGitBasedNetworkRepository class>>resetCacheDirectoryIfInvalid (which sets the cacheDirectory nil) won't work because if I understand things correctly `MCGitHubRepository cacheDirectory: nil` doesn't work ... or am I missing something here ....

Problem is that even if I do  `MCGitHubRepository cacheDirectory: nil`  the MCGitHubRepository instances still have the offending directory as part of the instvar 'directory' and so the MC load fails. That's why I think I need this code:

MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ].

Which effectively clears the 'directory' from the MCGitHubRepository instances. But..guess what? I cannot flush, because:

MCGitBasedNetworkRepository >> flushCache

  "the directory acts like a cache since we download the directory from a git-based repository (github, bitbucket, etc.)"

  super flushCache.

  self class flushDownloadCache.

  directory := nil

that method calls super and so this is executed:

MCFileTreeRepository >> flushCache

  "force properties to be reread ... if the directory exists, otherwise let nature

   take it's course"

  super flushCache.

  directory

    ifNotNil: [ 

      (MCFileTreeFileUtils current directoryExists: directory)   ->>> this will throw error now

        ifTrue: [ 

          repositoryProperties := nil.

          self repositoryProperties ] ]

So...to conclude, THIS piece of code does seem to workaround my problem:

MCGitHubRepository cacheDirectory: nil.

System commit. 

MCGitBasedNetworkRepository allSubInstances do: [ :rep | 

      " cannot send #directory: becaue it really expects a directory instance. 

      By puttin a nil in 'directory' we are able to run #flushCache. See MCFileTreeRepository >> flushCache.  for more details  "

      rep instVarNamed: 'directory' put: nil.

      rep flushCache ].

System commit. 

I didn't realize the bit about "would work except for error" ...

No I have a good picture and I do think that MCFileTreeRepository >> flushCache can and should be fixed[1] ... we should be able to unconditionally clear the cache in the fact of an error. Here's my proposed fix to MCGitBasedNetworkRepository >> flushCache from the bug report[1]:

flushCache
  "the directory acts like a cache since we download the directory from a git-based repository (github, bitbucket, etc.)"

  [super flushCache] on: Error do: [:ignored | "perhaps dump something to Transcript?" ].
  self class flushDownloadCache.
  directory := nil

Yes, I would dump the error into the transcript.

Okay here's a modified patch:

flushCache
  "the directory acts like a cache since we download the directory from a git-based repository (github, bitbucket, etc.)"

  [ super flushCache ]
    on: Error
    do: [ :ex |
      Transcript
        cr;
        show:
            'Error for: ' , self description printString , ' during flushCache: '
                , ex description printString ].
  self class flushDownloadCache.
  directory := nil


Could you test this one as well? ... There is still some exposure to permission errors for the superclass (MCFileTreeRepository), but in this case we're not dealing with a cache but the repo itself ... which is more serious and not really repairable by an automatic method ... there is a `project rehome` command that let's you point a different fileTree repo and that should be the solution for this particular case ...

I'll start the process of pushing this fix to Metacello ...

Dale

--

Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass