Re: Store Publish Speed VW7.7 -- very slow ...

Just to come back to this:

I just published a bundle, and saw the class names again. All 5 packages within the bundle were already published earlier; the only change was which version of the 3^rd package was included in the bundle (its old and new versions had both been published with 7.7, and had binFile=nil). The transcript listed only things from packages 1, 2 and 5; nothing from 3 or 4. Packages 3 and 4 are significantly bigger than the others, and are changed more frequently (1, 2 and 5 were last published in 2003/4; 3 and 4 were published this year, both at least once from 7.7). The things listed in the transcript were all classes and namespaces defined in packages 1, 2 and 5, and all shared variables defined in them.

All package versions published before 7.7 still have binFile = ‘’, all since have binFile = nil. As you know, bundles don’t have binFile.

HTH,

Steve

Were you seeing lists of class names go by when publishing? I think the only way that would happen is if it were publishing binary. If you weren't, then you might just be seeing slowdowns from the ODBC issue, and not be affected by the binary publishing issue. Is it still slow for you with that fix in place? Also, have you run the update77 script which generates a number of additional indexes? That should improve the speed of some operations considerably. It might slow down publishing slightly, as the database has to update the indexes, but I don't think that would account for the kind of symptoms people have described, or even be likely to be noticeable.

At 04:44 AM 2010-02-25, Steven Kelly wrote:

It doesn’t seem that loading or reconciling an old package with binFile = '' would cause binary publishes in 7.7.  I’ve gone through our repository, and nothing is being published binary: the only packages with parcel nonNil are from several years ago. Everything published before 7.7 has binFile = ''. Everything published from 7.7 has binFile = nil – including all the 7.7. base, which was reconciled with the 7.6 versions (which had binFile = ''), and the couple of dozen new versions of our own packages, which had been loaded into 7.7 from the 7.6 packages (which had binFile = '').
 
I haven’t removed the binFile: '' line from Package class>>newNamed:. We don’t have Store customization, other than automating base reconcile and loads with something like this:
 
vwVersions := #('7.7' '7.6' '7.5' '7.4.1' '7.4' '7.3.1' '7.3' '7.2.1' '7.2' '7.1' '7.0' '5i.4' '5i.3' '5i.2').
Store.Registry allPundlesSorted do: [:pundleModel | | dbVersion |
                dbVersion := (pundleModel class databaseClass allVersionsWithName: pundleModel name) detect: [:package | vwVersions includes: package version] ifNone: [].
                dbVersion notNil ifTrue:
                                [dbVersion getImagePundle reconcileWithDbVersion: dbVersion]].
 
orderedNameVersionDict do: [:pundleName :pundleVersion |
                Store.Package loadWithName: pundleName version: pundleVersion].
 
Steve
 
From: [hidden email] [[hidden email]] On Behalf Of Alan Knight
Sent: 22 February 2010 19:03
To: Dennis Smith; Samuel S. Shuster
Cc: [hidden email]
Subject: Re: [vwnc] Store Publish Speed VW7.7 -- very slow ...
 
OK, I think I have a theory on this. It seems to me that what's happening is that older versions do in fact publish with binFile as an empty string, but that StoreGlorp publishing publishes with it as nil. So we didn't observe this behaviour internally because we had switched over to publishing using StoreGlorp well before the feature to continue publishing something binary if it was previously published that way went in. So almost everything had been published using the new mechanism in the meantime. But someone who received 7.7, and loaded a package which had not been updated using a Storp version, then it would appear to our code as if it had been published binary previously, and it would attempt to do so again.

So I suspect we can both provide a patch such that it would check more carefully if something had been published binary, and an expression that would clean up the repository by removing that invalid indicator from any suspect packages. But we need to check the theory.

At 05:20 PM 2010-02-19, Dennis Smith wrote:

Thanks!

On 2/19/2010 5:15 PM, Samuel S. Shuster wrote:
> Dennis:
>
>   
>> One final (I hope) question on this -- is it true then that we now have
>> packages in the store DB which have binFile set to not-nil so that they
>> will continue to think that they need to dump binary??  If so, how do we
>> go about fixing that??
>>     
>
> I suppose I should have expected that... And the answer is yes, they may.
>
> Here is a general script using the new Glorp objects to fix this for ALL such packages in your database:
>
>       | session query |
>       session := Store.Glorp.StoreLoginFactory currentStoreSession.
>       query := Glorp.Query readManyOf: Store.Glorp.StorePackage where: [:each | each binFile notNil].
>       packages := session execute: query.
>       "Here you might select for packages not named 'AT Parser Compiler' and others you want to keep binary publishing"
>       "Something like ... packages := packages reject: [:each | 'some reason to not update these']"
>       packages do:
>                [:eachPackage
>                session inUnitOfWorkDo:
>                         [session register: eachPackage.
>                         eachPackage previous: eachPackage previous.
>                         eachPackage binFile: nil]].
>       session reset
>
> Now, you may wonder why there is that line in there:
>                         eachPackage previous: eachPackage previous.
>
> The reason is I don't remember if a Fix was in 7.7 for automatically filling in this value. It hurts nothing, but if the fix it isn't there, the parent/trace versions get lost.
>
> Now for some limited additional help... :
>
>       query := Query readManyOf: Store.Glorp.StorePackage where: [:each | each parcel notNil].
>       binarySavedPackages := session execute: query.
>
> This will give you all of the packages that HAVE been saved binary in your Database.
>
> Further if you then do something like
>
>       uniqueBinarySavedNames := binarySavedPackages
>                inject: Set new
>                into:
>                         [:sum :each |
>                         sum add: each name.
>                         sum].
>
> With that, you might be able to put together some kind of reject or whatever based on the packages or names to feed to the final update step.
>
> Just trying to be helpful.
>
>                                  And So It Goes
>                                       Sames
> ______________________________________________________________________
>
> Samuel S. Shuster [|]
> VisualWorks Engineering, Store Project
> Smalltalk Enables Success -- What Are YOU Using?
>
>
>
>
>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>   

--
Dennis Smith                                       +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              sip:[hidden email]
Canada                              http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
 
--
Alan Knight [|], Engineering Manager, Cincom Smalltalk
[hidden email]
[hidden email]
http://www.cincom.com/smalltalk

Just to come back to this:

I just published a bundle, and saw the class names again. All 5 packages within the bundle were already published earlier; the only change was which version of the 3^rd package was included in the bundle (its old and new versions had both been published with 7.7, and had binFile=nil). The transcript listed only things from packages 1, 2 and 5; nothing from 3 or 4. Packages 3 and 4 are significantly bigger than the others, and are changed more frequently (1, 2 and 5 were last published in 2003/4; 3 and 4 were published this year, both at least once from 7.7). The things listed in the transcript were all classes and namespaces defined in packages 1, 2 and 5, and all shared variables defined in them.

All package versions published before 7.7 still have binFile = ‘’, all since have binFile = nil. As you know, bundles don’t have binFile.

HTH,

Steve

Were you seeing lists of class names go by when publishing? I think the only way that would happen is if it were publishing binary. If you weren't, then you might just be seeing slowdowns from the ODBC issue, and not be affected by the binary publishing issue. Is it still slow for you with that fix in place? Also, have you run the update77 script which generates a number of additional indexes? That should improve the speed of some operations considerably. It might slow down publishing slightly, as the database has to update the indexes, but I don't think that would account for the kind of symptoms people have described, or even be likely to be noticeable.

At 04:44 AM 2010-02-25, Steven Kelly wrote:

It doesn’t seem that loading or reconciling an old package with binFile = '' would cause binary publishes in 7.7.  I’ve gone through our repository, and nothing is being published binary: the only packages with parcel nonNil are from several years ago. Everything published before 7.7 has binFile = ''. Everything published from 7.7 has binFile = nil – including all the 7.7. base, which was reconciled with the 7.6 versions (which had binFile = ''), and the couple of dozen new versions of our own packages, which had been loaded into 7.7 from the 7.6 packages (which had binFile = '').
 
I haven’t removed the binFile: '' line from Package class>>newNamed:. We don’t have Store customization, other than automating base reconcile and loads with something like this:
 
vwVersions := #('7.7' '7.6' '7.5' '7.4.1' '7.4' '7.3.1' '7.3' '7.2.1' '7.2' '7.1' '7.0' '5i.4' '5i.3' '5i.2').
Store.Registry allPundlesSorted do: [:pundleModel | | dbVersion |
                dbVersion := (pundleModel class databaseClass allVersionsWithName: pundleModel name) detect: [:package | vwVersions includes: package version] ifNone: [].
                dbVersion notNil ifTrue:
                                [dbVersion getImagePundle reconcileWithDbVersion: dbVersion]].
 
orderedNameVersionDict do: [:pundleName :pundleVersion |
                Store.Package loadWithName: pundleName version: pundleVersion].
 
Steve
 
From: [hidden email] [[hidden email]] On Behalf Of Alan Knight
Sent: 22 February 2010 19:03
To: Dennis Smith; Samuel S. Shuster
Cc: [hidden email]
Subject: Re: [vwnc] Store Publish Speed VW7.7 -- very slow ...
 
OK, I think I have a theory on this. It seems to me that what's happening is that older versions do in fact publish with binFile as an empty string, but that StoreGlorp publishing publishes with it as nil. So we didn't observe this behaviour internally because we had switched over to publishing using StoreGlorp well before the feature to continue publishing something binary if it was previously published that way went in. So almost everything had been published using the new mechanism in the meantime. But someone who received 7.7, and loaded a package which had not been updated using a Storp version, then it would appear to our code as if it had been published binary previously, and it would attempt to do so again.

So I suspect we can both provide a patch such that it would check more carefully if something had been published binary, and an expression that would clean up the repository by removing that invalid indicator from any suspect packages. But we need to check the theory.

At 05:20 PM 2010-02-19, Dennis Smith wrote:

Thanks!

On 2/19/2010 5:15 PM, Samuel S. Shuster wrote:
> Dennis:
>
>   
>> One final (I hope) question on this -- is it true then that we now have
>> packages in the store DB which have binFile set to not-nil so that they
>> will continue to think that they need to dump binary??  If so, how do we
>> go about fixing that??
>>     
>
> I suppose I should have expected that... And the answer is yes, they may.
>
> Here is a general script using the new Glorp objects to fix this for ALL such packages in your database:
>
>       | session query |
>       session := Store.Glorp.StoreLoginFactory currentStoreSession.
>       query := Glorp.Query readManyOf: Store.Glorp.StorePackage where: [:each | each binFile notNil].
>       packages := session execute: query.
>       "Here you might select for packages not named 'AT Parser Compiler' and others you want to keep binary publishing"
>       "Something like ... packages := packages reject: [:each | 'some reason to not update these']"
>       packages do:
>                [:eachPackage
>                session inUnitOfWorkDo:
>                         [session register: eachPackage.
>                         eachPackage previous: eachPackage previous.
>                         eachPackage binFile: nil]].
>       session reset
>
> Now, you may wonder why there is that line in there:
>                         eachPackage previous: eachPackage previous.
>
> The reason is I don't remember if a Fix was in 7.7 for automatically filling in this value. It hurts nothing, but if the fix it isn't there, the parent/trace versions get lost.
>
> Now for some limited additional help... :
>
>       query := Query readManyOf: Store.Glorp.StorePackage where: [:each | each parcel notNil].
>       binarySavedPackages := session execute: query.
>
> This will give you all of the packages that HAVE been saved binary in your Database.
>
> Further if you then do something like
>
>       uniqueBinarySavedNames := binarySavedPackages
>                inject: Set new
>                into:
>                         [:sum :each |
>                         sum add: each name.
>                         sum].
>
> With that, you might be able to put together some kind of reject or whatever based on the packages or names to feed to the final update step.
>
> Just trying to be helpful.
>
>                                  And So It Goes
>                                       Sames
> ______________________________________________________________________
>
> Samuel S. Shuster [|]
> VisualWorks Engineering, Store Project
> Smalltalk Enables Success -- What Are YOU Using?
>
>
>
>
>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>   

--
Dennis Smith                                       +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              sip:[hidden email]
Canada                              http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
 
--
Alan Knight [|], Engineering Manager, Cincom Smalltalk
[hidden email]
[hidden email]
http://www.cincom.com/smalltalk

Thanks Boris; do you mean the test in Store.Glorp.StorePundleWriter>>prepareBinarySaveFor:?

Looks like things are being published binary after all: “session read: StoreParcelRecord” returns lots of recent versions that really have StoreBinaryBlobs. Darn. However, it looks like those StoreParcelRecords are orphans: the packages in question have parcel=nil. So publishing is doing some unnecessary work, but loading those packages won’t load them binary. I think.

Steve