Store publishing performance question

Hi,

Ever since we've been using vw7.7, we've always had the feeling that publishing bundles got significantly slower compared to vw7.4.1.

Recently I published changes on about 100 packages and put a TimeProfiler around it to see where the time went.

It took 443 seconds to publish and about 60% of the time is spent in #updateStorePundleItems

PublishSpecificationList >>updateStorePundleItems

                | currentPundle currentPundleItems |

                currentPundle := self pundle storeModel asStorePundleIn: publisher targetSession.

                currentPundleItems := currentPundle withAllContainedItems.

                self do:

                               [:eachSpec |

                               | eachPundle |

                               eachSpec publish ifTrue:

                                               [eachPundle := currentPundleItems

                                                               detect: [:each | each storeModel = eachSpec dbPundle storeModel]

                                                               ifNone: [nil].

                                               eachPundle ifNotNil:

                                                               [eachSpec binarySave ifTrue: [self prepareBinaryIn: eachPundle].

                                                               self updatePundleIn: eachSpec to: eachPundle]]].

                self do: [:each | self applySpec: each]

I think that what this method tries to achieve is to refresh the list of pundles you're trying to publish, just in case anything changed in the image after the "publish window" opened.

I.e. I could open the publish window, change some code in the image, and then press publish and the publish window wouldn't be aware of my extra changes unless it does this refresh.

The comment in this method seems to suggest that:

PublishSpecificationList >>updatePundleIn: aPublishSpecification to: newPundle

                "Update our pundles just in case anything changed in the image since they were created."

Am I correct in assuming this? If so, would it be safe to replace the implementation of the method with this:

PublishSpecificationList >>updateStorePundleItems

                self do: [:each | self applySpec: each]

It would make publishing a whole lot faster for us.

Thanks,

Mark

Hi Alan,

In the mean time I tried my suggested "fix", but it didn't work. Some things start to crash right away. I did notice that most of the time is spent in #asFilename. Something to do with XMLParser>>wrapDataSource:

The comment in the method already mentions that there is a problem with file URLs:

                "The creation of FileURLs takes time:  47 seconds for the whole base as against 24 seconds if we do not create them.  We therefore introduce a performance optimization:  supply the InputSource with a string, dropping the asFilename asURI call here in favour of creating lazily as needed.  It appears that, at least for source-gathering purposes, the uri of the InputSource created here is never used, except for nil checks:  a similar optimisation has been in a StoreGlorp override since 2 February 2006 but only met the non-nil check case and never expanded to a URI, which suggests the URI may not be needed."

Anyway, thanks for your suggestion. I'll have to try it out in 7.8.

Thanks,

Mark

I don't know enough about the details to comment on whether that particular fix is safe or not, but I believe that's something that had a lot of unnecessary overhead that's been much improved in 7.8.

On 2011-03-10 8:45 AM, Mark Plas wrote:

Hi,

Ever since we've been using vw7.7, we've always had the feeling that publishing bundles got significantly slower compared to vw7.4.1.

Recently I published changes on about 100 packages and put a TimeProfiler around it to see where the time went.

It took 443 seconds to publish and about 60% of the time is spent in #updateStorePundleItems

PublishSpecificationList >>updateStorePundleItems

                | currentPundle currentPundleItems |

                currentPundle := self pundle storeModel asStorePundleIn: publisher targetSession.

                currentPundleItems := currentPundle withAllContainedItems.

                self do:

                               [:eachSpec |

                               | eachPundle |

                               eachSpec publish ifTrue:

                                               [eachPundle := currentPundleItems

                                                               detect: [:each | each storeModel = eachSpec dbPundle storeModel]

                                                               ifNone: [nil].

                                               eachPundle ifNotNil:

                                                               [eachSpec binarySave ifTrue: [self prepareBinaryIn: eachPundle].

                                                               self updatePundleIn: eachSpec to: eachPundle]]].

                self do: [:each | self applySpec: each]

I think that what this method tries to achieve is to refresh the list of pundles you're trying to publish, just in case anything changed in the image after the "publish window" opened.

I.e. I could open the publish window, change some code in the image, and then press publish and the publish window wouldn't be aware of my extra changes unless it does this refresh.

The comment in this method seems to suggest that:

PublishSpecificationList >>updatePundleIn: aPublishSpecification to: newPundle

                "Update our pundles just in case anything changed in the image since they were created."

Am I correct in assuming this? If so, would it be safe to replace the implementation of the method with this:

PublishSpecificationList >>updateStorePundleItems

                self do: [:each | self applySpec: each]

It would make publishing a whole lot faster for us.

Thanks,

Mark

_______________________________________________

vwnc mailing list

[hidden email]

http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

-- 

Alan Knight [|], Engineering Manager, Cincom Smalltalk

[hidden email]

[hidden email]

http://www.cincomsmalltalk.com