yesterday I did my daily batch conversion to gemstone and I came up with a slightly faster process.
If you use gofer I fear that all the monticello versions are completely parsed before being saved / uploaded. I tried to avoid this using wget to download all the versions and then script the upload ========================================================================== # bash, takes around 1min vs 10mins with gofer wget --mirror --page-requisites http://www.squeaksource.com/Citezen/ # pharo f := FSFilesystem disk workingDirectory / 'www.squeaksource.com' / 'Citezen'. files := f glob: [ :f| f basename endsWith: '.mcz']. files do: [ :file ||entity stream| stream := file readStream. (entity := ZnStreamingEntity type: ZnMimeType applicationOctetStream) stream: stream; contentLength: stream size. Transcript show: file; cr. "upload the version to gemstone" ZnClient new systemPolicy; beOneShot; signalProgress: true; ifFail: [ :exception | Transcript show: file basename; show: ' '; print: exception ]; username: 'XXX' password: 'XXX'; entity: entity; url: ( 'http://ss3.gemstone.com/ss/citezen/', file basename); put ] displayingProgress: 'Uploading mzcs' ========================================================================== best cami |
Hey Camillo,
On 15 Feb 2012, at 13:02, Camillo Bruni wrote: > yesterday I did my daily batch conversion to gemstone and I came up with a slightly faster process. > If you use gofer I fear that all the monticello versions are completely parsed before being saved / uploaded. > > I tried to avoid this using wget to download all the versions and then script the upload > > ========================================================================== > # bash, takes around 1min vs 10mins with gofer > wget --mirror --page-requisites http://www.squeaksource.com/Citezen/ > > # pharo > f := FSFilesystem disk workingDirectory / 'www.squeaksource.com' / 'Citezen'. > files := f glob: [ :f| f basename endsWith: '.mcz']. > > files do: [ :file ||entity stream| > stream := file readStream. > (entity := ZnStreamingEntity type: ZnMimeType applicationOctetStream) > stream: stream; > contentLength: stream size. > Transcript show: file; cr. > "upload the version to gemstone" > ZnClient new > systemPolicy; > beOneShot; > signalProgress: true; > ifFail: [ :exception | Transcript show: file basename; show: ' '; print: exception ]; > username: 'XXX' password: 'XXX'; > entity: entity; > url: ( 'http://ss3.gemstone.com/ss/citezen/', file basename); > put ] > displayingProgress: 'Uploading mzcs' > ========================================================================== > > best > cami Cool stuff, beautiful script ! I don't know enough about the internals of Gofer and/or Monticello, but doing what the wget does is easy in Smalltalk. repositoryUrl := 'http://www.squeaksource.com/ZincHTTPComponents/'. files := (MCHttpRepository new parseFileNamesFromStream: (ZnClient new get: repositoryUrl) entity readStream) select: [ :each | ZnMonticelloServerDelegate new isValidMczName: each ]. files do: [ :each | | mcz | Transcript cr; show: each. mcz := (ZnClient new get: 'http://www.squeaksource.com/ZincHTTPComponents/', each) entity. "…" ]. You can use the mcz entity directly as an argument to the next put. What do you think ? Let me know how fast is goes. Sven |
On 2012-02-15, at 13:36, Sven Van Caekenberghe wrote:
> Hey Camillo, > > On 15 Feb 2012, at 13:02, Camillo Bruni wrote: > >> yesterday I did my daily batch conversion to gemstone and I came up with a slightly faster process. >> If you use gofer I fear that all the monticello versions are completely parsed before being saved / uploaded. >> >> I tried to avoid this using wget to download all the versions and then script the upload >> >> ========================================================================== >> # bash, takes around 1min vs 10mins with gofer >> wget --mirror --page-requisites http://www.squeaksource.com/Citezen/ >> >> # pharo >> f := FSFilesystem disk workingDirectory / 'www.squeaksource.com' / 'Citezen'. >> files := f glob: [ :f| f basename endsWith: '.mcz']. >> >> files do: [ :file ||entity stream| >> stream := file readStream. >> (entity := ZnStreamingEntity type: ZnMimeType applicationOctetStream) >> stream: stream; >> contentLength: stream size. >> Transcript show: file; cr. >> "upload the version to gemstone" >> ZnClient new >> systemPolicy; >> beOneShot; >> signalProgress: true; >> ifFail: [ :exception | Transcript show: file basename; show: ' '; print: exception ]; >> username: 'XXX' password: 'XXX'; >> entity: entity; >> url: ( 'http://ss3.gemstone.com/ss/citezen/', file basename); >> put ] >> displayingProgress: 'Uploading mzcs' >> ========================================================================== >> >> best >> cami > > Cool stuff, beautiful script ! > > I don't know enough about the internals of Gofer and/or Monticello, but doing what the wget does is easy in Smalltalk. > > repositoryUrl := 'http://www.squeaksource.com/ZincHTTPComponents/'. > files := (MCHttpRepository new > parseFileNamesFromStream: (ZnClient new get: repositoryUrl) entity readStream) > select: [ :each | ZnMonticelloServerDelegate new isValidMczName: each ]. > files do: [ :each | | mcz | > Transcript cr; show: each. > mcz := (ZnClient new get: 'http://www.squeaksource.com/ZincHTTPComponents/', each) entity. > "…" ]. > > You can use the mcz entity directly as an argument to the next put. > > What do you think ? ah well if that works even better :D I am lazy :D and I don't have to think when using wget :P > Let me know how fast is goes. > > Sven > > > > > > |
In reply to this post by Sven Van Caekenberghe
Small typo, make that:
| repositoryUrl files | repositoryUrl := 'http://www.squeaksource.com/Citezen/'. files := (MCHttpRepository new parseFileNamesFromStream: (ZnClient new get: repositoryUrl; entity) readStream) select: [ :each | ZnMonticelloServerDelegate new isValidMczName: each ]. files. ZnClient new url: repositoryUrl; addPath: files first; get; entity. (Not that you wouldn't have been able to fix it, but I like good out of the box experience) On 15 Feb 2012, at 13:36, Sven Van Caekenberghe wrote: > Hey Camillo, > > On 15 Feb 2012, at 13:02, Camillo Bruni wrote: > >> yesterday I did my daily batch conversion to gemstone and I came up with a slightly faster process. >> If you use gofer I fear that all the monticello versions are completely parsed before being saved / uploaded. >> >> I tried to avoid this using wget to download all the versions and then script the upload >> >> ========================================================================== >> # bash, takes around 1min vs 10mins with gofer >> wget --mirror --page-requisites http://www.squeaksource.com/Citezen/ >> >> # pharo >> f := FSFilesystem disk workingDirectory / 'www.squeaksource.com' / 'Citezen'. >> files := f glob: [ :f| f basename endsWith: '.mcz']. >> >> files do: [ :file ||entity stream| >> stream := file readStream. >> (entity := ZnStreamingEntity type: ZnMimeType applicationOctetStream) >> stream: stream; >> contentLength: stream size. >> Transcript show: file; cr. >> "upload the version to gemstone" >> ZnClient new >> systemPolicy; >> beOneShot; >> signalProgress: true; >> ifFail: [ :exception | Transcript show: file basename; show: ' '; print: exception ]; >> username: 'XXX' password: 'XXX'; >> entity: entity; >> url: ( 'http://ss3.gemstone.com/ss/citezen/', file basename); >> put ] >> displayingProgress: 'Uploading mzcs' >> ========================================================================== >> >> best >> cami > > Cool stuff, beautiful script ! > > I don't know enough about the internals of Gofer and/or Monticello, but doing what the wget does is easy in Smalltalk. > > repositoryUrl := 'http://www.squeaksource.com/ZincHTTPComponents/'. > files := (MCHttpRepository new > parseFileNamesFromStream: (ZnClient new get: repositoryUrl) entity readStream) > select: [ :each | ZnMonticelloServerDelegate new isValidMczName: each ]. > files do: [ :each | | mcz | > Transcript cr; show: each. > mcz := (ZnClient new get: 'http://www.squeaksource.com/ZincHTTPComponents/', each) entity. > "…" ]. > > You can use the mcz entity directly as an argument to the next put. > > What do you think ? > > Let me know how fast is goes. > > Sven > > > > > > |
Free forum by Nabble | Edit this page |