remote store repository strategies
Locked
remote store repository strategies
 
|
We're finding that dealing with a remote store repository (especially over VPN) can be very painful. I've tried creating a local repository from a backup of the remote one, and using replication to maintain the local one, but there seem to be too many problems with that approach. On occaision I've published to the wrong one, and the parent version gets confused. Replicating from my local to the shared repository generates an additional submenu when trying to browse changes of a package, which irritates coworkers (a smaller concern, but still a concern).
I'm wondering how others have mitigated slow access to remote repositories?
I've hallucinated about a cloning mechanism which periodically syncs a local clone repository with the remote master, but unlike replication, preserving primary keys. It would allow loading and reconciling against the local clone, but force publishing to the master repository. I think there should be a quick way of checking if there is anything new in the master repository that needs cloning, maybe by checking the state of the sequences in the master repository against the values in the clone, but without incrementing them.
[hidden email] _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
Re: remote store repository strategies
|
|
There are database-level replication mechanisms that would
do as you suggest, but they will depend on your database.
It seems like many of your issues would be avoided if, having set up the local repository, local users used it exclusively. At 12:30 PM 2010-02-23, Dave Stevenson wrote: We're finding that dealing with a remote store repository (especially over VPN) can be very painful. I've tried creating a local repository from a backup of the remote one, and using replication to maintain the local one, but there seem to be too many problems with that approach. On occaision I've published to the wrong one, and the parent version gets confused. Replicating from my local to the shared repository generates an additional submenu when trying to browse changes of a package, which irritates coworkers (a smaller concern, but still a concern). --
Alan Knight [|], Engineering Manager, Cincom Smalltalk
_______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
Re: remote store repository strategies
|
|
In reply to this post by Dave Stevenson-3
Am 23.02.2010 um 18:30 schrieb Dave Stevenson:
Is it publishing or loading that's so slow? After changing the default search order for "Any" type packages from "Parcels first" to "Store first", I got an increase of 10 times or more in load speed. The actual bottleneck was our LAN, rather than the network connection ;-) File system searches (directory scans on SMB shares, in particular) can be horribly slow. If you don't use parcels anyway, that might be worth a try. Andre P.S: Perhaps Store should cache directory information of all .pcl files found while a load context lasts? _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
Re: remote store repository strategies
|
|
Hi Andre,
I know there have been some threads about Store speed in 7.7, but we're still in 7.6.
Everything is slower since they moved our repository from a PC in the next cube to a properly configured, secured and backed up machine in some server room (I don't even know what state it's in anymore, much less what city).
We have 'Search bundes [sic] and packages first' selected on the 'Store' -> 'Prerequisites' settings page, so I think that takes care of #any prerequisite versions.
I also don't map any network drives, since they slow down VW's file browser. Rather, I have shortcuts to open the few that I need without mapping them to a drive letter. I would only map briefly in rare circumstances, such as when installing VW from a file server.
In our typical case prereqs are almost always already loaded. Only when building a new image from visual.im do the prereqs come into play, but we've published the VW distribution anyway, so everything gets loaded from store.
Typically I do 'File' -> 'Update' in the More Recently Published Items tool
(which we have customized with filters:
for excluding pundles {like BOSS, which we only want to load from parcel, not package, because of uninstalled code}
same line of development {if I'm in the trunk, don't show me stuff with only newer branch versions, and vice-versa}
minimum blessing level {don't show me newer stuff marked broken}
branch designator substrings {if I'm working in the main trunk, don't show me '*foo*' branch versions, such as '7.6 - foo 1.0'}
and a 'Load All' button to get everything up to date)
We load frequently to stay current with other developer's changes (in the morning, after publishing what we've been working on, etc.). There's also the occaisional merge, so comparing and reconciling are frequent activities. Since publishing happens less frequently, I don't care if that hits a remote server. I'm looking at cloning/replication solutions for Postgresql, as Alan suggested, to see if there's something that lets me do read-only queries on my laptop (more recent versions, load, compare, reconcile, browse versions of a method, etc.) while writing to a remote master repository (publish). So far this looks helpful:
[hidden email] From: Andre Schnoor <[hidden email]> To: Dave Stevenson <[hidden email]> Cc: [hidden email] Sent: Tue, February 23, 2010 10:50:07 AM Subject: Re: [vwnc] remote store repository strategies Am 23.02.2010 um 18:30 schrieb Dave Stevenson:
Is it publishing or loading that's so slow?
After changing the default search order for "Any" type packages from "Parcels first" to "Store first", I got an increase of 10 times or more in load speed.
The actual bottleneck was our LAN, rather than the network connection ;-) File system searches (directory scans on SMB shares, in particular) can be horribly slow.
If you don't use parcels anyway, that might be worth a try.
Andre
P.S: Perhaps Store should cache directory information of all .pcl files found while a load context lasts?
_______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
Re: remote store repository strategies
|
|
In reply to this post by Dave Stevenson-3
Dave,
We're using the Store-Cache package. This will cache the database blobs on disk and will also cache some sql queries. It gives a great speedup when you're working remote or behind some high latency or low througput link. See the public repository. Cheers, Wouter On Tue, Feb 23, 2010 at 6:30 PM, Dave Stevenson <[hidden email]> wrote: > We're finding that dealing with a remote store repository (especially over > VPN) can be very painful. I've tried creating a local repository from a > backup of the remote one, and using replication to maintain the local one, > but there seem to be too many problems with that approach. On occaision I've > published to the wrong one, and the parent version gets confused. > Replicating from my local to the shared repository generates an additional > submenu when trying to browse changes of a package, which irritates > coworkers (a smaller concern, but still a concern). > > I'm wondering how others have mitigated slow access to remote repositories? > > I've hallucinated about a cloning mechanism which periodically syncs a local > clone repository with the remote master, but unlike replication, preserving > primary keys. It would allow loading and reconciling against the local > clone, but force publishing to the master repository. I think there should > be a quick way of checking if there is anything new in the master repository > that needs cloning, maybe by checking the state of the sequences in the > master repository against the values in the clone, but without incrementing > them. > > Dave Stevenson > [hidden email] > _______________________________________________ > vwnc mailing list > [hidden email] > http://lists.cs.uiuc.edu/mailman/listinfo/vwnc > > -- Wouter Gazendam AG5 B.V. Timorplein 37 1094 CC Amsterdam www.ag5.nl tel. 020-4630942 Tel: 020-4630942 Fax: 020-4630946 _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
Re: remote store repository strategies
|
|
In reply to this post by Dave Stevenson-3
Dear Dave,
a few trivial thoughts. > We're finding that dealing with a remote store repository (especially > over VPN) can be very painful. ... I'm wondering how others have > mitigated slow access to remote repositories? (At the risk of pointing out the obvious) many in the Cincom engineering team use ssh tunnels to our Store databases instead of accessing them through our VPN. I would not call using the latter 'very painful' but I certainly use tunnels as my default and see better speeds. > I've tried creating a local repository from a backup of the remote > one, and using replication to maintain the local one, but there seem > to be too many problems with that approach. On occaision I've > published to the wrong one, and the parent version gets confused. > Replicating from my local to the shared repository generates an > additional submenu when trying to browse changes of a package, which > irritates coworkers (a smaller concern, but still a concern). A changes submenu should only be there because items are reconciled to both databases. That would seem avoidable for your coworkers IIUC. > I've hallucinated about a cloning mechanism which periodically syncs a > local clone repository with the remote master, but unlike replication, > preserving primary keys. This sounds like standard database export/import facility from master to slave databases. It could be done via Glorp, reusing much replicator code, but if your DBMS has an admin facility, that may be faster and easier for you to set up. You are not processing the data at all, just exporting everything that has changed since the last checkpoint/export/whatever. The point of the replicator is to preserve the Store semantics while keeping the databases distinct. By accepting a sole point of publish synched to read-only clones, you make your problem simpler, so can bypass the replicator. > It would allow loading and reconciling against the local clone, but force publishing to the master repository. The 'guest' login of the Cincom OR bars publishing. Provide similar such logins with read-only priviledges for ordinary use of your local DB, equivalent in image name and password to the publish-enabled logins of your remote DB. You will still have to logout and in when you publish but you will not publish to the wrong place by accident. > I think there should be a quick way of checking if there is anything > new in the master repository that needs cloning, maybe by checking the > state of the sequences in the master repository against the values in > the clone, but without incrementing them. Indeed, that sounds like the Glorp-ish approach, but perhaps the admin facilities of your DB already provide something usable. It depends on the synch cycle time. A DB admin feature may be enough. At some rapid synch speed, knowing and using the actual schema becomes necessary. (Of course, if having the local updated on a daily basis is good enough, something as trivial as copying the remote's database file to local overnight could work. In some DBMSs, you can make the image think local and remote are the _same_ database, which has advantages but it then becomes very important that your local logins are read-only, that you never replicate between these DBs, that you are aware Store Garbage Collection against the remote during working hours could create odd behaviour in images with local-loaded GCed items, etc., - nothing a good process can't handle.) > We have 'Search bundes [sic] and packages first' selected > on the 'Store' -> 'Prerequisites' settings page, so I think that > takes care of #any prerequisite versions. I would expect this to be the _slower_ setting in many cases - but of course if it works for you ... HTH Yours faithfully Niall Ross _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
«
Return to VisualWorks
|
1 view|%1 views
| Free forum by Nabble | Edit this page |