SqueakSource API?

Is there an API for SqueakSource? I would like to retrieve all projects
from www.squeaksource.com and add all repositories to the Monticello
Browser. And I'd hate to use Web scraping. ;-)

If not, I could probably do it myself. It would probably be implemented
much like the rss feed feature. What do you think?

Before I forget, SqueakSource is really great! Thanks a lot!

Cheers,
- Bernhard

2006/2/19, Bernhard Pieber <[hidden email]>:
> Is there an API for SqueakSource?
No, there isn't yet.

> I would like to retrieve all projects
> from www.squeaksource.com and add all repositories to the Monticello
> Browser. And I'd hate to use Web scraping. ;-)
This sounds very interesting, it basically means client-side support
for SqS in Squeak. This would make SqS more like SqueakMap.

An other intersting option this opens is it allows us to run expensive
jobs like activity analysis on the second cpu.

> If not, I could probably do it myself. It would probably be implemented
> much like the rss feed feature. What do you think?
I don't know if we need that much overhead. You wan't a list of
projects and maybe their title, so why not use CSV? Much simpler, much
more efficient, no XML-parser required.

Cheers,
Philippe

"Philippe Marschall" <[hidden email]> wrote:
> An other intersting option this opens is it allows us to run expensive
> jobs like activity analysis on the second cpu.
I must admit that I did not understand this. Can you elaborate?

> > If not, I could probably do it myself. It would probably be implemented
> > much like the rss feed feature. What do you think?
> I don't know if we need that much overhead. You wan't a list of
> projects and maybe their title, so why not use CSV? Much simpler, much
> more efficient, no XML-parser required.
You are absolutely right, CSV would be much simpler for my use case.

Cheers,
Bernhard

2006/2/19, Bernhard Pieber <[hidden email]>:
> "Philippe Marschall" <[hidden email]> wrote:
> > An other intersting option this opens is it allows us to run expensive
> > jobs like activity analysis on the second cpu.
> I must admit that I did not understand this. Can you elaborate?

A long time ago, in a galaxy far far away, we had a realtime
statistics graphic for SqS (number of projects, ...). You can see this
in the SqS Paper submitted ESUG. As SqS grew to it's current size,
this caused all kinds of problems one of them was that it is too cpu
intensive. The code is still there, just commented out.
SqS now runs on a dual CPU machine but we can use only one CPU,
because we don't need no stinkin' Java threads. If we can access the
data from outside the image, we could set up a cron job, that starts a
script (or image) that fetches the data, computes the graphic, writes
it to the filesystem and then terminates. We can then serve the
graphic with apache.

Cheers
Philippe

On Sun, Feb 19, 2006 at 12:24:20PM +0100, Philippe Marschall wrote:
cron job?!? we don't need no stinkin' cron jobs neither ;)

 [[ | pipe backgroundJob ws |
   pipe := OSPipe new.
   backgroundJob :=
     OSProcess thisOSProcess forkHeadlessSqueakAndDoThenQuit:
       [OSProcess accessor nice: 1. "lower priority of background OS process"
       (99 to: 1 by: -1) do: [:e |
         pipe nextPutAll: e asString;
           nextPutAll:' bottle';
           nextPutAll: ((e = 1) ifTrue: [''] ifFalse: ['s']);
           nextPutAll: ' of beer on the wall, ';
           nextPutAll: e asString;
           nextPutAll:' bottle';
           nextPutAll: ((e = 1) ifTrue: [''] ifFalse: ['s']);
           nextPutAll: ' of beer'; cr;
           nextPutAll: 'take one down and pass it around, ';
           nextPutAll: ((e = 1) ifTrue: ['no more'] ifFalse: [e - 1]) asString;
           nextPutAll:' bottle';
           nextPutAll: ((e = 2) ifTrue: [''] ifFalse: ['s']);
           nextPutAll: ' of beer on the wall'; cr].
         pipe close].
   Transcript show: backgroundJob asString, ' started'; cr.
   pipe closeWriter. "don't need writer end, close it before the #upToEnd"
   "pipe writer blocks when pipe full, so we need to loop while reading to end"
   ws := WriteStream on: ''.
   [backgroundJob isComplete] whileFalse:
     [(Delay forMilliseconds: 200) wait.
     pipe upToEnd ifNotNilDo: [:s | ws nextPutAll: s]].
   pipe close.
   Transcript show: backgroundJob asString,
     ' completed, display results in 2 seconds'; cr.
   (Delay forSeconds: 2) wait.
   Transcript show: ws contents.
   Transcript cr; show: 'delay 5 seconds before forking next Squeak job'; cr.
   (Delay forSeconds: 5) wait]
     repeat] forkAt: Processor userBackgroundPriority.


It's usually more efficient to do it this way, because you don't need to
start up a new image every time you run the job. It also avoids the files
and cron jobs, which IMHO are even uglier than the aforementioned Java
threads.

Dave

 > cron job?!? we don't need no stinkin' cron jobs neither ;)

Yeah, but we have made a lot of bad experiences with long running
delays especially in exactly such cases. Cron jobs work. And IIRC SqS
runs on a Mac VM, so no forking either.

Philippe

On 19-Feb-06, at 9:16 AM, Philippe Marschall wrote:

>> cron job?!? we don't need no stinkin' cron jobs neither ;)
>
> Yeah, but we have made a lot of bad experiences with long running
> delays especially in exactly such cases.
There have been recent-ish improvements in long delays so if you have  
any way to test for the problem it would be very helpful.  See mantis  
1840 - http://bugs.impara.de/view.php?id=1840

> Cron jobs work. And IIRC SqS
> runs on a Mac VM, so no forking either.
Really? I'm pretty sure John will say OSX would do forking. Has dave  
lewis done an osprocess plugin for osx yet?


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Oxymorons: Exact estimate

Hi tim

I remember harvesting some changes of David.
Should these ones get harvested too?
Or are you "simply" asking for feedback :)...

Stef

On 19 févr. 06, at 19:15, tim Rowledge wrote:

I read the mantis entry....
so shout for feedback :)

Stef

On 19 févr. 06, at 19:33, stéphane ducasse wrote:

On 19-Feb-06, at 10:33 AM, stéphane ducasse wrote:

> Hi tim
>
> I remember harvesting some changes of David.
> Should these ones get harvested too?
> Or are you "simply" asking for feedback :)...
The only change I can directly comment on is the 1840 change to  
Delays. It's in your mantis pile right now so that you can consider  
what to do with it; my advice would be that we should try to get some  
opinion from a few more people with an interest in log delays. Dave  
Lewis, Lex and laza all took part in the mantis 854 discussion so  
they would be logical interested parties. Philippe seems a good idea  
too since he claims experience in problems with long delays.

Messing with delay is something to be done slowly, carefully and  
after much standing around pontificating with a bunch of grey bearded  
oldsters. My suggested change in 1840 is merely an opening suggestion.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
I'm so skeptical that I'm not sure I'm really a skeptic

Well the OSProcess plugin wouldn't quite work with the carbon vm  
because of:

a) x/11 (that was solvable by just commenting out a few things)
b) unix path names. (that was harder).
c) bundle versus unix library.


I'm working on (c) at the moment,  (b) is solved by using a 3.8.10 VM

At this time I need some folks to help in recompiling and testing the  
OSProcess plugin to support the mac carbon vm.

I'll note the carbon VM actually runs a number of OS threads at the  
moment.

1) For updating the low resolution clock every 1/60 of a second
2) For managing the mouse/keyboard/event input via carbon event manager.
3) The interpreter loop
4) OS thread for input/output  (created by carbon logic)
5) OS thread for sound  (created by carbon logic)
6) Lots of other threads if you use quicktime via FFI.


However since the carbon VM doesn't run headless forking the VM I'll  
guess is a problem... Mind starting up multiple headless unix squeak  
VM managed by a headful squeak VM
could be doable.

On 19-Feb-06, at 10:15 AM, tim Rowledge wrote:

--
========================================================================
===
John M. McIntosh <[hidden email]> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

On Sun, Feb 19, 2006 at 12:23:26PM -0800, John M McIntosh wrote:
> Well the OSProcess plugin wouldn't quite work with the carbon vm  
> because of:
>
> a) x/11 (that was solvable by just commenting out a few things)
> b) unix path names. (that was harder).
> c) bundle versus unix library.
>
>
> I'm working on (c) at the moment,  (b) is solved by using a 3.8.10 VM

Hi John,
Problem a) should be fixed in recent OSProcess also.

Martin Snelgrove sent me some OSProcess test results a while back,
running on the following:
wms running OSProcess unit tests 19 October 2005 11:18:36 pm
OSProcess platformName => unix
OSProcess platformSubtype => powerpc
OSProcess osVersion => darwin7.8.0
OSProcess vmVersion => Squeak3.7 of '4 September 2004' [latest update: #5989]

The #forkSqueak methods were all working in the unit tests, so
I think that forking a headless Squeak will work on OS X:

#testForkHeadlessSqueakAndDo->1 run, 1 passes, 0 expected failures, 0 failures, 0 errors, 0 unexpected passes
#testForkHeadlessSqueakAndDoThenQuit->1 run, 1 passes, 0 expected failures, 0 failures, 0 errors, 0 unexpected passes
#testForkSqueak->1 run, 1 passes, 0 expected failures, 0 failures, 0 errors, 0 unexpected passes
#testForkSqueakAndDo->1 run, 1 passes, 0 expected failures, 0 failures, 0 errors, 0 unexpected passes
#testForkSqueakAndDoThenQuit->1 run, 1 passes, 0 expected failures, 0 failures, 0 errors, 0 unexpected passes

Quite honestly I don't know the difference between a darwin and
a carbon (I thought carbon was for the brushes on my harley's
generator) but I was pleasantly surprised that #forkSqueak
was working on at least some Mac flavors, so I think this is a
doable thing.
 
> At this time I need some folks to help in recompiling and testing the  
> OSProcess plugin to support the mac carbon vm.

If you find a volunteer for this, please CC me and I'll provide
whatever remote support I can. I don't have a Mac to work with,
but OSProcess class>>allTestResults produces some useful output
that may allow me to help.
 
> I'll note the carbon VM actually runs a number of OS threads at the  
> moment.
>
> 1) For updating the low resolution clock every 1/60 of a second
> 2) For managing the mouse/keyboard/event input via carbon event manager.
> 3) The interpreter loop
> 4) OS thread for input/output  (created by carbon logic)
> 5) OS thread for sound  (created by carbon logic)
> 6) Lots of other threads if you use quicktime via FFI.

Here I suspect I have some work to do. OSPP does not try to account
for pthreads, which means that my signal handling is probably not
going to work right. There may be other things that need to be
pthread-aware also.

I don't know if pthreads are supported on all unix platforms (I
guess I am a little suspicious of anything called "posix standard"),
but I suppose it's pretty generic stuff nowadays. Which means I
probably aught to RTMF and add the required support in OSPP.

> However since the carbon VM doesn't run headless forking the VM I'll  
> guess is a problem... Mind starting up multiple headless unix squeak  
> VM managed by a headful squeak VM
> could be doable.

Definitely doable, as it seems to already work on some OS X flavors.

Dave

My pet use would be to have a sort of mini-squeak source running in  
my production seaside image.  IOW, my production image would just  
look like a SS repository but saves would actually update the running  
image.  Then I could push fixes directly from my development laptop  
to my deployed application on my production server.

So I'd also like a SS api because I think we could use it to build  
automated deployment systems similar to ruby's signal tower.

On Feb 19, 2006, at 1:59 AM, Philippe Marschall wrote:

> 2006/2/19, Bernhard Pieber <[hidden email]>:
>> Is there an API for SqueakSource?
> No, there isn't yet.

Hi Todd,

Todd Blanchard <[hidden email]> wrote:
> My pet use would be to have a sort of mini-squeak source running in  
> my production seaside image.  IOW, my production image would just  
> look like a SS repository but saves would actually update the running  
> image.  Then I could push fixes directly from my development laptop  
> to my deployed application on my production server.
This sounds like a useful use case. However, I don't understand why you
need full SqueakSource, i.e. the Web UI, for that. Wouldn't just
Monticello and a file upload be enough?

Cheers,
Bernhard

Am 20.02.2006 um 23:50 schrieb Bernhard Pieber:

Sure. But it would be *way cool* if you just open a regular web  
browser to update your image, including a nice UI :)

- Bert -