Persistence Options

Hi, I'm curious what our options are for persistence of Objects in
Amber. Since they are going to be used from node.js, JS and Amber. Is
there a way to do this (Persist Objects) on a massive multi-user
system ?

Hi,
My personal amber-fork on github contains an amber-server for package-persistence to be run on node.js - persistence is made via a minimalistic amber-mongodb-binding. Are you looking for something like that?
Regards,
Stefan

--
sent from my android-phone

Hi Stefan, I am not familiar with mongoDB, reading about it right now.
It might be just the ticket for development work, but I'm not sure how
many Web Hosts support it. I would be happy with a high performance
interface to a traditional RDBMS, but a fast, pure OODBMS would be
much better if it could be hosted easily on a third party server. I
have heard that Gemstone has a lot of overhead so I really haven't
looked closely at it yet. Thanks for the reply. Is your mongoDB code
available under some public license ? If so, where can I get it ?

Leon

On Jan 15, 1:43 pm, Stefan Krecher <[hidden email]>
wrote:

Seaside, often offered as "killerapp", as argument, why customers should change to Smalltalk/Gemstone/GLASS, Pharo/...DB - combination.

My tests have proofed, hat within any combination of Pharo/Seaside plus database - long queries via browser block short ones, what means, if a customer logs in into seaside, doing complex queries, he - in the meantime - blocks all other web queries. This is an absolute no-go!!!!

So, without ansync I/O, async TCP/IP, async OpenDBX (like async ODBC drivers) async FFI, async/multithreaded Seaside, MP COG VM (without big kernel/semaphore/mutex locking) i see any chances for Pharo to fullfill my minimum requirements on hispeed web interface.

Look even at Pharo itself. Completely blocked during updates, GC, socket I/O, p.e.

This is, hat has been solved in any software package i know from, since last 10 years, only pharoers seem to be happy with it, to see somethink working, however it works under heavy load.

Thats the level of the Smalltalk community in fact. Unusable, instable under higher loads.

This is fact, not reality. "Reality" here rather is a common illusion of Pharo Smalltalk community whose drivers are still unsable for heavy loads and will be in near future soon.

Benchmarks anybody of the "chief software architects" wanted?

regards, tnx 4 understanding, Guido Stepken

Hi!

Ehm, this is the Amber group so this discussion is in the wrong place - Amber is not even multithreaded, but of course running on node all will be asynch.

Regarding your notes AFAIK there is asynch file IO available (though I know little about it), TCP/IP has always been asynch, Dbxtalk (renamed) can work asynch if backend drivers support it I think, and I have no idea what you mean with Seaside not being multithreaded (it is). FFI is indeed blocking, although Eliot is working on that I think. What big lock in Cog are you referring to?

Furthermore updates in Pharo is just blocking because doing it in the background while working can cause inconsistencies, and socket io is not blocking. GC is incremental with only full gcs blocking and they are quite short.

Also, for scaling Seaside you would normally use a load balancer and sticky sessions.

regards, Göran

-- Sent from my Palm Pre 2, wohoo!

---

On Jan 16, 2012 19:41, Guido Stepken <[hidden email]> wrote:

Seaside, often offered as "killerapp", as argument, why customers should change to Smalltalk/Gemstone/GLASS, Pharo/...DB - combination.

My tests have proofed, hat within any combination of Pharo/Seaside plus database - long queries via browser block short ones, what means, if a customer logs in into seaside, doing complex queries, he - in the meantime - blocks all other web queries. This is an absolute no-go!!!!

So, without ansync I/O, async TCP/IP, async OpenDBX (like async ODBC drivers) async FFI, async/multithreaded Seaside, MP COG VM (without big kernel/semaphore/mutex locking) i see any chances for Pharo to fullfill my minimum requirements on hispeed web interface.

Look even at Pharo itself. Completely blocked during updates, GC, socket I/O, p.e.

This is, hat has been solved in any software package i know from, since last 10 years, only pharoers seem to be happy with it, to see somethink working, however it works under heavy load.

Thats the level of the Smalltalk community in fact. Unusable, instable under higher loads.

This is fact, not reality. "Reality" here rather is a common illusion of Pharo Smalltalk community whose drivers are still unsable for heavy loads and will be in near future soon.

Benchmarks anybody of the "chief software architects" wanted?

regards, tnx 4 understanding, Guido Stepken

Hi Göran!

Seaside suffers several design flaws.

1. When real load is put on seaside, that cannot be handeled by caching instances (e.g. Apache, NGINX), e.g. complex long running queries, seaside is blocking short queries. Under normal circumstances nobody takes notice, but under slighter load some of the clients suffer from timeouts. Design flaw in Seaside, design flaw in Pharo, design flaw in FFI, and furtheron - maybe even the ODBC driver, that is not implemented multithreaded.
Furtheron - Clients coming over slow lines, e.g. GSM, wäre not served correctly. Therefore a ICMP source quench in seaside must lead to a changed behaviour, means, that seaside should reduce clockcycles in the VM for having spare resources to serve faster clients. Instead, seaside is polling, doing loops at full power, busy waiting for the slow client to accept input AND blocking the database interface in the  meantime. This is *bad design*

This is a complete messy software design under aspects of concurrency and correct load balancing of resources, as well in seaside as in Pharo as in VM.

Luckily node.js is *designed* to execute all I/O in parallel without blocking anything. This engine really feels "smooth", serving up to 100.000 clients smoothly, even letting drip TCP/IP packets to GSM clients during serving fast clients upto bandwidth limit. (E.g. on new FreeBSD)

So i expect Amber or Ale's S8 engine to become "The new Smalltalk" in very near future, because its quite easy to create language bindings to millions of JavaScript libraries.

Pharo architects are stuck even in simple problems of concurrency, that are already solved in V8 engine. This will take them another few years to get essential things right.

Have fun!

Hi!

On 01/17/2012 10:05 AM, Guido Stepken wrote:
> Hi Göran!
>
> Seaside suffers several design flaws.
>
> 1. When real load is put on seaside, that cannot be handeled by caching
> instances (e.g. Apache, NGINX), e.g. complex long running queries,
> seaside is blocking short queries. Under normal circumstances nobody
> takes notice, but under slighter load some of the clients suffer from
> timeouts.

Can you explain what you mean here? Seaside is multithreaded - it
doesn't block. If you are talking about database queries that is another
thing - and then it all depends on what stuff you use.

> Design flaw in Seaside, design flaw in Pharo, design flaw in
> FFI, and furtheron - maybe even the ODBC driver, that is not implemented
> multithreaded.

As I said, FFI is blocking, yes. Although the new one is meant not to
be. Sockets are not blocking. Seaside is not blocking. Operations in the
IDE in Squeak/Pharo are typically run in the UI process thus simplifying
a LOT, but there is nothing preventing background compilation for
example. It just gets a bit hairy.

> Furtheron - Clients coming over slow lines, e.g. GSM, wäre not served
> correctly. Therefore a ICMP source quench in seaside must lead to a
> changed behaviour, means, that seaside should reduce clockcycles in the
> VM for having spare resources to serve faster clients.

Ok, this is stuff I know nothing about.

> Instead, seaside
> is polling, doing loops at full power, busy waiting for the slow client
> to accept input AND blocking the database interface in the  meantime.

What polling is this? Somewhere in the SocketPlugin or in the
Socket/SocketStream code on the Smalltalk side?

> This is *bad design*
>
> This is a complete messy software design under aspects of concurrency
> and correct load balancing of resources, as well in seaside as in Pharo
> as in VM.

Please point at the specific place in each:

1. Seaside. What in Seaside is "wrong"?
2. Pharo. What specifically are you referring to?
3. Finally, the VM. If you mean the FFI (which is not used for Sockets),
then say so, if you mean something else in the VM,he  what?

> Luckily node.js is *designed* to execute all I/O in parallel without
> blocking anything. This engine really feels "smooth", serving up to
> 100.000 clients smoothly, even letting drip TCP/IP packets to GSM
> clients during serving fast clients upto bandwidth limit. (E.g. on new
> FreeBSD)

Yes, I agree Nodejs is great - but it is also vastly different so the
comparison is not easy to make.

> So i expect Amber or Ale's S8 engine to become "The new Smalltalk" in
> very near future, because its quite easy to create language bindings to
> millions of JavaScript libraries.

Yes, we all agree on the potential here.

> Pharo architects are stuck even in simple problems of concurrency, that
> are already solved in V8 engine. This will take them another few years
> to get essential things right.

Ehm, you are aware that node is single threaded - right? There is NO
multithreading concurrency in V8 nor node. It is a single event loop.

> Have fun!

regards, Göran

Hi Leon,

Hi Göran!

Multithreaded from the perspective of the programmer means nothing. The c-libs downunder have to be implemented in the same manner. Down to OS, to be precise, multithreaded TCP/IP stack included.

If a client sends ICMP 5 signal - "please slower", the VM has to be aware of, what worker process is assigned to that IP and has to reduce clock cycles for that process running. Try it with seaside, independent on what Smalltalk engine, and you will see, that seaside indeed is multithreaded, but is running loops without effectively being able to serve the rest of the clients with more CPU cycles or even having more power to accept more connections. Please: Profile yourself!

V8, though only single process, does correctly reduce cycles, e.g. when a client falls down from HSDPA+ to GSM speed. We live in a mobile world!!!!! :-)

regards, Guido Stepken

All of your points are well taken and have been true of other
Smalltalk implementations even years ago when I wrote an interface
between ST and ODBC named SQLJunction. All of these problems can be
handled, but it is important to note that we are NOT necessarily
talking about Smalltalk at all. With Amber we are really talking about
how Javascript applications handle the problem. To my mind that really
opens up interesting questions. Maybe I should be posing this on the
JS forums :-)

Leon

On Jan 16, 10:41 am, Guido Stepken <[hidden email]> wrote:

Hi Guido,

S, Guido Stepken piše:

Well, I can say for Aida but I'm pretty sure Seaside is here the same.
Namely, if client slows down receiving a response, the sending process
(in Aida or Seaside framework) will be blocked on the Smalltalk
semaphore, while other sending processes will continue as normally. This
works on Pharo as well, because Pharo sockets don't block a whole image.
While FFI database connections block, yes, and this is a big Pharo
shortcoming, agreed.

Also ICMP protocol is low level part of TCP/IP, you won't usually "feel"
its messages on web framework or Pharo sockets level directly, but just
indirectly. Usually you just don't need to care, ok, at least I didn't
need yet while working with mobile web apps as well.

But generally I agree that we must incorporate more asynchronicity in
our Smalltalk VMs too, as is node.js wave forcing the others.

> >
> > V8, though only single process, does correctly reduce cycles, e.g. when
> > a client falls down from HSDPA+ to GSM speed. We live in a mobile
> > world!!!!! :-)
> >
> > regards, Guido Stepken
> >
-- Janko Mivšek Aida/Web Smalltalk Web Application Server http://www.aidaweb.si

Hi!

Sorry for top post (on bus, phone). Please respond to the following:
1. Why did you post this on the amber list in the first place? Just curious.
2. The source quench ICMP message is type 4 I believe, not 5. It has not been implemented for TCP in Linux or FreBSD since around 2004 and is being deprecated now. Also, I might be mistaken, but it seems to me to be a mechanism in the TCP/IP stack and not something relevant in the layers above, like our VMs.

Now, I am not saying networking in Smalltalk implementations is perfect, far from it. I just react on claims of facts that seem wrong to me.

Also, I beg to differ regarding multithreading, it works pretty fine in Smalltalk for a lot of us. Sure, most Smalltalks do not use native threads underneath but there are pros and cons.

regards, Göran

-- Sent from my Palm Pre 2, wohoo!

---

On Jan 17, 2012 17:19, Guido Stepken <[hidden email]> wrote:

Hi Göran!

Multithreaded from the perspective of the programmer means nothing. The c-libs downunder have to be implemented in the same manner. Down to OS, to be precise, multithreaded TCP/IP stack included.

If a client sends ICMP 5 signal - "please slower", the VM has to be aware of, what worker process is assigned to that IP and has to reduce clock cycles for that process running. Try it with seaside, independent on what Smalltalk engine, and you will see, that seaside indeed is multithreaded, but is running loops without effectively being able to serve the rest of the clients with more CPU cycles or even having more power to accept more connections. Please: Profile yourself!

V8, though only single process, does correctly reduce cycles, e.g. when a client falls down from HSDPA+ to GSM speed. We live in a mobile world!!!!! :-)

regards, Guido Stepken

Hi Göran!

Sorry, it was icmp type 4, of course! Depreciated? Yes. Due to DoS attacks. Nevertheless the problem remains: How are green threads in the vm informed, that they might reduce cycles now?
Threre are two mental models competing: one as Smalltalk being a total reaction machine to network ... events, the other is "polling and timeouts". The later one is responsible for waste of cpu cycles and battery power.
This is, why Steven Jobs banned FLASH from the iPhone and Adobe dropped development. Unrefactorable.
The third mental model is tracking, what possible reoccurring events are connected to what green thread and reducing its cpu cycles to a minimum, whatever loops and timers there may be awaiting the arrival of data.
Pharo, seaside is full of such "problems", which don't only cause waste of cpu cycles, but makes seaside unresponsive at minimum loads. For battery powered androids Pharo therefore is useless.

Why i post here? To avoid same principle design mistakes in Amber.

tnx, Guido Stepken

Am 18.01.2012 09:44 schrieb "Göran Krampe" <[hidden email]>:
>
> Hi!
>
> Sorry for top post (on bus, phone). Please respond to the following:
> 1. Why did you post this on the amber list in the first place? Just curious.
> 2. The source quench ICMP message is type 4 I believe, not 5. It has not been implemented for TCP in Linux or FreBSD since around 2004 and is being deprecated now. Also, I might be mistaken, but it seems to me to be a mechanism in the TCP/IP stack and not something relevant in the layers above, like our VMs.
>
> Now, I am not saying networking in Smalltalk implementations is perfect, far from it. I just react on claims of facts that seem wrong to me.
>
> Also, I beg to differ regarding multithreading, it works pretty fine in Smalltalk for a lot of us. Sure, most Smalltalks do not use native threads underneath but there are pros and cons.
>
>
> regards, Göran
>
>
>
> -- Sent from my Palm Pre 2, wohoo!
>
> ________________________________
> On Jan 17, 2012 17:19, Guido Stepken <[hidden email]> wrote:
>
> Hi Göran!
>
> Multithreaded from the perspective of the programmer means nothing. The c-libs downunder have to be implemented in the same manner. Down to OS, to be precise, multithreaded TCP/IP stack included.
>
> If a client sends ICMP 5 signal - "please slower", the VM has to be aware of, what worker process is assigned to that IP and has to reduce clock cycles for that process running. Try it with seaside, independent on what Smalltalk engine, and you will see, that seaside indeed is multithreaded, but is running loops without effectively being able to serve the rest of the clients with more CPU cycles or even having more power to accept more connections. Please: Profile yourself!
>
> V8, though only single process, does correctly reduce cycles, e.g. when a client falls down from HSDPA+ to GSM speed. We live in a mobile world!!!!! :-)
>
> regards, Guido Stepken

Hi!

On 01/18/2012 12:57 PM, Guido Stepken wrote:
> Hi Göran!
>
> Sorry, it was icmp type 4, of course! Depreciated? Yes. Due to DoS
> attacks. Nevertheless the problem remains: How are green threads in the
> vm informed, that they might reduce cycles now?

Note also that AFAICT the source quench message has *not been handled*
in Linux or FreeBSD for TCP the last 6 years. So I am not sure why you
are bringing it up? Correct me if I am wrong.

And even if they *did* handle it - as I said - isn't that a TCP/IP-stack
issue? From my cursory reading about it - it seems to me to NOT being
something that higher levels are meant to care about.

> Threre are two mental models competing: one as Smalltalk being a total
> reaction machine to network ... events, the other is "polling and
> timeouts". The later one is responsible for waste of cpu cycles and
> battery power.
> This is, why Steven Jobs banned FLASH from the iPhone and Adobe dropped
> development. Unrefactorable.

Not sure about that connection.

> The third mental model is tracking, what possible reoccurring events are
> connected to what green thread and reducing its cpu cycles to a minimum,
> whatever loops and timers there may be awaiting the arrival of data.

Ok, so let's get back to the question. Seaside in say Pharo/Squeak. The
implementation is a forking server with one "green" Smalltalk Process
being created for each incoming HTTP request. The SocketStream (which I
wrote the current implementation of) at the bottom there relies in turn
on class Socket, which in turn relies on the SocketPlugin.

Basically the Smalltalk code uses Semaphores to block until more
available data is there to get - so there is no polling going on there.
Not AFAIK. Again, please correct me if I am wrong.

> Pharo, seaside is full of such "problems", which don't only cause waste
> of cpu cycles, but makes seaside unresponsive at minimum loads. For
> battery powered androids Pharo therefore is useless.

Ok... well, I might take the time to move this discussion to vm-dev to
see what John, Ian and the others have to say.

> Why i post here? To avoid same principle design mistakes in Amber.

Ok, fine. That was a tad unclear from the post.

regards, Göran

Hi guys,

I think the pharo/vm/seaside mailing lists are better places to discuss
these matters :)

Cheers,
Nico

On Wed, 2012-01-18 at 13:24 +0100, Göran Krampe wrote: