Enterprise Software V Ecommerce [in smalltalk/seaside/cincom]

Hi all,

I’m a relative newbie (4 years) to programming and seaside, who
partnered up with some “old hands” (smalltalk/seaside is my first and
only language).

Initially we developed an Ecommerce (Istazaar) fashion platform for
designers to maintain their collections.  We stagnated last year due
to the time restraints on the “old hands” and I then began developing
a web based Enterprise Software (Vivport) platform for
document-management/knowledge-management/emails/tasks.

We now have two ventures at relatively equal stages of development.
I believe the Enterprise Software (Vivport) platform is the more
innovative however, now that Steve (one of the “old hands”) has some
time again he has been able to review its development (which
unfortunately I built unsupervised) and has major concerns in relation
to deploying this venture.

Instead of building the database; then building middleware, then UI;
I’ve basically just built a UI, Steve’s concerns with the outcomes of
this development process are outlined in his email to me below.

We were wondering if the smalltalk/seaside community has any views on
Steve’s concerns below that may influence our decision on which
venture to proceed with.
The platform has been developed to freely change between the user’s
personal account and their company(ies) account.  The personal
accounts are free so we would be attempting to server many users.

Any advice in relation to users/image, persistence, database, general
deployment etc for this type of venture (essentially a beefed up
gmail/cloud environment) would be greatly appreciated.

Thanks for your time.

Cheers,

Dirk


---------- Forwarded message ----------
From: Steven Berryman <[hidden email]>
Date: Sat, Feb 18, 2012 at 6:07 AM
Subject: RE: Vivport V Istazaar
To: dirk newbold <[hidden email]>

Dirk,

As mentioned I’m concerned with the approach you have taken with
Vivport and in one way this technology can lead you into a false sense
of how easy things are (trust me you couldn’t have done it in .NET or
java without infrastructure). The difference with the ecommerce site
is that there is little shared data and I knew that the only real
issue was storing users but that would be a simple mapping issue to a
table. Even the registered objects were designed to make storing in
the database easier so objects didn’t directly hold onto a registered
object. What you have built is a very complicated network of objects
for the benefit of the UI which for a prototype is great but would
never work in the real world. The type of project you are building
with Vivport is infrastructure and in one way the last thing you
should have done is the UI. One system I worked on had 3-4 months of
designing the infrastructure before a line of code was written.



The problem you have is that you have assumed that all objects would
be in memory and are accessible via threads and this just isn’t
possible. If I look at my local Microsoft exchange server it’s
currently 50GB for 5 users and the machine only has 4GB of ram so not
only do you need to deal with large amounts of data but you have not
addressed any multi-threading issues. For example Smalltalk provides
structures such as SharedQueue to deal with any shared data in the
memory and I’m guessing you didn’t use these? When everything is in
memory the world is very easy but for a system like yours you would
need to start with the database and design the schema. Most searches
would be done directly in sql (assuming you are using a relational
database) so that way you reduce the amount of data going to the
client. So when you open up Microsoft Outlook it will run a query on
the database to get your inbox and the email headers so you don’t pull
large amounts of data across the network. As you start to click on
emails it will pull the data over as required.

So here is an example to show multi-threading issues (I have not used
your code but I’m sure you get the point)

Lets’ assume you have a folder with 10 items

Dirk - actions                                  Danny - actions
folder items do: [ :item
“goes through this loop twice”
                                                    delete the last
item in the folder
“code falls over because the last element
in the collection does not exist.” ]

I’m sure you could try something like the above but you would need to
put in a break point in the Dirk loop so you could then go in and to
the removal from a different session. Now imagine the same thing with
5,000 users logged into your server all running threads across your
code. Your code would have to be thread safe and this is not a simple
task especially trying to do it on code not designed that way from the
beginning. You could end up locking up all the other sessions if you
don’t carefully design and keep your critical sections to a minimum.

If you are using collections that are hashed they will fail in very
strange ways if you have other threads removing or adding items as you
are iterating through the collection. Basically in the ecommerce site
we could sort of ignore the shared data issue as we shared very little
data between sessions.

Also you have added features that would be very expensive when using
databases. For example the running man relies on the status of some
piece of data and of course when everything is in the same memory
space this is easy. In a database you would have to cross a
transaction boundary and even the fastest database servers we have
(costing 200k per server) you would be lucky for a single transaction
to take between 1-10 milliseconds (costly when you scale up users and
make too many database calls). This is why Microsoft exchange works
the way it does as you can imagine how slow it would be if 5000 people
logged into the server at the same time. So for most document
management systems they would have a large database at the back end
and all the searches would be done as stored procedures that could be
based on some proprietary algorithm and none of this would be done in
the language itself such as Smalltalk. Until you have a database
schema designed you can’t even think about the UI.

If you want to prove to yourself then try and create 100,000 users all
with about 10MB of data and test to see if it works. See if you can
log in a few sessions and send emails around.

With the ecommerce site I could imagine how to add a database at the
backend because it would be very simple. I would have no idea how you
get this into a database even if you used an object database you would
still need a defined object model. It’s also the reason why I liked
the ecommerce site as it would be simple from the database end as it
would only contain users, basic static data and designer’s data. Also
I can see how we could have multiple servers to handler more traffic
with the ecommerce site but would have no idea how we could do the
same with your design.

Hope this makes sense. Let’s catch up next week.

Steve
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Hi Dirk,

It is certainly true that having a relatively clean separation between the layers (infrastructure, domain, application, and UI) is a good design principle. And Steve is right that building a complex network of objects that cannot fit in memory is not very scalable.

On the other hand, if you have a tool that allows you to act like you have a huge transactional object space that is shared by hundreds of concurrent sessions, where your "stored procedures" are written in Smalltalk, and Seaside support is built-in, then your prototype might not be so unrealistic. Have you heard of GemStone/S? Check out http://seaside.gemstone.com/, and the various links from there, including my blog at http://programminggems.wordpress.com/ and Dale's blog at http://gemstonesoup.wordpress.com/.

Feel free to ask more detailed questions...

James


On Feb 22, 2012, at 3:33 PM, dirk newbold wrote:

> Hi all,
>
> I’m a relative newbie (4 years) to programming and seaside, who
> partnered up with some “old hands” (smalltalk/seaside is my first and
> only language).
>
> Initially we developed an Ecommerce (Istazaar) fashion platform for
> designers to maintain their collections.  We stagnated last year due
> to the time restraints on the “old hands” and I then began developing
> a web based Enterprise Software (Vivport) platform for
> document-management/knowledge-management/emails/tasks.
>
> We now have two ventures at relatively equal stages of development.
> I believe the Enterprise Software (Vivport) platform is the more
> innovative however, now that Steve (one of the “old hands”) has some
> time again he has been able to review its development (which
> unfortunately I built unsupervised) and has major concerns in relation
> to deploying this venture.
>
> Instead of building the database; then building middleware, then UI;
> I’ve basically just built a UI, Steve’s concerns with the outcomes of
> this development process are outlined in his email to me below.
>
> We were wondering if the smalltalk/seaside community has any views on
> Steve’s concerns below that may influence our decision on which
> venture to proceed with.
> The platform has been developed to freely change between the user’s
> personal account and their company(ies) account.  The personal
> accounts are free so we would be attempting to server many users.
>
> Any advice in relation to users/image, persistence, database, general
> deployment etc for this type of venture (essentially a beefed up
> gmail/cloud environment) would be greatly appreciated.
>
> Thanks for your time.
>
> Cheers,
>
> Dirk
>
>
> ---------- Forwarded message ----------
> From: Steven Berryman <[hidden email]>
> Date: Sat, Feb 18, 2012 at 6:07 AM
> Subject: RE: Vivport V Istazaar
> To: dirk newbold <[hidden email]>
>
> Dirk,
>
> As mentioned I’m concerned with the approach you have taken with
> Vivport and in one way this technology can lead you into a false sense
> of how easy things are (trust me you couldn’t have done it in .NET or
> java without infrastructure). The difference with the ecommerce site
> is that there is little shared data and I knew that the only real
> issue was storing users but that would be a simple mapping issue to a
> table. Even the registered objects were designed to make storing in
> the database easier so objects didn’t directly hold onto a registered
> object. What you have built is a very complicated network of objects
> for the benefit of the UI which for a prototype is great but would
> never work in the real world. The type of project you are building
> with Vivport is infrastructure and in one way the last thing you
> should have done is the UI. One system I worked on had 3-4 months of
> designing the infrastructure before a line of code was written.
>
>
>
> The problem you have is that you have assumed that all objects would
> be in memory and are accessible via threads and this just isn’t
> possible. If I look at my local Microsoft exchange server it’s
> currently 50GB for 5 users and the machine only has 4GB of ram so not
> only do you need to deal with large amounts of data but you have not
> addressed any multi-threading issues. For example Smalltalk provides
> structures such as SharedQueue to deal with any shared data in the
> memory and I’m guessing you didn’t use these? When everything is in
> memory the world is very easy but for a system like yours you would
> need to start with the database and design the schema. Most searches
> would be done directly in sql (assuming you are using a relational
> database) so that way you reduce the amount of data going to the
> client. So when you open up Microsoft Outlook it will run a query on
> the database to get your inbox and the email headers so you don’t pull
> large amounts of data across the network. As you start to click on
> emails it will pull the data over as required.
>
> So here is an example to show multi-threading issues (I have not used
> your code but I’m sure you get the point)
>
> Lets’ assume you have a folder with 10 items
>
> Dirk - actions                                  Danny - actions
> folder items do: [ :item
> “goes through this loop twice”
>                                                     delete the last
> item in the folder
> “code falls over because the last element
> in the collection does not exist.” ]
>
> I’m sure you could try something like the above but you would need to
> put in a break point in the Dirk loop so you could then go in and to
> the removal from a different session. Now imagine the same thing with
> 5,000 users logged into your server all running threads across your
> code. Your code would have to be thread safe and this is not a simple
> task especially trying to do it on code not designed that way from the
> beginning. You could end up locking up all the other sessions if you
> don’t carefully design and keep your critical sections to a minimum.
>
> If you are using collections that are hashed they will fail in very
> strange ways if you have other threads removing or adding items as you
> are iterating through the collection. Basically in the ecommerce site
> we could sort of ignore the shared data issue as we shared very little
> data between sessions.
>
> Also you have added features that would be very expensive when using
> databases. For example the running man relies on the status of some
> piece of data and of course when everything is in the same memory
> space this is easy. In a database you would have to cross a
> transaction boundary and even the fastest database servers we have
> (costing 200k per server) you would be lucky for a single transaction
> to take between 1-10 milliseconds (costly when you scale up users and
> make too many database calls). This is why Microsoft exchange works
> the way it does as you can imagine how slow it would be if 5000 people
> logged into the server at the same time. So for most document
> management systems they would have a large database at the back end
> and all the searches would be done as stored procedures that could be
> based on some proprietary algorithm and none of this would be done in
> the language itself such as Smalltalk. Until you have a database
> schema designed you can’t even think about the UI.
>
> If you want to prove to yourself then try and create 100,000 users all
> with about 10MB of data and test to see if it works. See if you can
> log in a few sessions and send emails around.
>
> With the ecommerce site I could imagine how to add a database at the
> backend because it would be very simple. I would have no idea how you
> get this into a database even if you used an object database you would
> still need a defined object model. It’s also the reason why I liked
> the ecommerce site as it would be simple from the database end as it
> would only contain users, basic static data and designer’s data. Also
> I can see how we could have multiple servers to handler more traffic
> with the ecommerce site but would have no idea how we could do the
> same with your design.
>
> Hope this makes sense. Let’s catch up next week.
>
> Steve
> _______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside