[squeak-dev] Threaded ODBC

2008/5/11, Klaus D. Witzel <[hidden email]>:
UI updates in general have to happen in a very specific thread.

> Furthermore, you'll earn credit if
> you can predict the 100% base and the respective increments, of arbitrary db
> queries which are performed on some dedicated server multi-user, predict
> with a small epsilon (not like MS$ does it when they copy/move around
> files).

Progress bars in general have a indeterminate mode for such cases.

On Sun, 11 May 2008 12:47:29 +0200, Philippe Marschall wrote:

> 2008/5/11, Klaus D. Witzel <[hidden email]>:
[...]
Sure.

>> Furthermore, you'll earn credit if
>> you can predict the 100% base and the respective increments, of  
>> arbitrary db
>> queries which are performed on some dedicated server multi-user, predict
>> with a small epsilon (not like MS$ does it when they copy/move around
>> files).
>
> Progress bars in general have a indeterminate mode for such cases.

NP with me but, this is just an excuse from the IT-shamanes (not you nor  
me nor anybody else reading this ;) , for their inability to predict  
*time*.

On 5/11/08, Klaus D. Witzel <[hidden email]> wrote:

Out of curiosity, what would it be that you intend to let the users do while longish queries are tickling the db?

I ask with this background in mind: over several years I headed a team of developers who produced 1.5 releases/year of a (fat+GUI)client<->(shared+fast)db server app with between 6 and 60 GB data etc, 50+ installation sites (institutions with their own db server) and 100+ users, with 98% of the db data being updated over night (incrementally). We brought down the app's db response time from minutes to seconds but still never found any reason for using non-blocking queries, because nothing much was there that could have been done by the user in the meantime.

Do you just want to see non-freezing queries running or have some other reason for that need?

4.  One of our "big" applications is for data collection tools for regulatory reporting.  Much of our data isn't even IN a computer yet.  Employees abstract the data by hand from paper medical records.  We are trying to get SOME of what they need if it is available by generating nightly "work" lists as new patients are "coded" and end up in various reporting populations, extracting what data we can (either from pre-built queries or dealing with the special cases mentioned above), and presenting Aida-based "forms" for them to fill in the remaining data. At that point, I defer to the arguments presented already with regards to the user "waiting"
for their form to load... 

Then, of course, we will feed the combined data back into the warehouse...

So, in the end, a lot of our "customers" aren't users at all, but just us, the programmers, interactively using Smalltalk as a tool or developing other nightly processing tools.  I just liked the threaded connections I tried out in VisualWorks and 
Smalltalk/X because it made it easier form ME to work with the image, so I just wondered how hard it would be to implement.

Thanks for asking the question, though; I have been asking the same one of myself for several months now as we have been struggling to "choose" a Smalltalk product.  One thing is for sure, though, Smalltalk is DEFINITELY the right tool for the right job amidst the natural objects and subclasses of healthcare!

Hope some of that made sense,

Rob

Klaus,

I would be happy to help any way I can!  I and co-worker have access to Oracle, SQL server, Postgres, MySQL, DB2, and KBase data sources, to name a few.  And we are definitely good at "breaking" things.

Not so long ago, I was a decent assembly language programmer, but have never written a Squeak plugin before and still consider myself to be on the "novice" end of the Smalltalk spectrum, although I am starting to turn the corner.

Thanks for the invitation,

Rob

On 5/11/08, Klaus D. Witzel <[hidden email]> wrote:

On Sat, 10 May 2008 21:08:53 +0200, Rob Rothwell wrote:

On 5/10/08, Igor Stasenko wrote:

2008/5/10 Rob Rothwell :

I can help with windoze port of openDbx. Lets hope its wouldn't be too
hard.
Also i can help with making generic binding of openDBX to squeak by
implementing a plugin.
But i'm not yet ready to start with it :)

Sounds good.  Is there anything I can try to learn in the meantime that
would make me more useful?

Yes, sure. Igor+my have decided to build a new DB plugin for the MS$ platform, so that we perhaps no longer hear complaints about missing MS$+DB support (for the major/open DB vendors) in Squeak. You are invited to join us, we could make use of some input and also help with non-toy DB testing (and whatever you'd like to share or help with).

This is also an invitation to join the project, for Squeakers who asked for Oracle support on their MS$ platform.

Send email please.

Hi all,
Sorry for jumping to this thread so late, but it was weekend and I was
far from my computer :)
A little team of students and I are developing SqueakDBX... It is just
the begining, but we hope to provide a realiable DB plugin, who uses
asyncronic calls for performing the queries (and OpenDBX do that). This
project was rejected from SoC (the comunity felt that other projects
were more importat), but we kept working on, and now it is listed as a
possible SummerTalk project.
Currently, the code in www.squeaksource.com/SqueakDBX is just a proof
of concept and little more, but we are very exited about the preliminar
results. Of course, OpenDBX currently runs only on *nix systems (and
OSX, of course), and not windows, but I think it wont be difficult to
migrate it to windoze... Norbert, the mantainer of OpenDBX is very
helpful.

I invite you (Igor, Rob, Klaus and all the interested in developing a
modern DB plugin) to join efforts, not duplicating them.

Thanks,
Esteban

On 2008-05-11 12:02:07 -0300, "Rob Rothwell" <[hidden email]> said:

I'm not reading all the threads here, but certainly the Hydra VM work  
could dovetail nicely with work on database queries that lock up the  
image thus causing issues with serving up web requests.


On May 11, 2008, at 2:09 AM, Janko Mivšek wrote:

--
=
=
=
========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
=
=
=
========================================================================

On Mon, 12 May 2008 05:32:26 +0200, Esteban Lorenzano wrote:

> Hi all,
> Sorry for jumping to this thread so late, but it was weekend and I was  
> far from my computer :)
> A little team of students and I are developing SqueakDBX... It is just  
> the begining, but we hope to provide a realiable DB plugin, who uses  
> asyncronic calls for performing the queries (and OpenDBX do that). This  
> project was rejected from SoC (the comunity felt that other projects  
> were more importat),

Well, we asked GSoC for many more slots for our 20+ registered  
applications, got one but not six more, and so many projects (among them  
yours) could not make it.

> but we kept working on, and now it is listed as a possible SummerTalk  
> project.

Good news & good to see that you're keen to "sell" your project :)

> Currently, the code in www.squeaksource.com/SqueakDBX is just a proof of  
> concept and little more, but we are very exited about the preliminar  
> results. Of course, OpenDBX currently runs only on *nix systems (and  
> OSX, of course), and not windows, but I think it wont be difficult to  
> migrate it to windoze... Norbert, the mantainer of OpenDBX is very  
> helpful.

The way I see it (and I offered that opinion to Igor in email) is, that  
the OpenDBX project doesn't see much value in the MS$ platform. So many  
releases but no attempt for a port, not even a plan on their homepage (and  
Slashdot doesn't know about OpenDBX). You might want to take the many  
failed projects into consideration, for example Chilli and ADOdb, to name  
a few. We don't want to be locked in and Igor+my drafted a plan which  
explicitly avoids that and lets us put our effort *only* on our parts and  
*not* on theirs (except bug reports).

None of us (incl. your team?) is in the professional DB business and all  
we can do is use existing modules (like OpenDBX) as soon as there is  
years-long support available and they proved to perform very well on the  
MS$ platform and receive the respective DB vendor(s) support. We're  
talking about very expensive enterprise DB's and that's a point at which  
I'm not willing to accept a single compromise in our project.

> I invite you (Igor, Rob, Klaus and all the interested in developing a  
> modern DB plugin) to join efforts, not duplicating them.

Last night Igor designed the module layer and convinced me that OpenDBX  
can easily be made an external module (mini-driver) once it's available.  
Our project intends to support more than just one such mini-driver,  
loadable on demand. Part of this is Igor's mini-driver API which is all  
that needs to be adapted for making a new external DB module. And Igor  
insists on our plugin to manage multiple native threads and their db  
sessions; wink, blink ;)

Now, unfortunately MS$ is the platform that Rob and Frank want *now* and  
so OpenDBX is, for a loong, loong time, out of reach for us. This is also  
the reason behind our project, the *absence* of MS$ platform support.

We intend to avoid duplication by matching your project with our draft,  
and then see what can be re-used or needs to be rewritten + added. Perhaps  
your team can jump in with review+feeback, that would be great! :)

/Klaus
[...cut the HTML part...]

On Mon, May 12, 2008 at 4:33 AM, Klaus D. Witzel <[hidden email]> wrote:

2008/5/12 Rob Rothwell <[hidden email]>:
Yes, that's a needed functionality, because different database
frontends may or may not support threading.
But we want VM to never be blocked by a calls to crappy modules, right? :)

Yes, and some functionality can be visible at higher levels, so you
potentially can choose what mini-driver to use.

>
> And now, without knowing anything about your design, and probably sounding
> foolish, could your "transaction manager" be written in Squeak itself,
> running headless and passing requests to the appropriate drivers?
>

Not sure what is 'transaction manager' here. But surely, you can make
own abstraction for working with DBs in squeak.
The plugin API should serve as a uniform way for talking with
different DB frontends via minidrivers.

>From my mail to Klaus:

In ST code i can write:

DBManager>>useMiniDriver: aLibraryPath
<primitive: #connectMiniDriver>

this primitive should load a OS shared library, and using known
minidriver API ask for its capabilities and DB types it supports,
so, then users of plugin can write:

self ifWin32: [ driver := DBManager useMiniDriver: 'ADO.dll'. ].
driver enumSupportedDBTypes includes: 'MSSQL' ifTrue: [ .... ]

this will be a low-level support code in ST. A high level code would be

connect: dbType dbHost: ... dbName: ... dbUser: ... dbPass: .. options: ..

or, if users know what they do:

connectUsingLibrary: 'mysqlMiniDriver.so' dbHost: ... dbName: ...
dbUser: ... dbPass: .. options: ..

In this way, we implement plugin once, and as product we having:
- ST package
- plugin
- an API for minidrivers

then developers can implement own minidrivers using pure C - simply by
adopting API with external library they use.

>> Now, unfortunately MS$ is the platform that Rob and Frank want *now* and
>> so OpenDBX is, for a loong, loong time, out of reach for us. This is also
>> the reason behind our project, the *absence* of MS$ platform support.
>
> This is true!
>


--
Best regards,
Igor Stasenko AKA sig.

Hi Matthew -

This is very interesting work. Your script presents a unique opportunity
to actually compare the traitified version of the class kernel with one
that doesn't have these modifications applied and compare notes
directly. I am still in the process of doing that, but one thing that I
almost instantly noticed was the subjective feeling that "sanity had
suddenly returned" to a previously insanely complex part of the system.
And I'm slowly starting to understand why that is.

Let's do a little exercise: One thing I regularly do when there are
changes to the class kernel is to go through the main hook for method
addition (addSelector:withMethod:) to see if there are any changes along
the way. When you do this in the image without traits you end up with an
exploration chain like here:

* addSelector:withMethod: has one implementor which calls
addSelector:withMethod:notifying:
* addSelector:withMethod:notifying: has two implementors (in Behavior as
well as ClassDescription) where the latter is basically just adding the
SystemChangeNotifier around its superclass implementation. Both versions
call addSelectorSilently:withMethod:
* addSelectorSilently:withMethod: has the same pattern (the base
implementation in Behavior and an added notification in
ClassDescription) which calls methodDictAddSelectorSilently:withMethod:
* methodDictAddSelectorSilently:withMethod: calls then
#basicAddSelector:withMethod: which ultimately does the work.

Coincidentally, the last two methods are completely superfluous and
could just be folded into Behavior>>addSelectorSilently:withMethod:
(which is the implementation in 3.8).

The total number of entities to deal with in this case is seven methods
(five if you discount the last two) in two classes.

Now let's do the same for the traits version:
* addSelector:withMethod: starts out with showing four implementors; two
traits (TCompilingBehavior and TPureBehavior) as well as two classes,
TraitBehavior and Behavior. The implementations look identical but it is
not clear if they are and how one would actually find out.
* addSelector:withMethod:notifying: shows *seven* implementors; but with
only two obviously different versions which are the ones ultimately used
by Behavior and ClassDescription
* addSelectorSilently:withMethod: shows again seven implementor; this
time with four apparently different implementations
* methodDictAddSelectorSilently:withMethod: shows three implementors
with only one apparent implementation of the method
* and finally, basicAddSelector:withMethod: shows another four
implementors with only one apparent version of the method.

Total number of entities dealt with in the process: 25 methods, 4
classes, 3 traits. Even if you ignore those parts that aren't present in
the first image, you still end up with 18 methods, 2 classes, 3 traits.
In other words, the relative complexity one needs to deal with to
perform the same task is about 3-5 times higher.

I think that's the main problem that I have with the implementation at
hand. Any measure that I can think of to compare complexity comes up
with at least a doubling of the complexity introduced in the class
kernel, many of them show *much* larger increases. This is very similar
to the issues observed here:

http://lists.squeakfoundation.org/pipermail/squeak-dev/2007-August/119295.html

where the amount of complexity explodes similarly. I'm not completely
sure at this point whether the issues should be attributed to lack of
tool support, or whether they are just the natural result of using MI
inappropriately. I currently tend towards the latter because even with
better tools significant amounts of extra complexity would remain and
there is no real justification for the use of MI in that part of the system.

In any case, this is truly a *great* opportunity for people who haven't
made up their mind to compare both versions directly and see which one
they prefer.

Cheers,
   - Andreas

Matthew Fulmer wrote:

On 2008-05-12 05:33:21 -0300, "Klaus D. Witzel" <[hidden email]> said:

He, I wasn't complaining... just mentioning it. Maybe my limitatios in
english made it looks as a complain, but I far away of that feeling

Yes, I insist,  we can join our efforts, maybe we can collaborate not
just testing but working on the design/devolopment of squeak side of
the plugin.
This mini-driver intent to be capable to manage diferent backends such
as ADO, OpenDBX, etc?
We can drive our design to match that plugin to.
Anyway I recoment you take a look at the design of OpenDBX, it is very
easy and clear, and maybe, at least, a place of inspiration.

On Mon, 12 May 2008 14:48:33 +0200, Esteban Lorenzano wrote:

NP it was just an explanation for what "rejected" means asked by someone  
in the audience.
[...]
>>  We intend to avoid duplication by matching your project with our  
>> draft,  and then see what can be re-used or needs to be rewritten  
>> + added. Perhaps  your team can jump in with review+feeback, that would  
>> be great! :)
>
> Yes, I insist,

Heh! :) Okay, collaboration is always appreciated.

> we can join our efforts, maybe we can collaborate not just testing but  
> working on the design/devolopment of squeak side of the plugin.

Some hours ago a consolidated draft was sent to team members, at this  
point in time we should feel free from having to consider "prior art" and  
I'd appreciate users bring in their priorities from daily (nightly!)  
routine (Rob has talked about many of his requirements and expectations in  
this thread, Frank has Oracle as priority #1).

> This mini-driver intent to be capable to manage diferent backends such  
> as ADO, OpenDBX, etc?

Precisely (well, who said that his mini-driver API adaptation of  
msado15.dll will be a piece of cake ;)

> We can drive our design to match that plugin to.

Lots of synergy, yes no doubt.

> Anyway I recoment you take a look at the design of OpenDBX, it is very  
> easy and clear, and maybe, at least, a place of inspiration.

Yes, is on the plan after our draft has strong support within the team.  
You'll be cc'ed as soon as wiki.squeak.org has the news (presumably not  
before S3 in Potsdam this week).

>>  /Klaus
>>
>>> Thanks,
>>> Esteban
[...]