Smalltalk › Squeak › Squeak - Dev

[squeak-dev] ANN: SqueakDBX

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

10 messages Options

EstebanLM

[squeak-dev] ANN: SqueakDBX

We are very glad to announce the first beta release of SqueakDBX.

The aim of this project is to build an OpenDBX (http://www.linuxnetworks.de/doc/index.php/OpenDBX/) plugin which will allow users to perform relational database operations (DDL, DML and SQL) through a truly open source library.

This project has been selected as part of ESUG, SummerTalk 2008 (please, visit http://www.esug.org/Promotion/SummerTalk).

Through this project, the squeak community will hopefully be able to interact with major database engines such as Oracle and MSSQL besides open sources Postgresql and MySQL. Moreover, integrating this plugin with GLORP (http://www.glorp.org/), will allow us to generate a complete and open source solution to relational data base access.

We have been working in this project since january and we finally get our first beta release. This release does not have all the features we want it to have, just some of them, but we think is very usable.

The features for this release are:

Tested on 3.10 and Pharo.
Support for Linux and OSX.

Proved on windows (through MinGW), but some changes on OpenDBX are still needed (next version will have full compatibility).

Tested on PostgreSQL, MySQL and Oracle.

MS SQL Server, Firebird, Interbase, SQLite, SQLite3 and Sybase tests will be available as soon as possible.

Transactional management.
Automatic conversion of resultset columns (a String) into squeak types.

Large objects (clob, blob, arrays, and so on) are not yet supported.

Special OpenDBX options: multi-statments, compression, paged results.
Automated database connection release on garbage collection (although manual disconnection is recommended ;-)
Error handling

Good news:

Lots of unit tests (82 right now, and more coming)
The code is commented
Full documentation at wiki
There are some benchmarks done and it seems to be as fast as native driver
There are some benchmarking (more needed), but shows we are reaching good performance (http://wiki.squeak.org/squeak/6063)

We think its important for you to know:

We just test with MySQL 5, Postgres 8.3 and Oracle 10g XE.
We just test with Ubuntu and OSX, and very few (not really enough) in windows.

Special thanks. We really want to thanks to:

Of course, Stephane Ducasse and all ESUG people, for the interest (and funding) showed on this project.
Norbert Sendetzky (OpenDBX author) because all the help he gave us not only with OpenDBX but also with SqueakDBX.

Help wanted. We are really convinced about open source projects. A software project is better when people share opinions, work, advice, comments, etc.

We would really appreciate if you:

Give us opinions, comments, ideas, new features, complaints and so on.
tell us if you find a bug.
tell us if you test SqueakDBX with other RDMBS or OS different from the ones we tested. Just to know if it works or not :)

Full documentation, installation and getting started instructions can be found at wiki page: http://wiki.squeak.org/squeak/6052

Packages can be installed from Universe (3.10) or SqueakMap. Current version is 0.7.1

Sources can be download from SqueakSource http://www.squeaksource.com/SqueakDBX (it requires FFI installed)

Many thanks,

SqueakDBX team

Ramon Leon-5

RE: [squeak-dev] ANN: SqueakDBX

> Sources can be download from SqueakSource
> http://www.squeaksource.com/SqueakDBX
> <http://www.squeaksource.com/SqueakDBX> (it requires FFI installed)
>
> Many thanks,
> SqueakDBX team

So by using FFI it locks the VM and serializes all the requests through a
single pipe allowing a long running query to block all other queries
essentially freezing the VM?

Ramon Leon
http://onsmalltalk.com

Andreas.Raab

[squeak-dev] Re: ANN: SqueakDBX

Ramon Leon wrote:

>> Sources can be download from SqueakSource
>> http://www.squeaksource.com/SqueakDBX
>> <http://www.squeaksource.com/SqueakDBX> (it requires FFI installed)
>>
>> Many thanks,
>> SqueakDBX team
>
> So by using FFI it locks the VM and serializes all the requests through a
> single pipe allowing a long running query to block all other queries
> essentially freezing the VM?

Depends on whether OpenDBX (or the driver) support asynchronous mode.

Cheers,
- Andreas

Ramon Leon-5

RE: [squeak-dev] Re: ANN: SqueakDBX

> > So by using FFI it locks the VM and serializes all the
> requests through a
> > single pipe allowing a long running query to block all other queries
> > essentially freezing the VM?
>
> Depends on whether OpenDBX (or the driver) support asynchronous mode.
>
> Cheers,
> - Andreas

But of course, hence the question mark!

Ramon Leon
http://onsmalltalk.com

EstebanLM

[squeak-dev] Re: ANN: SqueakDBX

Hi,
The driver uses asynchronous calls for comunicating with it's
respective engines, as far as the engine allow it, that's one of the
powerful features it has.
This is how we think a query must be:

1. call to execute- an external call
2. wait for answer (a request to ask if there is already an answer,
otherwise, yields for N time)- another external call
2.1. loop step 2 until there is an answer.
3. retrieve result rows, one by one -an external call each one

Of course, all of this calls are through FFI, and therefore,
blocking... but is better to have many mini-blocks than just one large
block.

That's the theory... right now, the fact is that very large operations
on same connection are freezing the image... we are working on that :)

Cheers,
Esteban

On 2008-08-20 15:09:17 -0300, "Ramon Leon" <[hidden email]> said:

>
>>> So by using FFI it locks the VM and serializes all the
>> requests through a
>>> single pipe allowing a long running query to block all other queries
>>> essentially freezing the VM?
>>
>> Depends on whether OpenDBX (or the driver) support asynchronous mode.
>>
>> Cheers,
>> - Andreas
>
> But of course, hence the question mark!
>
> Ramon Leon
> http://onsmalltalk.com

EstebanLM

[squeak-dev] Re: ANN: SqueakDBX

But...
I just tested this:

[ self executeAndRetreiveAVeryLongQuery ] forkAt: self userBackgroundPriority.

And it works perfect!
No freeze is observed :)

And parallel operations seems to work too... so out "real-tiny-blocks"
approuch seems to work.

Cheers,
Esteban

On 2008-08-20 16:00:16 -0300, Esteban Lorenzano <[hidden email]> said:

> Hi,
> The driver uses asynchronous calls for comunicating with it's
> respective engines, as far as the engine allow it, that's one of the
> powerful features it has.
> This is how we think a query must be:
>
> 1. call to execute- an external call
> 2. wait for answer (a request to ask if there is already an answer,
> otherwise, yields for N time)- another external call
> 2.1. loop step 2 until there is an answer.
> 3. retrieve result rows, one by one -an external call each one
>
> Of course, all of this calls are through FFI, and therefore,
> blocking... but is better to have many mini-blocks than just one large
> block.
>
> That's the theory... right now, the fact is that very large operations
> on same connection are freezing the image... we are working on that :)
>
> Cheers,
> Esteban
>
> On 2008-08-20 15:09:17 -0300, "Ramon Leon" <[hidden email]> said:
>
>>
>>>> So by using FFI it locks the VM and serializes all the
>>> requests through a
>>>> single pipe allowing a long running query to block all other queries
>>>> essentially freezing the VM?
>>>
>>> Depends on whether OpenDBX (or the driver) support asynchronous mode.
>>>
>>> Cheers,
>>> - Andreas
>>
>> But of course, hence the question mark!
>>
>> Ramon Leon
>> http://onsmalltalk.com

EstebanLM

[squeak-dev] Re: ANN: SqueakDBX

On 2008-08-20 16:16:00 -0300, Esteban Lorenzano <[hidden email]> said:

> But...
> I just tested this:
>
> [ self executeAndRetreiveAVeryLongQuery ] forkAt: self userBackgroundPriority.
>
> And it works perfect!
> No freeze is observed :)
>
> And parallel operations seems to work too... so out "real-tiny-blocks"
> approuch seems to work.

I meant: our "real-tiny-blocks" approach :)

>
> Cheers,
> Esteban
>
> On 2008-08-20 16:00:16 -0300, Esteban Lorenzano <[hidden email]> said:
>
>> Hi,
>> The driver uses asynchronous calls for comunicating with it's
>> respective engines, as far as the engine allow it, that's one of the
>> powerful features it has.
>> This is how we think a query must be:
>>
>> 1. call to execute- an external call
>> 2. wait for answer (a request to ask if there is already an answer,
>> otherwise, yields for N time)- another external call
>> 2.1. loop step 2 until there is an answer.
>> 3. retrieve result rows, one by one -an external call each one
>>
>> Of course, all of this calls are through FFI, and therefore,
>> blocking... but is better to have many mini-blocks than just one large
>> block.
>>
>> That's the theory... right now, the fact is that very large operations
>> on same connection are freezing the image... we are working on that :)
>>
>> Cheers,
>> Esteban
>>
>> On 2008-08-20 15:09:17 -0300, "Ramon Leon" <[hidden email]> said:
>>
>>>
>>>>> So by using FFI it locks the VM and serializes all the
>>>> requests through a
>>>>> single pipe allowing a long running query to block all other queries
>>>>> essentially freezing the VM?
>>>>
>>>> Depends on whether OpenDBX (or the driver) support asynchronous mode.
>>>>
>>>> Cheers,
>>>> - Andreas
>>>
>>> But of course, hence the question mark!
>>>
>>> Ramon Leon
>>> http://onsmalltalk.com

Ramon Leon-5

RE: [squeak-dev] Re: ANN: SqueakDBX

In reply to this post by EstebanLM

>
> [ self executeAndRetreiveAVeryLongQuery ] forkAt: self
> userBackgroundPriority.
>
> And it works perfect!
> No freeze is observed :)

But will this work...

[ self executeAndRetreiveAVeryLongQuery ] forkAt: self
userBackgroundPriority.

[ self executeAndRetreiveAShortQuery ] forkAt: self userBackgroundPriority.

And the short query finish first even though the longer one started first?

Ramon Leon
http://onsmalltalk.com

EstebanLM

[squeak-dev] Re: ANN: SqueakDBX

He, I'm testing it now... in a while I upload results :)

On 2008-08-20 17:13:31 -0300, "Ramon Leon" <[hidden email]> said:

>>
>> [ self executeAndRetreiveAVeryLongQuery ] forkAt: self
>> userBackgroundPriority.
>>
>> And it works perfect!
>> No freeze is observed :)
>
> But will this work...
>
> [ self executeAndRetreiveAVeryLongQuery ] forkAt: self
> userBackgroundPriority.
>
> [ self executeAndRetreiveAShortQuery ] forkAt: self userBackgroundPriority.
>
> And the short query finish first even though the longer one started first?
>
> Ramon Leon
> http://onsmalltalk.com

EstebanLM

[squeak-dev] Re: ANN: SqueakDBX

Ok, new tiny-blocks tests:

I tested:

1) Inserting 100.000 rows into two tables.
2) Select 100.000 rows from a table.

First of all, running with background priority image isn't freezing,
and FFI calls are not perceived. This is important because this allows
me to still render seaside pages (for example) for another user, when
there is one user making a big query.
Second, I followed rows insertion (simple "select count(*) from bla1"
in psql tool) and the insertions is happening in parallel.

Of course FFI calls are still happening, and there are many
"micro-blocks", so the total time expended in each process is equal to
the addition of all FFI blocks + squeak calculations. But is important
to note that (2 parallel queries time) < (query1 time + query2 time)...

I'm going to put all this benckmarks in the wiki page (and the sources
in squeaksource)

Cheers,
Esteban

On 2008-08-20 17:17:13 -0300, Esteban Lorenzano <[hidden email]> said:

> He, I'm testing it now... in a while I upload results :)
>
> On 2008-08-20 17:13:31 -0300, "Ramon Leon" <[hidden email]> said:
>
>>>
>>> [ self executeAndRetreiveAVeryLongQuery ] forkAt: self
>>> userBackgroundPriority.
>>>
>>> And it works perfect!
>>> No freeze is observed :)
>>
>> But will this work...
>>
>> [ self executeAndRetreiveAVeryLongQuery ] forkAt: self
>> userBackgroundPriority.
>>
>> [ self executeAndRetreiveAShortQuery ] forkAt: self userBackgroundPriority.
>>
>> And the short query finish first even though the longer one started first?
>>
>> Ramon Leon
>> http://onsmalltalk.com