Garage, Glorp, PostgreSQL and Cursors

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Garage, Glorp, PostgreSQL and Cursors

Tommaso DS
Hello,

I was trying the version of Glorp ported to Garage, and I must say it
runs smoothly, nice job!

I am reading large amount of data from a Postgres db, and I was curious
about the use of cursors.
I remember that the old version of Glorp for Pharo did not support
cursors in Postgresql, has this changed in Garage?

I tried to run some tests, using a query like:

(Query readManyOf: MyClass)
    collectionType: GlorpCursoredStream;
    yourself.

but with a large number of rows (~300k), the whole image crashes.

Is this still not supported, or am I doing something wrong?

Thanks,
Tommaso

Reply | Threaded
Open this post in threaded view
|

Re: Garage, Glorp, PostgreSQL and Cursors

Yuriy Mironenko
I'm little bit shy, but what is Garage?

2015-08-03 15:33 GMT+03:00 Tommaso Dal Sasso <[hidden email]>:
Hello,

I was trying the version of Glorp ported to Garage, and I must say it
runs smoothly, nice job!

I am reading large amount of data from a Postgres db, and I was curious
about the use of cursors.
I remember that the old version of Glorp for Pharo did not support
cursors in Postgresql, has this changed in Garage?

I tried to run some tests, using a query like:

(Query readManyOf: MyClass)
    collectionType: GlorpCursoredStream;
    yourself.

but with a large number of rows (~300k), the whole image crashes.

Is this still not supported, or am I doing something wrong?

Thanks,
Tommaso


Reply | Threaded
Open this post in threaded view
|

Re: Garage, Glorp, PostgreSQL and Cursors

vmusulainen
03/08/15 16:10, Yuriy Mironenko [via Smalltalk] пишет:
I'm little bit shy, but what is Garage?


http://pharo.org/news/garage-database

2015-08-03 15:33 GMT+03:00 Tommaso Dal Sasso <[hidden email]>:
Hello,

I was trying the version of Glorp ported to Garage, and I must say it
runs smoothly, nice job!

I am reading large amount of data from a Postgres db, and I was curious
about the use of cursors.
I remember that the old version of Glorp for Pharo did not support
cursors in Postgresql, has this changed in Garage?

I tried to run some tests, using a query like:

(Query readManyOf: MyClass)
    collectionType: GlorpCursoredStream;
    yourself.

but with a large number of rows (~300k), the whole image crashes.

Is this still not supported, or am I doing something wrong?

Thanks,
Tommaso





If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/Garage-Glorp-PostgreSQL-and-Cursors-tp4840760p4840762.html
To start a new topic under Pharo Smalltalk Users, email [hidden email]
To unsubscribe from Pharo Smalltalk Users, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Garage, Glorp, PostgreSQL and Cursors

abergel
In reply to this post by Yuriy Mironenko
I'm little bit shy, but what is Garage?

Good question!

Alexandre



2015-08-03 15:33 GMT+03:00 Tommaso Dal Sasso <[hidden email]>:
Hello,

I was trying the version of Glorp ported to Garage, and I must say it
runs smoothly, nice job!

I am reading large amount of data from a Postgres db, and I was curious
about the use of cursors.
I remember that the old version of Glorp for Pharo did not support
cursors in Postgresql, has this changed in Garage?

I tried to run some tests, using a query like:

(Query readManyOf: MyClass)
    collectionType: GlorpCursoredStream;
    yourself.

but with a large number of rows (~300k), the whole image crashes.

Is this still not supported, or am I doing something wrong?

Thanks,
Tommaso



Reply | Threaded
Open this post in threaded view
|

Re: Garage, Glorp, PostgreSQL and Cursors

NorbertHartl
In reply to this post by Yuriy Mironenko

Am 03.08.2015 um 15:11 schrieb Юрий Мироненко <[hidden email]>:

I'm little bit shy, but what is Garage?

http://pharo.org/news/garage-database

Norbert


2015-08-03 15:33 GMT+03:00 Tommaso Dal Sasso <[hidden email]>:
Hello,

I was trying the version of Glorp ported to Garage, and I must say it
runs smoothly, nice job!

I am reading large amount of data from a Postgres db, and I was curious
about the use of cursors.
I remember that the old version of Glorp for Pharo did not support
cursors in Postgresql, has this changed in Garage?

I tried to run some tests, using a query like:

(Query readManyOf: MyClass)
    collectionType: GlorpCursoredStream;
    yourself.

but with a large number of rows (~300k), the whole image crashes.

Is this still not supported, or am I doing something wrong?

Thanks,
Tommaso



Reply | Threaded
Open this post in threaded view
|

Re: Garage, Glorp, PostgreSQL and Cursors

Sean P. DeNigris
Administrator
In reply to this post by vmusulainen
I'm also confused. What is the relationship between OpenDBX and Garage? IIRC DBXTalk was the previous name of OpenDBX, right? Maybe an FAQ is in order as the DB-related names seem to be multiplying ;)
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Garage, Glorp, PostgreSQL and Cursors

EstebanLM

> On 03 Aug 2015, at 15:35, Sean P. DeNigris <[hidden email]> wrote:
>
> vmusulainen wrote
>> http://pharo.org/news/garage-database
>
> I'm also confused. What is the relationship between OpenDBX and Garage? IIRC
> DBXTalk was the previous name of OpenDBX, right? Maybe an FAQ is in order as
> the DB-related names seem to be multiplying ;)

not really.

DBXTalk is the suite that includes the OpenDBX driver (and others)
it also includes Garage, as a common layer for different database drivers, one of them OpenDBX

Esteban

>
>
>
> -----
> Cheers,
> Sean
> --
> View this message in context: http://forum.world.st/Garage-Glorp-PostgreSQL-and-Cursors-tp4840760p4840771.html
> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
>


Reply | Threaded
Open this post in threaded view
|

Re: Garage, Glorp, PostgreSQL and Cursors

Esteban A. Maringolo
In reply to this post by Sean P. DeNigris
Garage is a (IMO lousy named) Pharo database interface abstraction,
like Perl's DBI, ADODB, or JDBC.
Any program that uses Garage's objects to access the database should
be abstract enough to enable you to replace the backend by something
else without changing the programming logic (e.g. replace PostgreSQL
by SQLite).

DBXTalk is the same, but at the database driver level, it uses the
OpenDBX database drivers.

So the relation would be:
DB Interface -> DB Adapter -> DB Driver

With these possible combinations (among others):
Garage -> DBXTalk -> OpenDBX libs (libopendbx.so/dll)
Garage -> PostgreSQLv2 -> (written in Smalltalk)
Garage -> SQLite -> (written in Smalltalk)

The confusion comes also when you add GLORP (ORM) to the combo, which
is the most common use case when dealing with RDBMS, with the added
misnomer of GLORP calling "Driver" what in practice is an Adapter
(pattern) written in Smalltalk .

So you have the following
GlorpDriver (Adapter) -> DB Interface

So:
GlorpDBXTalk -> Garage
GlorpPostgresV2 -> PostgresV2
GlorpSQLite -> PunQLite


The benefits of having an abstraction layer like Garage are many,
maybe the misnomer and the lack of coordination provokes this
confusion.

Regards!

--
Esteban A. Maringolo


















Esteban A. Maringolo


2015-08-03 10:35 GMT-03:00 Sean P. DeNigris <[hidden email]>:

> vmusulainen wrote
>> http://pharo.org/news/garage-database
>
> I'm also confused. What is the relationship between OpenDBX and Garage? IIRC
> DBXTalk was the previous name of OpenDBX, right? Maybe an FAQ is in order as
> the DB-related names seem to be multiplying ;)
>
>
>
> -----
> Cheers,
> Sean
> --
> View this message in context: http://forum.world.st/Garage-Glorp-PostgreSQL-and-Cursors-tp4840760p4840771.html
> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: Garage, Glorp, PostgreSQL and Cursors

Sean P. DeNigris
Administrator
Esteban A. Maringolo wrote
Garage is...
Thank you! Very helpful. It seems like this would be a great clarification to add to http://guillep.github.io/DBXTalk/ , no?
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Garage, Glorp, PostgreSQL and Cursors

EstebanLM
yes… just that is not accurate :)
is basically like that, just that DBXTalk is not a driver, is an umbrella project. The OpenDBX drivers is still called OpenDBXDriver :P

Esteban

> On 03 Aug 2015, at 20:55, Sean P. DeNigris <[hidden email]> wrote:
>
> Esteban A. Maringolo wrote
>> Garage is...
>
> Thank you! Very helpful. It seems like this would be a great clarification
> to add to http://guillep.github.io/DBXTalk/ , no?
>
>
>
> -----
> Cheers,
> Sean
> --
> View this message in context: http://forum.world.st/Garage-Glorp-PostgreSQL-and-Cursors-tp4840760p4840827.html
> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
>


Reply | Threaded
Open this post in threaded view
|

Re: Garage, Glorp, PostgreSQL and Cursors

Esteban A. Maringolo
2015-08-03 16:20 GMT-03:00 Esteban Lorenzano <[hidden email]>:
> yes… just that is not accurate :)
> is basically like that, just that DBXTalk is not a driver, is an umbrella project. The OpenDBX drivers is still called OpenDBXDriver :P

TooManyAbstractionsException signal.

;)

Esteban A. Maringolo

Reply | Threaded
Open this post in threaded view
|

Re: Garage, Glorp, PostgreSQL and Cursors

Guillermo Polito
Helloooooo :)

First Tomasso, it's nice to know it's working well for you :). We need feedback and contributions, even the smallests (like docs, or blogposts with your experiences or tutorials).

Answering your question, GarageGlorp does not yet use the cursors from the Garage drivers. I do not know what exactly should be added to the Glorp's GarageAdapter, maybe we can take a look at it if you are available.

For the rest:

OpenDBX -> it is a database abstraction written in C. It was developed outside the community, though we contribute if necessary. There is also a driver in Pharo providing bindings against this opendbx library.
Garage -> a database abstraction in Pharo. It provides a common interface to talk with several drivers (e.g., opendbx, native postgres, native mysql...)
Glorp -> an object relational mapping framework that connects to a database (therefore it needs some database driver) and converts relational data into objects.
DBXTalk -> the umbrella project of all the above (and their connections)

Cheers,
Guille

El mar., 4 de ago. de 2015 a la(s) 12:36 a. m., Esteban A. Maringolo <[hidden email]> escribió:
2015-08-03 16:20 GMT-03:00 Esteban Lorenzano <[hidden email]>:
> yes… just that is not accurate :)
> is basically like that, just that DBXTalk is not a driver, is an umbrella project. The OpenDBX drivers is still called OpenDBXDriver :P

TooManyAbstractionsException signal.

;)

Esteban A. Maringolo

Reply | Threaded
Open this post in threaded view
|

Re: Garage, Glorp, PostgreSQL and Cursors

Arturo Zambrano
Thanks Guillermo, we were struggling with all these names also.
This reference is important, and IMHO it should be pasted also in the page of every DB related project, so that we can have a map and now where we are.

best
arturo

On Tue, Aug 4, 2015 at 11:25 AM, Guillermo Polito <[hidden email]> wrote:
OpenDBX -> it is a database abstraction written in C. It was developed outside the community, though we contribute if necessary. There is also a driver in Pharo providing bindings against this opendbx library.
Garage -> a database abstraction in Pharo. It provides a common interface to talk with several drivers (e.g., opendbx, native postgres, native mysql...)
Glorp -> an object relational mapping framework that connects to a database (therefore it needs some database driver) and converts relational data into objects.
DBXTalk -> the umbrella project of all the above (and their connections)

Reply | Threaded
Open this post in threaded view
|

Re: Garage, Glorp, PostgreSQL and Cursors

Tommaso DS
In reply to this post by Guillermo Polito


On 04/08/15 16:25, Guillermo Polito wrote:

> Helloooooo :)
>
> First Tomasso, it's nice to know it's working well for you :). We need
> feedback and contributions, even the smallests (like docs, or blogposts
> with your experiences or tutorials).
>
> Answering your question, GarageGlorp does not yet use the cursors from
> the Garage drivers. I do not know what exactly should be added to the
> Glorp's GarageAdapter, maybe we can take a look at it if you are available.
>

Hi Guillermo,

thanks, that would be really interesting.

Unfortunately right now I'm in a hurry to read some data from posgtgres,
so I'll manage to do that "by hand" (i.e. without cursors), but I think
that if we want to use Pharo for data analysis, being able to process
large volume of data in a lean way is a must (it is not the first time I
need it) and the option of glorp+cursors to stream the data from the db
without kill the ram would be amazing.

I would be happy to help enable this feature for Garage, but I don't
really know where to look, do you have any hints?

Tommaso


> For the rest:
>
> *OpenDBX* -> it is a database abstraction written in C. It was developed
> outside the community, though we contribute if necessary. There is also
> a driver in Pharo providing bindings against this opendbx library.
> *Garage* -> a database abstraction in Pharo. It provides a common
> interface to talk with several drivers (e.g., opendbx, native postgres,
> native mysql...)
> *Glorp* -> an object relational mapping framework that connects to a
> database (therefore it needs some database driver) and converts
> relational data into objects.
> *DBXTalk* -> the umbrella project of all the above (and their connections)
>
> Cheers,
> Guille
>
> El mar., 4 de ago. de 2015 a la(s) 12:36 a. m., Esteban A. Maringolo
> <[hidden email] <mailto:[hidden email]>> escribió:
>
>     2015-08-03 16:20 GMT-03:00 Esteban Lorenzano <[hidden email]
>     <mailto:[hidden email]>>:
>     > yes… just that is not accurate :)
>     > is basically like that, just that DBXTalk is not a driver, is an
>     umbrella project. The OpenDBX drivers is still called OpenDBXDriver :P
>
>     TooManyAbstractionsException signal.
>
>     ;)
>
>     Esteban A. Maringolo
>