Parsing SQL in a Pharo system

The PostgresV3 code in squeak uses subclasses of Compiler and Parser to make sure that
PG3SchemaMirror subclasses can parse plsql code. That looks like a nice way to work.

What do we need to do to do something similar with the Opal based infrastructure?
What subclasses do I need, what methods should I override?

Stephan

On 01 Oct 2014, at 14:30, Stephan Eggermont <[hidden email]> wrote:

> The PostgresV3 code in squeak uses subclasses of Compiler and Parser to make sure that
> PG3SchemaMirror subclasses can parse plsql code. That looks like a nice way to work.
>
> What do we need to do to do something similar with the Opal based infrastructure?
> What subclasses do I need, what methods should I override?
>

Normally it should be enough to replace all Parser and Compiler reference by

Smalltalk compilerClass

(as this gives you a Facade that implement still the old APIs of both parser and compiler).

        Marcus

2014-10-01 9:30 GMT-03:00 Stephan Eggermont <[hidden email]>:
> The PostgresV3 code in squeak uses subclasses of Compiler and Parser to make sure that
> PG3SchemaMirror subclasses can parse plsql code. That looks like a nice way to work.

Are you using/planning to use PostgresV3?

What are its advantages over current driver (PostgresV2)?

Regards,

Esteban A. Maringolo

Esteban wrote:
>Are you using/planning to use PostgresV3?

>What are its advantages over current driver (PostgresV2)?

I was puzzled that a smart developer like Levente
decides not to use Glorp. And I wondered why
SqueakDBX decided to use V2 instead of V3,
a protocol introduced in 2003 or so.

Stephan

2014-10-02 15:19 GMT-03:00 Stephan Eggermont <[hidden email]>:
> Esteban wrote:
>>Are you using/planning to use PostgresV3?
>
>>What are its advantages over current driver (PostgresV2)?
>
> I was puzzled that a smart developer like Levente
> decides not to use Glorp.

I don't want to sound harsh, but there is no VISIBLE interest from the
Pharo board/committee regarding ORM and/or RDBMS support other than a
CI task. Few days ago I asked a question in the Pharo-Business
regarding this, and got NO REPLY.

This week I was looking into Python's SQLAlchemy [1] to find how close
it is to GLORP current features, and also found out how far it
seems/feels to a small
community like ours. Not to mention things like jOOQ[2] ([1] provides
some of its features though).

I'm making no demands here, just giving my opinion about a "business
feature" (DBs) that I'd like better supported.

> And I wondered why SqueakDBX decided to use V2 instead of V3,
> a protocol introduced in 2003 or so.

Is the V2/V3 spec a PostgreSQL protocol spec or something made up in
the Squeak community?
I know something changed in PgSQL protocols around version 7.2.


Esteban A. Maringolo
[1] http://www.sqlalchemy.org/
[2] http://www.jooq.org/

Esteban,

On 02 Oct 2014, at 20:43, Esteban A. Maringolo <[hidden email]> wrote:

I understand your pain, you're looking for people that are in the same boat, apparently there aren't that many. But there certainly are some (I use(d) Glorp+PostgresV2 myself, I believe Mariano does too), but it seems nobody wants to take the lead to push this (even) further. I am not sure this is necessarily bad, RDBMS does not move that much, but it would obviously be better to have more users.

We are a small(er) community, it is what it is, but we are growing.

>> And I wondered why SqueakDBX decided to use V2 instead of V3,
>> a protocol introduced in 2003 or so.
>
> Is the V2/V3 spec a PostgreSQL protocol spec or something made up in
> the Squeak community?
> I know something changed in PgSQL protocols around version 7.2.

V2 and V3 are indeed two different wire level protocols for a DB client to talk to PostgreSQL. Obviously, V3 came after V2, it should be considered an improvement, but V2 remains supported. As far as I understood it, V3 is binary while V2 is text oriented, the former should be faster. I am sure there are feature lists comparing the two somewhere out there.

Of course, the implementation quality of the driver is important too to get good performance.

We'll see how far Stephan gets, I am interested. I really hope we can get the V3 code running in Pharo without too much compatibility issues.

> Esteban A. Maringolo
> [1] http://www.sqlalchemy.org/
> [2] http://www.jooq.org/

Sven

On 2/10/14 20:43, Esteban A. Maringolo wrote:
> I don't want to sound harsh, but there is no VISIBLE interest from the
> Pharo board/committee regarding ORM and/or RDBMS support other than a
> CI task.
Do you really mean that the board is a kind of jerks and idiots that are
lazy bastards?

If people that have interest in ORM do not help us how can make progress.
When did you send a fix to Pharo? Remember Pharo is Yours.
Esteban is working like a mad on making sure that Spur is coming to us.
May be we can
say to esteban to stop Spur and 64 bits after and some VM bug fix and
support for integrating better and ....
to focus on Glorp and DBTalk.

Now when as a community did we help the consortium, the board and the
community?
> Few days ago I asked a question in the Pharo-Business
> regarding this, and got NO REPLY.

I did not see it.
>
> This week I was looking into Python's SQLAlchemy [1] to find how close
> it is to GLORP current features, and also found out how far it
> seems/feels to a small
> community like ours. Not to mention things like jOOQ[2] ([1] provides
> some of its features though).
>
> I'm making no demands here, just giving my opinion about a "business
> feature" (DBs) that I'd like better supported.
This is on our roadmap. With the money of the consrortium I hope that we
will be able
to pay someone (I have an idea) to improve this support but there is no
magic
     - if people that needs it do not give an hand....
     - if people that needs it do not participate to the consortium

then it will be at our speed.
Because other Pharo clients like Thales wants multi windowing support,
we need
better Text, better events ....

May pythoners are cooler than Pharoers when it comes to helping write
code and documentation?

HI, 

Well, what you say sounds harsh... but is not too far from truth, sadly. What happens is that current manpower for every task is very limited (it's me)... and well, urgent stuff is always hunting us.

But esteban this is not true.
Pharo is not you sorry. Pharo is OURS.

We have the willing, and we have spent some time on it, but it will be a lot easier if there is also community effort put on improve things... every single bugfix, comment or whatever helps, but also more important efforts are needed. 

We would bless any effort/collaboration in that direction. I suppose we can also put bounties to fulfil certain tasks (that's how first DBXTalk suite was made). 

To have a description of what we need (a list of features, etc.) would be cool too. 

exactly!

I alwys thought that improving the infrastructure was magically done. This is not the case? Santa claus is a lie. How sad.
Stef

On Thu, Oct 02, 2014 at 03:43:05PM -0300, Esteban A. Maringolo wrote:
> Is the V2/V3 spec a PostgreSQL protocol spec or something made up in
> the Squeak community?

V2/V3 are PostgreSQL wire protocol versions. V3 supports parameter binding
and bulk copying, whereas V2 does not.

There is some additional info at this squeak-dev thread:

  http://forum.world.st/Status-of-PostgresV3-td4780110.html

Looking at the updated packages, there seems to be support in PostgresV3 (here
meaning the Smalltalk software) for parameter binding at the protocol level,
but the API does not provide any facility to use it.

Pierce

PostgresV3-Core/Pool looks perfectly usable.

There is
PG3Connection>>md5HashMessage: aString

        ^Smalltalk globals
                at: #MD5
                ifPresent: [ :md5 | (md5 hashMessage: aString) hex asLowercase ]
                ifAbsent: [ WebUtils md5Digest: aString ]

Which works in Pharo 4, though WebUtils doesn't exist anymore.
Replacing it is trivial

md5HashMessage: aString

        ^Smalltalk globals
                at: #MD5
                ifPresent: [ :md5 | (md5 hashMessage: aString) hex asLowercase ]
                ifAbsent: [ (Smalltalk globals at: #WebUtils) md5Digest: aString ]

Stephan

2014-10-03 2:47 GMT-03:00 stepharo <[hidden email]>:
> On 2/10/14 20:43, Esteban A. Maringolo wrote:
>> I don't want to sound harsh, but there is no VISIBLE interest from the
>> Pharo board/committee regarding ORM and/or RDBMS support other than a
>> CI task.

> Do you really mean that the board is a kind of jerks and idiots that are
> lazy bastards?

Please slow down, I never, ever, suggested even remotely that.
I said there is no VISIBLE interest. In the latest Consortium Report
it isn't even listed.
It's not a topic that attracts interest as Spec and other things does.

> If people that have interest in ORM do not help us how can make progress.
> When did you send a fix to Pharo? Remember Pharo is Yours.

I use it daily for my business, which on some aspects of its
technology stack like DB feels more like a bet than a safe decision.

I submitted a few commits to GLORP and PostgresV2 packages, does that count?

> Esteban is working like a mad on making sure that Spur is coming to us.
> May be we can say to esteban to stop Spur and 64 bits after and some
> VM bug fix and support for integrating better and ....
> to focus on Glorp and DBTalk.

I know, and I understand the priorities given the limited manpower.
That's why I mentioned I was making no demands, I was simply giving my opinion.

> Now when as a community did we help the consortium, the board and the
> community?
>>
>> Few days ago I asked a question in the Pharo-Business
>> regarding this, and got NO REPLY.
> I did not see it.

Sept 30, subject "Better ORM tools/alternatives"


>> I'm making no demands here, just giving my opinion about a "business
>> feature" (DBs) that I'd like better supported.
> This is on our roadmap.

Not in https://github.com/pharo-project/pharo-workingRoadmaps.

> With the money of the consrortium I hope that we will be able
> to pay someone (I have an idea) to improve this support but there is no
> magic
>     - if people that needs it do not give an hand....
>     - if people that needs it do not participate to the consortium
> then it will be at our speed.
> Because other Pharo clients like Thales wants multi windowing support, we
> need better Text, better events ....

Please don't excuse yourself or the consortium, there is no need, I
understand what it is to have a big backlog and a limited
manpower/money to tackle all of it in a short time.

> May pythoners are cooler than Pharoers when it comes to helping write code
> and documentation?

I have no idea, I don't participate in any Python community.

But please consider that accusing me (or anybody) of treating you as
jerks or suggesting that if I don't send fixes I shouldn't emit
opinions is totally discouraging.

I know you're passionate and you live&love Pharo, and hence you speak
that way, and I can understand that, but even if I understand who, how
and why you're saying something it somehow hurts the feeling of
belonging to the community (to which I feel part of).

Best regards!

2014-10-02 17:26 GMT-03:00 Sven Van Caekenberghe <[hidden email]>:
The quid here is who takes the lead, I said I could contribute and
code, but taking the lead on a project requires more than coding
skills.

The point that RDBMS is a stable thing a benefit in this case, and
that's why those who use GLORP today (like me), only got
bitten by a few bugs due to an outdated port.
And as long as we use PostgreSQL we'll be in the safer zone.

>> Is the V2/V3 spec a PostgreSQL protocol spec or something made up in
>> the Squeak community?
>> I know something changed in PgSQL protocols around version 7.2.
>
> V2 and V3 are indeed two different wire level protocols for a DB client to talk to PostgreSQL. Obviously, V3 came after V2, it should be considered an improvement, but V2 remains supported. As far as I understood it, V3 is binary while V2 is text oriented, the former should be faster. I am sure there are feature lists comparing the two somewhere out there.
> Of course, the implementation quality of the driver is important too to get good performance.

I haven't reached any limits of the V2 driver yet. If I have a
bottleneck it is down the DB or at the app level.

Thank you!

PostgresV3-CodeMirror is very interesting.
7 classes and less than 1K lines of code.

In the base class PG3SchemaMirror, the
compilerClass, parserClass, sourceCodeTemplate
and shoutParserClass are overridden to return
PG3-specific classes. That allows simple
integration with other languages, in this case
plsql.

In Pharo we don't have shoutParserClass
defined on Behavior. Where do I set
what highlighter is to be used for a method?

Stephan

On 3/10/14 14:32, Stephan Eggermont wrote: What was webUtils?
Because I would raise an error instead.

>> Do you really mean that the board is a kind of jerks and idiots that are
>> lazy bastards?
> Please slow down, I never, ever, suggested even remotely that.

I know but do not say that the board is not interested. We just issue
today a roadmap to inria and database is one of the items.

>> If people that have interest in ORM do not help us how can make progress.
>> When did you send a fix to Pharo? Remember Pharo is Yours.
> I use it daily for my business, which on some aspects of its
> technology stack like DB feels more like a bet than a safe decision.

???
>
> I submitted a few commits to GLORP and PostgresV2 packages, does that count?

Yes it does. This is important.
>> Esteban is working like a mad on making sure that Spur is coming to us.
>> May be we can say to esteban to stop Spur and 64 bits after and some
>> VM bug fix and support for integrating better and ....
>> to focus on Glorp and DBTalk.
> I know, and I understand the priorities given the limited manpower.
> That's why I mentioned I was making no demands, I was simply giving my opinion.

Ok

> Sept 30, subject "Better ORM tools/alternatives"

I will check. Too busy with something else.
>
>>> I'm making no demands here, just giving my opinion about a "business
>>> feature" (DBs) that I'd like better supported.
>> This is on our roadmap.
> Not in https://github.com/pharo-project/pharo-workingRoadmaps.
No these ones are just random notes about what we should not forget.
Like cleaning the huge mess in Fonts that produced mess in Roassal and
Moose.

>
>> May pythoners are cooler than Pharoers when it comes to helping write code
>> and documentation?
> I have no idea, I don't participate in any Python community.
>
> But please consider that accusing me (or anybody) of treating you as
> jerks or suggesting that if I don't send fixes I shouldn't emit
> opinions is totally discouraging.

OK I was in attacking mode.
>
> I know you're passionate and you live&love Pharo, and hence you speak
> that way, and I can understand that, but even if I understand who, how
> and why you're saying something it somehow hurts the feeling of
> belonging to the community (to which I feel part of).

Ok I will be nicer in the future.
But my point is that the best way to help you is to start on what is
important for you and try to make it work better.
We can offer jenkins servers, disc space and some help but we cannot do
everything.

Stef

Thanks for the information.
May be Olivier could help us there.


On 3/10/14 12:59, Pierce Ng wrote: