Smalltalk › Pharo › Pharo Smalltalk Users

Generate class hierarchy from JSON Schema

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

20 messages Options

SergeStinckwich

Generate class hierarchy from JSON Schema

Hi there,

anyone has experience about generating a class hierarchy from a JSON Schema ? Code available somewhere ?

Thank you.

Serge Stinckwic

Int. Research Unit

on Modelling/Simulation of Complex Systems (UMMISCO)

Sorbonne University

(SU)

French National Research Institute for Sustainable Development (IRD)

niversity of Yaoundé I, Cameroon

"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich

NorbertHartl

Re: Generate class hierarchy from JSON Schema

We have

https://github.com/zweidenker/JSONSchema

which is an implementation of basic stuff in JSON schema. But it does not compile classes it just creates a runtime model. To be honest I never understood why we make the effort to compile classes for parsed model. The runtime model works usually pretty well. The only reason I can see is that it is easier to store somewhere.

If you like our code and you miss things, we are eager to add the stuff we not had (yet) reason to do so.

Norbert

Am 18.01.2020 um 09:20 schrieb Serge Stinckwich <[hidden email]>:

Hi there,

anyone has experience about generating a class hierarchy from a JSON Schema ? Code available somewhere ?

Thank you.

--
Serge Stinckwic
h

Int. Research Unit
on Modelling/Simulation of Complex Systems (UMMISCO)
Sorbonne University
(SU)
French National Research Institute for Sustainable Development (IRD)
U
niversity of Yaoundé I, Cameroon
"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich

SergeStinckwich

Re: Generate class hierarchy from JSON Schema

Great work ! I will have a look.

I want to be able to generate JSON files following a JSON Schema with a high level API. So my idea was to generate automatically from JSON Schema the class hierarchy but a runtime model might be enough maybe.

Regards,

Sent from my iPhone

On 18 Jan 2020, at 09:28, Norbert Hartl <[hidden email]> wrote:

We have

https://github.com/zweidenker/JSONSchema

which is an implementation of basic stuff in JSON schema. But it does not compile classes it just creates a runtime model. To be honest I never understood why we make the effort to compile classes for parsed model. The runtime model works usually pretty well. The only reason I can see is that it is easier to store somewhere.

If you like our code and you miss things, we are eager to add the stuff we not had (yet) reason to do so.

Norbert

Am 18.01.2020 um 09:20 schrieb Serge Stinckwich <[hidden email]>:

Hi there,

anyone has experience about generating a class hierarchy from a JSON Schema ? Code available somewhere ?

Thank you.

--
Serge Stinckwic
h

Int. Research Unit
on Modelling/Simulation of Complex Systems (UMMISCO)
Sorbonne University
(SU)
French National Research Institute for Sustainable Development (IRD)
U
niversity of Yaoundé I, Cameroon
"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich

darth-cheney

Re: Generate class hierarchy from JSON Schema

Serge I would also be interested in seeing what you come up with.

On Sat, Jan 18, 2020 at 3:56 AM <[hidden email]> wrote:

Great work ! I will have a look.

I want to be able to generate JSON files following a JSON Schema with a high level API. So my idea was to generate automatically from JSON Schema the class hierarchy but a runtime model might be enough maybe.

Regards,

Sent from my iPhone

On 18 Jan 2020, at 09:28, Norbert Hartl <[hidden email]> wrote:

We have

https://github.com/zweidenker/JSONSchema

which is an implementation of basic stuff in JSON schema. But it does not compile classes it just creates a runtime model. To be honest I never understood why we make the effort to compile classes for parsed model. The runtime model works usually pretty well. The only reason I can see is that it is easier to store somewhere.

If you like our code and you miss things, we are eager to add the stuff we not had (yet) reason to do so.

Norbert

Am 18.01.2020 um 09:20 schrieb Serge Stinckwich <[hidden email]>:

Hi there,

anyone has experience about generating a class hierarchy from a JSON Schema ? Code available somewhere ?

Thank you.

--
Serge Stinckwic
h

Int. Research Unit
on Modelling/Simulation of Complex Systems (UMMISCO)
Sorbonne University
(SU)
French National Research Institute for Sustainable Development (IRD)
U
niversity of Yaoundé I, Cameroon
"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich

Eric

Pharo Smalltalk Users mailing list

Re: Generate class hierarchy from JSON Schema

In reply to this post by SergeStinckwich

I have some work around building class hierarchies from database schema - specifically a Rails style Active Record schema.

It is a bit of a work in progress and is intended to be folded into Glorp but you can see where I am at https://github.com/pharo-db/glorp/blob/13-Drive-DescriptorSystem-creation-entirely-from-database-schema-and-naming-conventions/Glorp/ActiveRecordDescriptorSystem.class.st#L108

Maybe it helps?

On Jan 18, 2020, at 12:20 AM, Serge Stinckwich <[hidden email]> wrote:

Hi there,

anyone has experience about generating a class hierarchy from a JSON Schema ? Code available somewhere ?

Thank you.

--
Serge Stinckwic
h

Int. Research Unit
on Modelling/Simulation of Complex Systems (UMMISCO)
Sorbonne University
(SU)
French National Research Institute for Sustainable Development (IRD)
U
niversity of Yaoundé I, Cameroon
"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich

eftomi

Re: Generate class hierarchy from JSON Schema

Todd, this is great, I didn't know that you are working on ActiveRecord.

BTW, I have some minor additions to SQLServerPlatform (support for National
Language Character Set) and OraclePlatform (Decimal data type) - is there a
central git repository for Glorp so that I could PR to?

Thanks and best wishes,
Tomaz

--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Pharo Smalltalk Users mailing list

Re: Generate class hierarchy from JSON Schema

The official is https://github.com/pharo-rdbms/glorp (The one I forked at pharo-db - a username I need to change as it is causing confusion).

I've been mostly working with P3 and SQLite together. I've standardized the query results format for fetching field definitions so if you want to add support for another db, please note the format of the result set and structure yours appropriately.

I would very much like all the databases to be hosted under pharo-rdbms and all of them to have glorp adaptors and the additional meta structure queries I'm adding to the GlorpPlatform classes to enable ActiveRecord. I feel like Pharo's RDBMS story has been lagging and the lack of ready to go database support keeps people from building apps on it.

There are a zillion little business CRUD apps that could be knocked out with spec2 or Seaside.

Please, dive in and make improvements.

> On Jan 20, 2020, at 9:15 AM, eftomi <[hidden email]> wrote:
>
> Todd, this is great, I didn't know that you are working on ActiveRecord.
>
> BTW, I have some minor additions to SQLServerPlatform (support for National
> Language Character Set) and OraclePlatform (Decimal data type) - is there a
> central git repository for Glorp so that I could PR to?
>
> Thanks and best wishes,
> Tomaz
>
>
>
> --
> Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
>

SergeStinckwich

Re: Generate class hierarchy from JSON Schema

In reply to this post by SergeStinckwich

I had a look Norbert, but some features are missing like support for anyOf or multiple types like: 

{
    "type": ["string", "null"]
}

I need some time to understand how to add them.

On Sat, Jan 18, 2020 at 9:55 AM <[hidden email]> wrote:

Great work ! I will have a look.

I want to be able to generate JSON files following a JSON Schema with a high level API. So my idea was to generate automatically from JSON Schema the class hierarchy but a runtime model might be enough maybe.

Regards,

Sent from my iPhone

On 18 Jan 2020, at 09:28, Norbert Hartl <[hidden email]> wrote:

We have

https://github.com/zweidenker/JSONSchema

which is an implementation of basic stuff in JSON schema. But it does not compile classes it just creates a runtime model. To be honest I never understood why we make the effort to compile classes for parsed model. The runtime model works usually pretty well. The only reason I can see is that it is easier to store somewhere.

If you like our code and you miss things, we are eager to add the stuff we not had (yet) reason to do so.

Norbert

Am 18.01.2020 um 09:20 schrieb Serge Stinckwich <[hidden email]>:

Hi there,

anyone has experience about generating a class hierarchy from a JSON Schema ? Code available somewhere ?

Thank you.

--
Serge Stinckwic
h

Int. Research Unit
on Modelling/Simulation of Complex Systems (UMMISCO)
Sorbonne University
(SU)
French National Research Institute for Sustainable Development (IRD)
U
niversity of Yaoundé I, Cameroon
"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich

Serge Stinckwic

Int. Research Unit

on Modelling/Simulation of Complex Systems (UMMISCO)

Sorbonne University

(SU)

French National Research Institute for Sustainable Development (IRD)

niversity of Yaoundé I, Cameroon

"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich

NorbertHartl

Re: Generate class hierarchy from JSON Schema

Serge,

as I mentioned at first you need to complain. I meant it that if you miss stuff I'm eager to add that. Of course it would be lovely if you do it yourself. Just saying that if there is need I can look into. Don't forget open issues on the git repo and add examples what you want to parse or submit a PR with a red test that would be best.

Norbert

Am 20.01.2020 um 19:34 schrieb Serge Stinckwich <[hidden email]>:

I had a look Norbert, but some features are missing like support for anyOf or multiple types like:
{
"type": ["string", "null"]
}

I need some time to understand how to add them.

On Sat, Jan 18, 2020 at 9:55 AM <[hidden email]> wrote:
Great work ! I will have a look.

I want to be able to generate JSON files following a JSON Schema with a high level API. So my idea was to generate automatically from JSON Schema the class hierarchy but a runtime model might be enough maybe.

Regards,

Sent from my iPhone

On 18 Jan 2020, at 09:28, Norbert Hartl <[hidden email]> wrote:

We have

https://github.com/zweidenker/JSONSchema

which is an implementation of basic stuff in JSON schema. But it does not compile classes it just creates a runtime model. To be honest I never understood why we make the effort to compile classes for parsed model. The runtime model works usually pretty well. The only reason I can see is that it is easier to store somewhere.

If you like our code and you miss things, we are eager to add the stuff we not had (yet) reason to do so.

Norbert

Am 18.01.2020 um 09:20 schrieb Serge Stinckwich <[hidden email]>:

Hi there,

anyone has experience about generating a class hierarchy from a JSON Schema ? Code available somewhere ?

Thank you.

--
Serge Stinckwic
h

Int. Research Unit
on Modelling/Simulation of Complex Systems (UMMISCO)
Sorbonne University
(SU)
French National Research Institute for Sustainable Development (IRD)
U
niversity of Yaoundé I, Cameroon
"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich

--
Serge Stinckwic
h

Int. Research Unit
on Modelling/Simulation of Complex Systems (UMMISCO)
Sorbonne University
(SU)
French National Research Institute for Sustainable Development (IRD)
U
niversity of Yaoundé I, Cameroon
"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich

eftomi

Re: Generate class hierarchy from JSON Schema

In reply to this post by eftomi

Thanks, Todd - I'll prepare a PR for https://github.com/pharo-rdbms/glorp to update SQLServerPlatform and OraclePlatform.

I already used the standardized query results format in PharoADO which works nicely with Glorp. You can find PharoADO here: https://github.com/eftomi/pharo-ado. Maybe it can be included into the pharo-rdbms collection if you feel that it's mature enough. PharoADO can be used standalone (as interface to ActiveX Data Objects) or with Glorp.

However, there are some "high level" tests in Glorp package that are still red with Oracle and SQLServer, you can find the list on GitHub under "issues". In particular, I didn't have time yet to implement blobs - PharoADO uses PharoCOM (by Pablo) and firstly we have to settle the Variant SafeArray data type in that package. Now that we have stable Pharo 8.0 64 bit for Windows, PharoCOM should be checked for 64-bits, too.

Anyway, I agree with you that the database support is one of the key success factors for a development platform. I'm focusing into this since I hope that one day I could teach the concepts of business solutions development fully in Pharo :-)

Best wishes,

Tomaz

The official is https://github.com/pharo-rdbms/glorp (The one I forked at pharo-db - a username I need to change as it is causing confusion).

I've been mostly working with P3 and SQLite together. I've standardized the query results format for fetching field definitions so if you want to add support for another db, please note the format of the result set and structure yours appropriately.

I would very much like all the databases to be hosted under pharo-rdbms and all of them to have glorp adaptors and the additional meta structure queries I'm adding to the GlorpPlatform classes to enable ActiveRecord. I feel like Pharo's RDBMS story has been lagging and the lack of ready to go database support keeps people from building apps on it.

There are a zillion little business CRUD apps that could be knocked out with spec2 or Seaside.

Please, dive in and make improvements.

Pharo Smalltalk Users mailing list

Re: Generate class hierarchy from JSON Schema

That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle). I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions. I just need to get many to many with link tables done and it is good to go. I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream

" Format:

-------------------------+-------------------+--------+----------+-----------------------+----

id | character varying | 255 | 0 | ''::character varying | 1

Then I can build a whole class model and descriptor system off the schema.

On Jan 22, 2020, at 9:55 AM, Tomaž Turk <[hidden email]> wrote:

Thanks, Todd - I'll prepare a PR for https://github.com/pharo-rdbms/glorp to update SQLServerPlatform and OraclePlatform.

I already used the standardized query results format in PharoADO which works nicely with Glorp. You can find PharoADO here: https://github.com/eftomi/pharo-ado. Maybe it can be included into the pharo-rdbms collection if you feel that it's mature enough. PharoADO can be used standalone (as interface to ActiveX Data Objects) or with Glorp.

However, there are some "high level" tests in Glorp package that are still red with Oracle and SQLServer, you can find the list on GitHub under "issues". In particular, I didn't have time yet to implement blobs - PharoADO uses PharoCOM (by Pablo) and firstly we have to settle the Variant SafeArray data type in that package. Now that we have stable Pharo 8.0 64 bit for Windows, PharoCOM should be checked for 64-bits, too.

Anyway, I agree with you that the database support is one of the key success factors for a development platform. I'm focusing into this since I hope that one day I could teach the concepts of business solutions development fully in Pharo :-)

Best wishes,
Tomaz

The official is https://github.com/pharo-rdbms/glorp (The one I forked at pharo-db - a username I need to change as it is causing confusion).

I've been mostly working with P3 and SQLite together. I've standardized the query results format for fetching field definitions so if you want to add support for another db, please note the format of the result set and structure yours appropriately.

I would very much like all the databases to be hosted under pharo-rdbms and all of them to have glorp adaptors and the additional meta structure queries I'm adding to the GlorpPlatform classes to enable ActiveRecord. I feel like Pharo's RDBMS story has been lagging and the lack of ready to go database support keeps people from building apps on it.

There are a zillion little business CRUD apps that could be knocked out with spec2 or Seaside.

Please, dive in and make improvements.

eftomi

Re: Generate class hierarchy from JSON Schema

In reply to this post by eftomi

That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle). I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions. I just need to get many to many with link tables done and it is good to go. I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream
" Format:
name | type | length | nullable | default_value | pk
-------------------------+-------------------+--------+----------+-----------------------+----
id | character varying | 255 | 0 | ''::character varying | 1

This is great news about ActiveRecord!

ADO is, unfortunately, a Windows library, so PharoADO is limited to Windows. Maybe as a next step we should focus into Garage and the addition of SQL Server and Oracle support. If I only had time ...

Stéphane, thanks for the support.

Best wishes,

Tomaz

Esteban A. Maringolo

Re: Generate class hierarchy from JSON Schema

Hi,

To feed a Pharo app I need to read from an old MDB file, via ODBC.

So I tested PharoADO and, after adding a missing Variant Type [1], I got my query working (at least read only).

However the time of ADOClient to return a query with 10K rows of 5 columns each is extremely slow (+65000ms) compared, e.g., with Dolphin Smalltalk via ODBC (62ms).

The time of the execute call to return is instantaneous, so I think the issue might be in the creation of the result set.

Is this a known issue?

Should I check something else?

Thanks in advance,

[1] https://github.com/tesonep/pharo-com/pull/10

Esteban A. Maringolo

On Fri, Jan 24, 2020 at 5:59 AM Tomaž Turk <[hidden email]> wrote:

That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle). I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions. I just need to get many to many with link tables done and it is good to go. I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream
" Format:
name | type | length | nullable | default_value | pk
-------------------------+-------------------+--------+----------+-----------------------+----
id | character varying | 255 | 0 | ''::character varying | 1

This is great news about ActiveRecord!

ADO is, unfortunately, a Windows library, so PharoADO is limited to Windows. Maybe as a next step we should focus into Garage and the addition of SQL Server and Oracle support. If I only had time ...

Stéphane, thanks for the support.

Best wishes,
Tomaz

Esteban A. Maringolo

Re: Generate class hierarchy from JSON Schema

As an additional reference, I attach the profile tally for the query I'm mentioning.

The culprit seems to be the calculation of the ADOField properties that might be calculated on every call.

Regards,

Esteban A. Maringolo

On Wed, Mar 11, 2020 at 2:13 PM Esteban Maringolo <[hidden email]> wrote:

Hi,

To feed a Pharo app I need to read from an old MDB file, via ODBC.

So I tested PharoADO and, after adding a missing Variant Type [1], I got my query working (at least read only).

However the time of ADOClient to return a query with 10K rows of 5 columns each is extremely slow (+65000ms) compared, e.g., with Dolphin Smalltalk via ODBC (62ms).

The time of the execute call to return is instantaneous, so I think the issue might be in the creation of the result set.

Is this a known issue?

Should I check something else?

Thanks in advance,

[1] https://github.com/tesonep/pharo-com/pull/10
Esteban A. Maringolo

On Fri, Jan 24, 2020 at 5:59 AM Tomaž Turk <[hidden email]> wrote:

That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle). I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions. I just need to get many to many with link tables done and it is good to go. I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream
" Format:
name | type | length | nullable | default_value | pk
-------------------------+-------------------+--------+----------+-----------------------+----
id | character varying | 255 | 0 | ''::character varying | 1

This is great news about ActiveRecord!

ADO is, unfortunately, a Windows library, so PharoADO is limited to Windows. Maybe as a next step we should focus into Garage and the addition of SQL Server and Oracle support. If I only had time ...

Stéphane, thanks for the support.

Best wishes,
Tomaz

ADOClient-Tally.txt (29K) Download Attachment

eftomi

Re: Generate class hierarchy from JSON Schema

Hi Esteban,

Thanks for reporting this issue. I haven't made any "stress" tests yet for ADOClient. Could you please try to use ADORecordset as described in the readme? There you have a direct access to each record at a time, and then you can create your own "loop" to process the query result. In this way you can avoid the loop in ADOClient>>query: method.

Best wishes,

Tomaz

------ Original Message ------

From: "Esteban Maringolo" <[hidden email]>

To: "Tomaž Turk" <[hidden email]>; "Any question about pharo is welcome" <[hidden email]>

Sent: 11. 03. 2020 18:25:51

Subject: Re: [Pharo-users] Generate class hierarchy from JSON Schema

As an additional reference, I attach the profile tally for the query I'm mentioning.

The culprit seems to be the calculation of the ADOField properties that might be calculated on every call.

Regards,

Esteban A. Maringolo

On Wed, Mar 11, 2020 at 2:13 PM Esteban Maringolo <[hidden email]> wrote:
Hi,

To feed a Pharo app I need to read from an old MDB file, via ODBC.

So I tested PharoADO and, after adding a missing Variant Type [1], I got my query working (at least read only).

However the time of ADOClient to return a query with 10K rows of 5 columns each is extremely slow (+65000ms) compared, e.g., with Dolphin Smalltalk via ODBC (62ms).

The time of the execute call to return is instantaneous, so I think the issue might be in the creation of the result set.

Is this a known issue?

Should I check something else?

Thanks in advance,

[1] https://github.com/tesonep/pharo-com/pull/10
Esteban A. Maringolo

On Fri, Jan 24, 2020 at 5:59 AM Tomaž Turk <[hidden email]> wrote:

That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle). I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions. I just need to get many to many with link tables done and it is good to go. I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream
" Format:
name | type | length | nullable | default_value | pk
-------------------------+-------------------+--------+----------+-----------------------+----
id | character varying | 255 | 0 | ''::character varying | 1

This is great news about ActiveRecord!

ADO is, unfortunately, a Windows library, so PharoADO is limited to Windows. Maybe as a next step we should focus into Garage and the addition of SQL Server and Oracle support. If I only had time ...

Stéphane, thanks for the support.

Best wishes,
Tomaz

Esteban A. Maringolo

Re: Generate class hierarchy from JSON Schema

The profiling using the recordset was even worst. I changed the iteration [1] to add everything to a collection, but most of the time is spent in #calculateMethods and #calculateProperties. Which is performed everytime for each ADOField.

I wouldn't consider this a stress test, but maybe the methods and properties should be cached, or a different class of dispatcher should be used where the methods and properties are defined ahead of time instead of being discovered when first invoked.

Regards,

[1] "Modified code"

rst := ADORecordset createInstance .
rst open: 'SELECT * FROM PLAYER' activeConnection: conn cursorType: 2 lockType: 3 options: -1.
results := OrderedCollection new.
[rst eof] whileFalse: [
| row |
row := Array new: rst fields count.
1 to: row size do: [ :idx | row at: idx put: (rst fields item: idx) value ].
results add: row.
rst moveNext
].

Esteban A. Maringolo

On Wed, Mar 11, 2020 at 3:02 PM Tomaž Turk <[hidden email]> wrote:

Hi Esteban,

Thanks for reporting this issue. I haven't made any "stress" tests yet for ADOClient. Could you please try to use ADORecordset as described in the readme? There you have a direct access to each record at a time, and then you can create your own "loop" to process the query result. In this way you can avoid the loop in ADOClient>>query: method.

Best wishes,
Tomaz

------ Original Message ------

From: "Esteban Maringolo" <[hidden email]>

To: "Tomaž Turk" <[hidden email]>; "Any question about pharo is welcome" <[hidden email]>

Sent: 11. 03. 2020 18:25:51

Subject: Re: [Pharo-users] Generate class hierarchy from JSON Schema

As an additional reference, I attach the profile tally for the query I'm mentioning.

The culprit seems to be the calculation of the ADOField properties that might be calculated on every call.

Regards,

Esteban A. Maringolo

On Wed, Mar 11, 2020 at 2:13 PM Esteban Maringolo <[hidden email]> wrote:
Hi,

To feed a Pharo app I need to read from an old MDB file, via ODBC.

So I tested PharoADO and, after adding a missing Variant Type [1], I got my query working (at least read only).

However the time of ADOClient to return a query with 10K rows of 5 columns each is extremely slow (+65000ms) compared, e.g., with Dolphin Smalltalk via ODBC (62ms).

The time of the execute call to return is instantaneous, so I think the issue might be in the creation of the result set.

Is this a known issue?

Should I check something else?

Thanks in advance,

[1] https://github.com/tesonep/pharo-com/pull/10
Esteban A. Maringolo

On Fri, Jan 24, 2020 at 5:59 AM Tomaž Turk <[hidden email]> wrote:

That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle). I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions. I just need to get many to many with link tables done and it is good to go. I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream
" Format:
name | type | length | nullable | default_value | pk
-------------------------+-------------------+--------+----------+-----------------------+----
id | character varying | 255 | 0 | ''::character varying | 1

This is great news about ActiveRecord!

ADO is, unfortunately, a Windows library, so PharoADO is limited to Windows. Maybe as a next step we should focus into Garage and the addition of SQL Server and Oracle support. If I only had time ...

Stéphane, thanks for the support.

Best wishes,
Tomaz

ADORecordset-tally.txt (30K) Download Attachment

Esteban A. Maringolo

Re: Generate class hierarchy from JSON Schema

I did a small change in ADORecordset to have the fields as an instVar and also in ADOFields to have each ADOField in an `items` instVar.

It is still slow compared to an ODBC API, but I got a 20x speedup (~1000ms), that is spent mostly on the actual call to the dispatcher to fetch each cell value.

I look forward for an ODBC API, but at least to read any other data source, this seems to work pretty well.

I already sent a pull-request.

Esteban A. Maringolo

On Wed, Mar 11, 2020 at 3:31 PM Esteban Maringolo <[hidden email]> wrote:

The profiling using the recordset was even worst. I changed the iteration [1] to add everything to a collection, but most of the time is spent in #calculateMethods and #calculateProperties. Which is performed everytime for each ADOField.

I wouldn't consider this a stress test, but maybe the methods and properties should be cached, or a different class of dispatcher should be used where the methods and properties are defined ahead of time instead of being discovered when first invoked.

Regards,

[1] "Modified code"
rst := ADORecordset createInstance .
rst open: 'SELECT * FROM PLAYER' activeConnection: conn cursorType: 2 lockType: 3 options: -1.
results := OrderedCollection new.
[rst eof] whileFalse: [
| row |
row := Array new: rst fields count.
1 to: row size do: [ :idx | row at: idx put: (rst fields item: idx) value ].
results add: row.
rst moveNext
].

Esteban A. Maringolo

On Wed, Mar 11, 2020 at 3:02 PM Tomaž Turk <[hidden email]> wrote:
Hi Esteban,

Thanks for reporting this issue. I haven't made any "stress" tests yet for ADOClient. Could you please try to use ADORecordset as described in the readme? There you have a direct access to each record at a time, and then you can create your own "loop" to process the query result. In this way you can avoid the loop in ADOClient>>query: method.

Best wishes,
Tomaz

------ Original Message ------

From: "Esteban Maringolo" <[hidden email]>

To: "Tomaž Turk" <[hidden email]>; "Any question about pharo is welcome" <[hidden email]>

Sent: 11. 03. 2020 18:25:51

Subject: Re: [Pharo-users] Generate class hierarchy from JSON Schema

As an additional reference, I attach the profile tally for the query I'm mentioning.

The culprit seems to be the calculation of the ADOField properties that might be calculated on every call.

Regards,

Esteban A. Maringolo

On Wed, Mar 11, 2020 at 2:13 PM Esteban Maringolo <[hidden email]> wrote:
Hi,

To feed a Pharo app I need to read from an old MDB file, via ODBC.

So I tested PharoADO and, after adding a missing Variant Type [1], I got my query working (at least read only).

However the time of ADOClient to return a query with 10K rows of 5 columns each is extremely slow (+65000ms) compared, e.g., with Dolphin Smalltalk via ODBC (62ms).

The time of the execute call to return is instantaneous, so I think the issue might be in the creation of the result set.

Is this a known issue?

Should I check something else?

Thanks in advance,

[1] https://github.com/tesonep/pharo-com/pull/10
Esteban A. Maringolo

On Fri, Jan 24, 2020 at 5:59 AM Tomaž Turk <[hidden email]> wrote:

That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle). I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions. I just need to get many to many with link tables done and it is good to go. I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream
" Format:
name | type | length | nullable | default_value | pk
-------------------------+-------------------+--------+----------+-----------------------+----
id | character varying | 255 | 0 | ''::character varying | 1

This is great news about ActiveRecord!

ADO is, unfortunately, a Windows library, so PharoADO is limited to Windows. Maybe as a next step we should focus into Garage and the addition of SQL Server and Oracle support. If I only had time ...

Stéphane, thanks for the support.

Best wishes,
Tomaz

Esteban A. Maringolo

Re: Generate class hierarchy from JSON Schema

Follow up.

Seems that playing with these COM/ADO objects in the playground messes with the VM; so I'm getting weird image crashes, where the image closes without any notice.

Do you experience the same?

Esteban A. Maringolo

On Wed, Mar 11, 2020 at 4:11 PM Esteban Maringolo <[hidden email]> wrote:

I did a small change in ADORecordset to have the fields as an instVar and also in ADOFields to have each ADOField in an `items` instVar.
It is still slow compared to an ODBC API, but I got a 20x speedup (~1000ms), that is spent mostly on the actual call to the dispatcher to fetch each cell value.

I look forward for an ODBC API, but at least to read any other data source, this seems to work pretty well.

I already sent a pull-request.

Esteban A. Maringolo

On Wed, Mar 11, 2020 at 3:31 PM Esteban Maringolo <[hidden email]> wrote:

The profiling using the recordset was even worst. I changed the iteration [1] to add everything to a collection, but most of the time is spent in #calculateMethods and #calculateProperties. Which is performed everytime for each ADOField.

I wouldn't consider this a stress test, but maybe the methods and properties should be cached, or a different class of dispatcher should be used where the methods and properties are defined ahead of time instead of being discovered when first invoked.

Regards,

[1] "Modified code"
rst := ADORecordset createInstance .
rst open: 'SELECT * FROM PLAYER' activeConnection: conn cursorType: 2 lockType: 3 options: -1.
results := OrderedCollection new.
[rst eof] whileFalse: [
| row |
row := Array new: rst fields count.
1 to: row size do: [ :idx | row at: idx put: (rst fields item: idx) value ].
results add: row.
rst moveNext
].

Esteban A. Maringolo

On Wed, Mar 11, 2020 at 3:02 PM Tomaž Turk <[hidden email]> wrote:
Hi Esteban,

Thanks for reporting this issue. I haven't made any "stress" tests yet for ADOClient. Could you please try to use ADORecordset as described in the readme? There you have a direct access to each record at a time, and then you can create your own "loop" to process the query result. In this way you can avoid the loop in ADOClient>>query: method.

Best wishes,
Tomaz

------ Original Message ------

From: "Esteban Maringolo" <[hidden email]>

To: "Tomaž Turk" <[hidden email]>; "Any question about pharo is welcome" <[hidden email]>

Sent: 11. 03. 2020 18:25:51

Subject: Re: [Pharo-users] Generate class hierarchy from JSON Schema

As an additional reference, I attach the profile tally for the query I'm mentioning.

The culprit seems to be the calculation of the ADOField properties that might be calculated on every call.

Regards,

Esteban A. Maringolo

On Wed, Mar 11, 2020 at 2:13 PM Esteban Maringolo <[hidden email]> wrote:
Hi,

To feed a Pharo app I need to read from an old MDB file, via ODBC.

So I tested PharoADO and, after adding a missing Variant Type [1], I got my query working (at least read only).

However the time of ADOClient to return a query with 10K rows of 5 columns each is extremely slow (+65000ms) compared, e.g., with Dolphin Smalltalk via ODBC (62ms).

The time of the execute call to return is instantaneous, so I think the issue might be in the creation of the result set.

Is this a known issue?

Should I check something else?

Thanks in advance,

[1] https://github.com/tesonep/pharo-com/pull/10
Esteban A. Maringolo

On Fri, Jan 24, 2020 at 5:59 AM Tomaž Turk <[hidden email]> wrote:

That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle). I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions. I just need to get many to many with link tables done and it is good to go. I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream
" Format:
name | type | length | nullable | default_value | pk
-------------------------+-------------------+--------+----------+-----------------------+----
id | character varying | 255 | 0 | ''::character varying | 1

This is great news about ActiveRecord!

ADO is, unfortunately, a Windows library, so PharoADO is limited to Windows. Maybe as a next step we should focus into Garage and the addition of SQL Server and Oracle support. If I only had time ...

Stéphane, thanks for the support.

Best wishes,
Tomaz

eftomi

Re: Generate class hierarchy from JSON Schema

Thanks for the PR! Maybe something can be done also on the PharoCOM level within calculateMethods and calculateProperties.

I didn't experience any crashes, although I developed PharoADO on 32bits in Pharo 7 and Pablo upgraded PharoCOM to 64bit after Pharo 8 was released. Besides, I haven't managed yet to re-check if all the COM instances are properly released after usage.

Best wishes

Tomaz

------ Original Message ------

From: "Esteban Maringolo" <[hidden email]>

To: "Tomaž Turk" <[hidden email]>

Cc: "Any question about pharo is welcome" <[hidden email]>

Sent: 11. 03. 2020 20:35:10

Subject: Re: Re[2]: [Pharo-users] Generate class hierarchy from JSON Schema

Follow up.

Seems that playing with these COM/ADO objects in the playground messes with the VM; so I'm getting weird image crashes, where the image closes without any notice.

Do you experience the same?

Esteban A. Maringolo

On Wed, Mar 11, 2020 at 4:11 PM Esteban Maringolo <[hidden email]> wrote:
I did a small change in ADORecordset to have the fields as an instVar and also in ADOFields to have each ADOField in an `items` instVar.
It is still slow compared to an ODBC API, but I got a 20x speedup (~1000ms), that is spent mostly on the actual call to the dispatcher to fetch each cell value.

I look forward for an ODBC API, but at least to read any other data source, this seems to work pretty well.

I already sent a pull-request.

Esteban A. Maringolo

On Wed, Mar 11, 2020 at 3:31 PM Esteban Maringolo <[hidden email]> wrote:

The profiling using the recordset was even worst. I changed the iteration [1] to add everything to a collection, but most of the time is spent in #calculateMethods and #calculateProperties. Which is performed everytime for each ADOField.

I wouldn't consider this a stress test, but maybe the methods and properties should be cached, or a different class of dispatcher should be used where the methods and properties are defined ahead of time instead of being discovered when first invoked.

Regards,

[1] "Modified code"
rst := ADORecordset createInstance .
rst open: 'SELECT * FROM PLAYER' activeConnection: conn cursorType: 2 lockType: 3 options: -1.
results := OrderedCollection new.
[rst eof] whileFalse: [
| row |
row := Array new: rst fields count.
1 to: row size do: [ :idx | row at: idx put: (rst fields item: idx) value ].
results add: row.
rst moveNext
].

Esteban A. Maringolo

On Wed, Mar 11, 2020 at 3:02 PM Tomaž Turk <[hidden email]> wrote:
Hi Esteban,

Thanks for reporting this issue. I haven't made any "stress" tests yet for ADOClient. Could you please try to use ADORecordset as described in the readme? There you have a direct access to each record at a time, and then you can create your own "loop" to process the query result. In this way you can avoid the loop in ADOClient>>query: method.

Best wishes,
Tomaz

------ Original Message ------

From: "Esteban Maringolo" <[hidden email]>

To: "Tomaž Turk" <[hidden email]>; "Any question about pharo is welcome" <[hidden email]>

Sent: 11. 03. 2020 18:25:51

Subject: Re: [Pharo-users] Generate class hierarchy from JSON Schema

As an additional reference, I attach the profile tally for the query I'm mentioning.

The culprit seems to be the calculation of the ADOField properties that might be calculated on every call.

Regards,

Esteban A. Maringolo

On Wed, Mar 11, 2020 at 2:13 PM Esteban Maringolo <[hidden email]> wrote:
Hi,

To feed a Pharo app I need to read from an old MDB file, via ODBC.

So I tested PharoADO and, after adding a missing Variant Type [1], I got my query working (at least read only).

However the time of ADOClient to return a query with 10K rows of 5 columns each is extremely slow (+65000ms) compared, e.g., with Dolphin Smalltalk via ODBC (62ms).

The time of the execute call to return is instantaneous, so I think the issue might be in the creation of the result set.

Is this a known issue?

Should I check something else?

Thanks in advance,

[1] https://github.com/tesonep/pharo-com/pull/10
Esteban A. Maringolo

On Fri, Jan 24, 2020 at 5:59 AM Tomaž Turk <[hidden email]> wrote:

That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle). I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions. I just need to get many to many with link tables done and it is good to go. I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream
" Format:
name | type | length | nullable | default_value | pk
-------------------------+-------------------+--------+----------+-----------------------+----
id | character varying | 255 | 0 | ''::character varying | 1

This is great news about ActiveRecord!

ADO is, unfortunately, a Windows library, so PharoADO is limited to Windows. Maybe as a next step we should focus into Garage and the addition of SQL Server and Oracle support. If I only had time ...

Stéphane, thanks for the support.

Best wishes,
Tomaz

eftomi

Re: Generate class hierarchy from JSON Schema

Esteban, if time permits, could you please describe what situations lead to
crashes, and with which images?

Thanks,
Tomaz

--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html