Generate class hierarchy from JSON Schema

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Generate class hierarchy from JSON Schema

SergeStinckwich
Hi there,

anyone has experience about generating a class hierarchy from a JSON Schema ? Code available somewhere ?

Thank you.

--
Serge Stinckwic
h

Int. Research Unit
 on Modelling/Simulation of Complex Systems (UMMISCO)
Sorbonne University
 (SU)
French National Research Institute for Sustainable Development (IRD)
U
niversity of Yaoundé I, Cameroon
"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich
Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

NorbertHartl
We have 


which is an implementation of basic stuff in JSON schema. But it does not compile classes it just creates a runtime model. To be honest I never understood why we make the effort to compile classes for parsed model. The runtime model works usually pretty well. The only reason I can see is that it is easier to store somewhere. 

If you like our code and you miss things, we are eager to add the stuff we not had (yet) reason to do so.

Norbert

Am 18.01.2020 um 09:20 schrieb Serge Stinckwich <[hidden email]>:

Hi there,

anyone has experience about generating a class hierarchy from a JSON Schema ? Code available somewhere ?

Thank you.

--
Serge Stinckwic
h

Int. Research Unit
 on Modelling/Simulation of Complex Systems (UMMISCO)
Sorbonne University
 (SU)
French National Research Institute for Sustainable Development (IRD)
U
niversity of Yaoundé I, Cameroon
"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich

Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

SergeStinckwich
Great work ! I will have a look.

I want to be able to generate JSON files following a JSON Schema with a high level API. So my idea was to generate automatically from JSON Schema the class hierarchy but a runtime model might be enough maybe.

Regards,

Sent from my iPhone

On 18 Jan 2020, at 09:28, Norbert Hartl <[hidden email]> wrote:

We have 


which is an implementation of basic stuff in JSON schema. But it does not compile classes it just creates a runtime model. To be honest I never understood why we make the effort to compile classes for parsed model. The runtime model works usually pretty well. The only reason I can see is that it is easier to store somewhere. 

If you like our code and you miss things, we are eager to add the stuff we not had (yet) reason to do so.

Norbert

Am 18.01.2020 um 09:20 schrieb Serge Stinckwich <[hidden email]>:

Hi there,

anyone has experience about generating a class hierarchy from a JSON Schema ? Code available somewhere ?

Thank you.

--
Serge Stinckwic
h

Int. Research Unit
 on Modelling/Simulation of Complex Systems (UMMISCO)
Sorbonne University
 (SU)
French National Research Institute for Sustainable Development (IRD)
U
niversity of Yaoundé I, Cameroon
"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich

Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

darth-cheney
Serge I would also be interested in seeing what you come up with.

On Sat, Jan 18, 2020 at 3:56 AM <[hidden email]> wrote:
Great work ! I will have a look.

I want to be able to generate JSON files following a JSON Schema with a high level API. So my idea was to generate automatically from JSON Schema the class hierarchy but a runtime model might be enough maybe.

Regards,

Sent from my iPhone

On 18 Jan 2020, at 09:28, Norbert Hartl <[hidden email]> wrote:

We have 


which is an implementation of basic stuff in JSON schema. But it does not compile classes it just creates a runtime model. To be honest I never understood why we make the effort to compile classes for parsed model. The runtime model works usually pretty well. The only reason I can see is that it is easier to store somewhere. 

If you like our code and you miss things, we are eager to add the stuff we not had (yet) reason to do so.

Norbert

Am 18.01.2020 um 09:20 schrieb Serge Stinckwich <[hidden email]>:

Hi there,

anyone has experience about generating a class hierarchy from a JSON Schema ? Code available somewhere ?

Thank you.

--
Serge Stinckwic
h

Int. Research Unit
 on Modelling/Simulation of Complex Systems (UMMISCO)
Sorbonne University
 (SU)
French National Research Institute for Sustainable Development (IRD)
U
niversity of Yaoundé I, Cameroon
"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich



--
Eric
Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

Pharo Smalltalk Users mailing list
In reply to this post by SergeStinckwich
I have some work around building class hierarchies from database schema - specifically a Rails style Active Record schema.


Maybe it helps?

On Jan 18, 2020, at 12:20 AM, Serge Stinckwich <[hidden email]> wrote:

Hi there,

anyone has experience about generating a class hierarchy from a JSON Schema ? Code available somewhere ?

Thank you.

--
Serge Stinckwic
h

Int. Research Unit
 on Modelling/Simulation of Complex Systems (UMMISCO)
Sorbonne University
 (SU)
French National Research Institute for Sustainable Development (IRD)
U
niversity of Yaoundé I, Cameroon
"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich

Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

eftomi
Todd, this is great, I didn't know that you are working on ActiveRecord.

BTW, I have some minor additions to SQLServerPlatform (support for National
Language Character Set) and OraclePlatform (Decimal data type) - is there a
central git repository for Glorp so that I could PR to?

Thanks and best wishes,
Tomaz  



--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

Pharo Smalltalk Users mailing list
The official is https://github.com/pharo-rdbms/glorp (The one I forked at pharo-db - a username I need to change as it is causing confusion).

I've been mostly working with P3 and SQLite together.  I've standardized the query results format for fetching field definitions so if you want to add support for another db, please note the format of the result set and structure yours appropriately.

I would very much like all the databases to be hosted under pharo-rdbms and all of them to have glorp adaptors and the additional meta structure queries I'm adding to the GlorpPlatform classes to enable ActiveRecord.  I feel like Pharo's RDBMS story has been lagging and the lack of ready to go database support keeps people from building apps on it.  

There are a zillion little business CRUD apps that could be knocked out with spec2 or Seaside.

Please, dive in and make improvements.

> On Jan 20, 2020, at 9:15 AM, eftomi <[hidden email]> wrote:
>
> Todd, this is great, I didn't know that you are working on ActiveRecord.
>
> BTW, I have some minor additions to SQLServerPlatform (support for National
> Language Character Set) and OraclePlatform (Decimal data type) - is there a
> central git repository for Glorp so that I could PR to?
>
> Thanks and best wishes,
> Tomaz  
>
>
>
> --
> Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
>


Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

SergeStinckwich
In reply to this post by SergeStinckwich
I had a look Norbert, but some features are missing like support for anyOf or multiple types like:
{
    "type": ["string", "null"]
}

I need some time to understand how to add them.


On Sat, Jan 18, 2020 at 9:55 AM <[hidden email]> wrote:
Great work ! I will have a look.

I want to be able to generate JSON files following a JSON Schema with a high level API. So my idea was to generate automatically from JSON Schema the class hierarchy but a runtime model might be enough maybe.

Regards,

Sent from my iPhone

On 18 Jan 2020, at 09:28, Norbert Hartl <[hidden email]> wrote:

We have 


which is an implementation of basic stuff in JSON schema. But it does not compile classes it just creates a runtime model. To be honest I never understood why we make the effort to compile classes for parsed model. The runtime model works usually pretty well. The only reason I can see is that it is easier to store somewhere. 

If you like our code and you miss things, we are eager to add the stuff we not had (yet) reason to do so.

Norbert

Am 18.01.2020 um 09:20 schrieb Serge Stinckwich <[hidden email]>:

Hi there,

anyone has experience about generating a class hierarchy from a JSON Schema ? Code available somewhere ?

Thank you.

--
Serge Stinckwic
h

Int. Research Unit
 on Modelling/Simulation of Complex Systems (UMMISCO)
Sorbonne University
 (SU)
French National Research Institute for Sustainable Development (IRD)
U
niversity of Yaoundé I, Cameroon
"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich



--
Serge Stinckwic
​h​

Int. Research Unit
 on Modelling/Simulation of Complex Systems (UMMISCO)
​Sorbonne University
 (SU)
French National Research Institute for Sustainable Development (IRD)​
U
​niversity of Yaoundé I​, Cameroon
"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich
Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

NorbertHartl
Serge,

as I mentioned at first you need to complain. I meant it that if you miss stuff I'm eager to add that. Of course it would be lovely if you do it yourself. Just saying that if there is need I can look into. Don't forget open issues on the git repo and add examples what you want to parse or submit a PR with a red test that would be best.

Norbert


Am 20.01.2020 um 19:34 schrieb Serge Stinckwich <[hidden email]>:

I had a look Norbert, but some features are missing like support for anyOf or multiple types like:
{
    "type": ["string", "null"]
}

I need some time to understand how to add them.


On Sat, Jan 18, 2020 at 9:55 AM <[hidden email]> wrote:
Great work ! I will have a look.

I want to be able to generate JSON files following a JSON Schema with a high level API. So my idea was to generate automatically from JSON Schema the class hierarchy but a runtime model might be enough maybe.

Regards,

Sent from my iPhone

On 18 Jan 2020, at 09:28, Norbert Hartl <[hidden email]> wrote:

We have 


which is an implementation of basic stuff in JSON schema. But it does not compile classes it just creates a runtime model. To be honest I never understood why we make the effort to compile classes for parsed model. The runtime model works usually pretty well. The only reason I can see is that it is easier to store somewhere. 

If you like our code and you miss things, we are eager to add the stuff we not had (yet) reason to do so.

Norbert

Am 18.01.2020 um 09:20 schrieb Serge Stinckwich <[hidden email]>:

Hi there,

anyone has experience about generating a class hierarchy from a JSON Schema ? Code available somewhere ?

Thank you.

--
Serge Stinckwic
h

Int. Research Unit
 on Modelling/Simulation of Complex Systems (UMMISCO)
Sorbonne University
 (SU)
French National Research Institute for Sustainable Development (IRD)
U
niversity of Yaoundé I, Cameroon
"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich



--
Serge Stinckwic
​h​

Int. Research Unit
 on Modelling/Simulation of Complex Systems (UMMISCO)
​Sorbonne University
 (SU)
French National Research Institute for Sustainable Development (IRD)​
U
​niversity of Yaoundé I​, Cameroon
"Programs must be written for people to read, and only incidentally for machines to execute."
https://twitter.com/SergeStinckwich

Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

eftomi
In reply to this post by eftomi
Thanks, Todd - I'll prepare a PR for https://github.com/pharo-rdbms/glorp to update SQLServerPlatform and OraclePlatform.

I already used the standardized query results format in PharoADO which works nicely with Glorp. You can find PharoADO here: https://github.com/eftomi/pharo-ado. Maybe it can be included into the pharo-rdbms collection if you feel that it's mature enough. PharoADO can be used standalone (as interface to ActiveX Data Objects) or with Glorp.

However, there are some "high level" tests in Glorp package that are still red with Oracle and SQLServer, you can find the list on GitHub under "issues". In particular, I didn't have time yet to implement blobs - PharoADO uses PharoCOM (by Pablo) and firstly we have to settle the Variant SafeArray data type in that package. Now that we have stable Pharo 8.0 64 bit for Windows, PharoCOM should be checked for 64-bits, too. 

Anyway, I agree with you that the database support is one of the key success factors for a development platform. I'm focusing into this since I hope that one day I could teach the concepts of business solutions development fully in Pharo :-)

Best wishes,
Tomaz



The official is https://github.com/pharo-rdbms/glorp (The one I forked at pharo-db - a username I need to change as it is causing confusion).
 
I've been mostly working with P3 and SQLite together. I've standardized the query results format for fetching field definitions so if you want to add support for another db, please note the format of the result set and structure yours appropriately.
 
I would very much like all the databases to be hosted under pharo-rdbms and all of them to have glorp adaptors and the additional meta structure queries I'm adding to the GlorpPlatform classes to enable ActiveRecord. I feel like Pharo's RDBMS story has been lagging and the lack of ready to go database support keeps people from building apps on it.
 
There are a zillion little business CRUD apps that could be knocked out with spec2 or Seaside.
 
Please, dive in and make improvements.
 
Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

Pharo Smalltalk Users mailing list
That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle).  I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions.  I just need to get many to many with link tables done and it is good to go.  I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream
" Format:
      name           |       type        | length | nullable |     default_value     | pk 
-------------------------+-------------------+--------+----------+-----------------------+----
 id                      | character varying |    255 |        0 | ''::character varying |  1 

Then I can build a whole class model and descriptor system off the schema.

On Jan 22, 2020, at 9:55 AM, Tomaž Turk <[hidden email]> wrote:

Thanks, Todd - I'll prepare a PR for https://github.com/pharo-rdbms/glorp to update SQLServerPlatform and OraclePlatform.

I already used the standardized query results format in PharoADO which works nicely with Glorp. You can find PharoADO here: https://github.com/eftomi/pharo-ado. Maybe it can be included into the pharo-rdbms collection if you feel that it's mature enough. PharoADO can be used standalone (as interface to ActiveX Data Objects) or with Glorp.

However, there are some "high level" tests in Glorp package that are still red with Oracle and SQLServer, you can find the list on GitHub under "issues". In particular, I didn't have time yet to implement blobs - PharoADO uses PharoCOM (by Pablo) and firstly we have to settle the Variant SafeArray data type in that package. Now that we have stable Pharo 8.0 64 bit for Windows, PharoCOM should be checked for 64-bits, too. 

Anyway, I agree with you that the database support is one of the key success factors for a development platform. I'm focusing into this since I hope that one day I could teach the concepts of business solutions development fully in Pharo :-)

Best wishes,
Tomaz



The official is https://github.com/pharo-rdbms/glorp (The one I forked at pharo-db - a username I need to change as it is causing confusion).
 
I've been mostly working with P3 and SQLite together. I've standardized the query results format for fetching field definitions so if you want to add support for another db, please note the format of the result set and structure yours appropriately.
 
I would very much like all the databases to be hosted under pharo-rdbms and all of them to have glorp adaptors and the additional meta structure queries I'm adding to the GlorpPlatform classes to enable ActiveRecord. I feel like Pharo's RDBMS story has been lagging and the lack of ready to go database support keeps people from building apps on it.
 
There are a zillion little business CRUD apps that could be knocked out with spec2 or Seaside.
 
Please, dive in and make improvements.

Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

eftomi
In reply to this post by eftomi

That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle).  I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions.  I just need to get many to many with link tables done and it is good to go.  I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream
" Format:
      name           |       type        | length | nullable |     default_value     | pk 
-------------------------+-------------------+--------+----------+-----------------------+----
 id                      | character varying |    255 |        0 | ''::character varying |  1 


This is great news about ActiveRecord! 

ADO is, unfortunately, a Windows library, so PharoADO is limited to Windows. Maybe as a next step we should focus into Garage and the addition of SQL Server and Oracle support. If I only had time ...

Stéphane, thanks for the support. 

Best wishes,
Tomaz






Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

Esteban A. Maringolo
Hi,

To feed a Pharo app I need to read from an old MDB file, via ODBC.

So I tested PharoADO and, after adding a missing Variant Type [1], I got my query working (at least read only).

However the time of ADOClient to return a query with 10K rows of 5 columns each is extremely slow (+65000ms) compared, e.g., with Dolphin Smalltalk via ODBC (62ms).

The time of the execute call to return is instantaneous, so I think the issue might be in the creation of the result set.

Is this a known issue?

Should I check something else?

Thanks in advance, 


[1]  https://github.com/tesonep/pharo-com/pull/10  
Esteban A. Maringolo


On Fri, Jan 24, 2020 at 5:59 AM Tomaž Turk <[hidden email]> wrote:

That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle).  I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions.  I just need to get many to many with link tables done and it is good to go.  I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream
" Format:
      name           |       type        | length | nullable |     default_value     | pk 
-------------------------+-------------------+--------+----------+-----------------------+----
 id                      | character varying |    255 |        0 | ''::character varying |  1 


This is great news about ActiveRecord! 

ADO is, unfortunately, a Windows library, so PharoADO is limited to Windows. Maybe as a next step we should focus into Garage and the addition of SQL Server and Oracle support. If I only had time ...

Stéphane, thanks for the support. 

Best wishes,
Tomaz






Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

Esteban A. Maringolo
As an additional reference, I attach the profile tally  for the query I'm mentioning.

The culprit seems to be the calculation of the ADOField properties that might be calculated on every call.

Regards,

Esteban A. Maringolo


On Wed, Mar 11, 2020 at 2:13 PM Esteban Maringolo <[hidden email]> wrote:
Hi,

To feed a Pharo app I need to read from an old MDB file, via ODBC.

So I tested PharoADO and, after adding a missing Variant Type [1], I got my query working (at least read only).

However the time of ADOClient to return a query with 10K rows of 5 columns each is extremely slow (+65000ms) compared, e.g., with Dolphin Smalltalk via ODBC (62ms).

The time of the execute call to return is instantaneous, so I think the issue might be in the creation of the result set.

Is this a known issue?

Should I check something else?

Thanks in advance, 


[1]  https://github.com/tesonep/pharo-com/pull/10  
Esteban A. Maringolo


On Fri, Jan 24, 2020 at 5:59 AM Tomaž Turk <[hidden email]> wrote:

That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle).  I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions.  I just need to get many to many with link tables done and it is good to go.  I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream
" Format:
      name           |       type        | length | nullable |     default_value     | pk 
-------------------------+-------------------+--------+----------+-----------------------+----
 id                      | character varying |    255 |        0 | ''::character varying |  1 


This is great news about ActiveRecord! 

ADO is, unfortunately, a Windows library, so PharoADO is limited to Windows. Maybe as a next step we should focus into Garage and the addition of SQL Server and Oracle support. If I only had time ...

Stéphane, thanks for the support. 

Best wishes,
Tomaz







ADOClient-Tally.txt (29K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

eftomi
Hi Esteban,

Thanks for reporting this issue. I haven't made any "stress" tests yet for ADOClient. Could you please try to use ADORecordset as described in the readme? There you have a direct access to each record at a time, and then you can create your own "loop" to process the query result. In this way you can avoid the loop in ADOClient>>query: method.

Best wishes,
Tomaz

------ Original Message ------
From: "Esteban Maringolo" <[hidden email]>
To: "Tomaž Turk" <[hidden email]>; "Any question about pharo is welcome" <[hidden email]>
Sent: 11. 03. 2020 18:25:51
Subject: Re: [Pharo-users] Generate class hierarchy from JSON Schema

As an additional reference, I attach the profile tally  for the query I'm mentioning.

The culprit seems to be the calculation of the ADOField properties that might be calculated on every call.

Regards,

Esteban A. Maringolo


On Wed, Mar 11, 2020 at 2:13 PM Esteban Maringolo <[hidden email]> wrote:
Hi,

To feed a Pharo app I need to read from an old MDB file, via ODBC.

So I tested PharoADO and, after adding a missing Variant Type [1], I got my query working (at least read only).

However the time of ADOClient to return a query with 10K rows of 5 columns each is extremely slow (+65000ms) compared, e.g., with Dolphin Smalltalk via ODBC (62ms).

The time of the execute call to return is instantaneous, so I think the issue might be in the creation of the result set.

Is this a known issue?

Should I check something else?

Thanks in advance, 


[1]  https://github.com/tesonep/pharo-com/pull/10  
Esteban A. Maringolo


On Fri, Jan 24, 2020 at 5:59 AM Tomaž Turk <[hidden email]> wrote:

That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle).  I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions.  I just need to get many to many with link tables done and it is good to go.  I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream
" Format:
      name           |       type        | length | nullable |     default_value     | pk 
-------------------------+-------------------+--------+----------+-----------------------+----
 id                      | character varying |    255 |        0 | ''::character varying |  1 


This is great news about ActiveRecord! 

ADO is, unfortunately, a Windows library, so PharoADO is limited to Windows. Maybe as a next step we should focus into Garage and the addition of SQL Server and Oracle support. If I only had time ...

Stéphane, thanks for the support. 

Best wishes,
Tomaz






Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

Esteban A. Maringolo

The profiling using the recordset was even worst. I changed the iteration [1] to add everything to a collection, but most of the time is spent in #calculateMethods and #calculateProperties. Which is performed everytime for each ADOField.

I wouldn't consider this a stress test, but maybe the methods and properties should be cached, or a different class of dispatcher should be used where the methods and properties are defined ahead of time instead of being discovered when first invoked.

Regards,


[1] "Modified code"
rst := ADORecordset createInstance .
rst open: 'SELECT * FROM PLAYER' activeConnection: conn cursorType: 2 lockType: 3 options: -1.
results := OrderedCollection new.
[rst eof] whileFalse: [
  | row |
  row := Array new: rst fields count.
  1 to: row size do: [ :idx | row at: idx put: (rst fields item: idx) value ].
  results add: row.
  rst moveNext
 ].



Esteban A. Maringolo


On Wed, Mar 11, 2020 at 3:02 PM Tomaž Turk <[hidden email]> wrote:
Hi Esteban,

Thanks for reporting this issue. I haven't made any "stress" tests yet for ADOClient. Could you please try to use ADORecordset as described in the readme? There you have a direct access to each record at a time, and then you can create your own "loop" to process the query result. In this way you can avoid the loop in ADOClient>>query: method.

Best wishes,
Tomaz

------ Original Message ------
From: "Esteban Maringolo" <[hidden email]>
To: "Tomaž Turk" <[hidden email]>; "Any question about pharo is welcome" <[hidden email]>
Sent: 11. 03. 2020 18:25:51
Subject: Re: [Pharo-users] Generate class hierarchy from JSON Schema

As an additional reference, I attach the profile tally  for the query I'm mentioning.

The culprit seems to be the calculation of the ADOField properties that might be calculated on every call.

Regards,

Esteban A. Maringolo


On Wed, Mar 11, 2020 at 2:13 PM Esteban Maringolo <[hidden email]> wrote:
Hi,

To feed a Pharo app I need to read from an old MDB file, via ODBC.

So I tested PharoADO and, after adding a missing Variant Type [1], I got my query working (at least read only).

However the time of ADOClient to return a query with 10K rows of 5 columns each is extremely slow (+65000ms) compared, e.g., with Dolphin Smalltalk via ODBC (62ms).

The time of the execute call to return is instantaneous, so I think the issue might be in the creation of the result set.

Is this a known issue?

Should I check something else?

Thanks in advance, 


[1]  https://github.com/tesonep/pharo-com/pull/10  
Esteban A. Maringolo


On Fri, Jan 24, 2020 at 5:59 AM Tomaž Turk <[hidden email]> wrote:

That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle).  I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions.  I just need to get many to many with link tables done and it is good to go.  I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream
" Format:
      name           |       type        | length | nullable |     default_value     | pk 
-------------------------+-------------------+--------+----------+-----------------------+----
 id                      | character varying |    255 |        0 | ''::character varying |  1 


This is great news about ActiveRecord! 

ADO is, unfortunately, a Windows library, so PharoADO is limited to Windows. Maybe as a next step we should focus into Garage and the addition of SQL Server and Oracle support. If I only had time ...

Stéphane, thanks for the support. 

Best wishes,
Tomaz







ADORecordset-tally.txt (30K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

Esteban A. Maringolo
I did a small change in ADORecordset to have the fields as an instVar and also in ADOFields to have each ADOField in an `items` instVar. 
It is still slow compared to an ODBC API, but I got a 20x speedup (~1000ms), that is spent mostly on the actual call to the dispatcher to fetch each cell value.

I look forward for an ODBC API, but at least to read any other data source, this seems to work pretty well.

I already sent a pull-request.

Esteban A. Maringolo


On Wed, Mar 11, 2020 at 3:31 PM Esteban Maringolo <[hidden email]> wrote:

The profiling using the recordset was even worst. I changed the iteration [1] to add everything to a collection, but most of the time is spent in #calculateMethods and #calculateProperties. Which is performed everytime for each ADOField.

I wouldn't consider this a stress test, but maybe the methods and properties should be cached, or a different class of dispatcher should be used where the methods and properties are defined ahead of time instead of being discovered when first invoked.

Regards,


[1] "Modified code"
rst := ADORecordset createInstance .
rst open: 'SELECT * FROM PLAYER' activeConnection: conn cursorType: 2 lockType: 3 options: -1.
results := OrderedCollection new.
[rst eof] whileFalse: [
  | row |
  row := Array new: rst fields count.
  1 to: row size do: [ :idx | row at: idx put: (rst fields item: idx) value ].
  results add: row.
  rst moveNext
 ].



Esteban A. Maringolo


On Wed, Mar 11, 2020 at 3:02 PM Tomaž Turk <[hidden email]> wrote:
Hi Esteban,

Thanks for reporting this issue. I haven't made any "stress" tests yet for ADOClient. Could you please try to use ADORecordset as described in the readme? There you have a direct access to each record at a time, and then you can create your own "loop" to process the query result. In this way you can avoid the loop in ADOClient>>query: method.

Best wishes,
Tomaz

------ Original Message ------
From: "Esteban Maringolo" <[hidden email]>
To: "Tomaž Turk" <[hidden email]>; "Any question about pharo is welcome" <[hidden email]>
Sent: 11. 03. 2020 18:25:51
Subject: Re: [Pharo-users] Generate class hierarchy from JSON Schema

As an additional reference, I attach the profile tally  for the query I'm mentioning.

The culprit seems to be the calculation of the ADOField properties that might be calculated on every call.

Regards,

Esteban A. Maringolo


On Wed, Mar 11, 2020 at 2:13 PM Esteban Maringolo <[hidden email]> wrote:
Hi,

To feed a Pharo app I need to read from an old MDB file, via ODBC.

So I tested PharoADO and, after adding a missing Variant Type [1], I got my query working (at least read only).

However the time of ADOClient to return a query with 10K rows of 5 columns each is extremely slow (+65000ms) compared, e.g., with Dolphin Smalltalk via ODBC (62ms).

The time of the execute call to return is instantaneous, so I think the issue might be in the creation of the result set.

Is this a known issue?

Should I check something else?

Thanks in advance, 


[1]  https://github.com/tesonep/pharo-com/pull/10  
Esteban A. Maringolo


On Fri, Jan 24, 2020 at 5:59 AM Tomaž Turk <[hidden email]> wrote:

That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle).  I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions.  I just need to get many to many with link tables done and it is good to go.  I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream
" Format:
      name           |       type        | length | nullable |     default_value     | pk 
-------------------------+-------------------+--------+----------+-----------------------+----
 id                      | character varying |    255 |        0 | ''::character varying |  1 


This is great news about ActiveRecord! 

ADO is, unfortunately, a Windows library, so PharoADO is limited to Windows. Maybe as a next step we should focus into Garage and the addition of SQL Server and Oracle support. If I only had time ...

Stéphane, thanks for the support. 

Best wishes,
Tomaz






Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

Esteban A. Maringolo
Follow up.

Seems that playing with these COM/ADO objects in the playground messes with the VM; so I'm getting weird image crashes, where the image closes without any notice. 

Do you experience the same?


Esteban A. Maringolo


On Wed, Mar 11, 2020 at 4:11 PM Esteban Maringolo <[hidden email]> wrote:
I did a small change in ADORecordset to have the fields as an instVar and also in ADOFields to have each ADOField in an `items` instVar. 
It is still slow compared to an ODBC API, but I got a 20x speedup (~1000ms), that is spent mostly on the actual call to the dispatcher to fetch each cell value.

I look forward for an ODBC API, but at least to read any other data source, this seems to work pretty well.

I already sent a pull-request.

Esteban A. Maringolo


On Wed, Mar 11, 2020 at 3:31 PM Esteban Maringolo <[hidden email]> wrote:

The profiling using the recordset was even worst. I changed the iteration [1] to add everything to a collection, but most of the time is spent in #calculateMethods and #calculateProperties. Which is performed everytime for each ADOField.

I wouldn't consider this a stress test, but maybe the methods and properties should be cached, or a different class of dispatcher should be used where the methods and properties are defined ahead of time instead of being discovered when first invoked.

Regards,


[1] "Modified code"
rst := ADORecordset createInstance .
rst open: 'SELECT * FROM PLAYER' activeConnection: conn cursorType: 2 lockType: 3 options: -1.
results := OrderedCollection new.
[rst eof] whileFalse: [
  | row |
  row := Array new: rst fields count.
  1 to: row size do: [ :idx | row at: idx put: (rst fields item: idx) value ].
  results add: row.
  rst moveNext
 ].



Esteban A. Maringolo


On Wed, Mar 11, 2020 at 3:02 PM Tomaž Turk <[hidden email]> wrote:
Hi Esteban,

Thanks for reporting this issue. I haven't made any "stress" tests yet for ADOClient. Could you please try to use ADORecordset as described in the readme? There you have a direct access to each record at a time, and then you can create your own "loop" to process the query result. In this way you can avoid the loop in ADOClient>>query: method.

Best wishes,
Tomaz

------ Original Message ------
From: "Esteban Maringolo" <[hidden email]>
To: "Tomaž Turk" <[hidden email]>; "Any question about pharo is welcome" <[hidden email]>
Sent: 11. 03. 2020 18:25:51
Subject: Re: [Pharo-users] Generate class hierarchy from JSON Schema

As an additional reference, I attach the profile tally  for the query I'm mentioning.

The culprit seems to be the calculation of the ADOField properties that might be calculated on every call.

Regards,

Esteban A. Maringolo


On Wed, Mar 11, 2020 at 2:13 PM Esteban Maringolo <[hidden email]> wrote:
Hi,

To feed a Pharo app I need to read from an old MDB file, via ODBC.

So I tested PharoADO and, after adding a missing Variant Type [1], I got my query working (at least read only).

However the time of ADOClient to return a query with 10K rows of 5 columns each is extremely slow (+65000ms) compared, e.g., with Dolphin Smalltalk via ODBC (62ms).

The time of the execute call to return is instantaneous, so I think the issue might be in the creation of the result set.

Is this a known issue?

Should I check something else?

Thanks in advance, 


[1]  https://github.com/tesonep/pharo-com/pull/10  
Esteban A. Maringolo


On Fri, Jan 24, 2020 at 5:59 AM Tomaž Turk <[hidden email]> wrote:

That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle).  I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions.  I just need to get many to many with link tables done and it is good to go.  I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream
" Format:
      name           |       type        | length | nullable |     default_value     | pk 
-------------------------+-------------------+--------+----------+-----------------------+----
 id                      | character varying |    255 |        0 | ''::character varying |  1 


This is great news about ActiveRecord! 

ADO is, unfortunately, a Windows library, so PharoADO is limited to Windows. Maybe as a next step we should focus into Garage and the addition of SQL Server and Oracle support. If I only had time ...

Stéphane, thanks for the support. 

Best wishes,
Tomaz






Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

eftomi
Thanks for the PR! Maybe something can be done also on the PharoCOM level within calculateMethods and calculateProperties.

I didn't experience any crashes, although I developed PharoADO on 32bits in Pharo 7 and Pablo upgraded PharoCOM to 64bit after Pharo 8 was released. Besides, I haven't managed yet to re-check if all the COM instances are properly released after usage.

Best wishes
Tomaz

------ Original Message ------
From: "Esteban Maringolo" <[hidden email]>
To: "Tomaž Turk" <[hidden email]>
Cc: "Any question about pharo is welcome" <[hidden email]>
Sent: 11. 03. 2020 20:35:10
Subject: Re: Re[2]: [Pharo-users] Generate class hierarchy from JSON Schema

Follow up.

Seems that playing with these COM/ADO objects in the playground messes with the VM; so I'm getting weird image crashes, where the image closes without any notice. 

Do you experience the same?


Esteban A. Maringolo


On Wed, Mar 11, 2020 at 4:11 PM Esteban Maringolo <[hidden email]> wrote:
I did a small change in ADORecordset to have the fields as an instVar and also in ADOFields to have each ADOField in an `items` instVar. 
It is still slow compared to an ODBC API, but I got a 20x speedup (~1000ms), that is spent mostly on the actual call to the dispatcher to fetch each cell value.

I look forward for an ODBC API, but at least to read any other data source, this seems to work pretty well.

I already sent a pull-request.

Esteban A. Maringolo


On Wed, Mar 11, 2020 at 3:31 PM Esteban Maringolo <[hidden email]> wrote:

The profiling using the recordset was even worst. I changed the iteration [1] to add everything to a collection, but most of the time is spent in #calculateMethods and #calculateProperties. Which is performed everytime for each ADOField.

I wouldn't consider this a stress test, but maybe the methods and properties should be cached, or a different class of dispatcher should be used where the methods and properties are defined ahead of time instead of being discovered when first invoked.

Regards,


[1] "Modified code"
rst := ADORecordset createInstance .
rst open: 'SELECT * FROM PLAYER' activeConnection: conn cursorType: 2 lockType: 3 options: -1.
results := OrderedCollection new.
[rst eof] whileFalse: [
  | row |
  row := Array new: rst fields count.
  1 to: row size do: [ :idx | row at: idx put: (rst fields item: idx) value ].
  results add: row.
  rst moveNext
 ].



Esteban A. Maringolo


On Wed, Mar 11, 2020 at 3:02 PM Tomaž Turk <[hidden email]> wrote:
Hi Esteban,

Thanks for reporting this issue. I haven't made any "stress" tests yet for ADOClient. Could you please try to use ADORecordset as described in the readme? There you have a direct access to each record at a time, and then you can create your own "loop" to process the query result. In this way you can avoid the loop in ADOClient>>query: method.

Best wishes,
Tomaz

------ Original Message ------
From: "Esteban Maringolo" <[hidden email]>
To: "Tomaž Turk" <[hidden email]>; "Any question about pharo is welcome" <[hidden email]>
Sent: 11. 03. 2020 18:25:51
Subject: Re: [Pharo-users] Generate class hierarchy from JSON Schema

As an additional reference, I attach the profile tally  for the query I'm mentioning.

The culprit seems to be the calculation of the ADOField properties that might be calculated on every call.

Regards,

Esteban A. Maringolo


On Wed, Mar 11, 2020 at 2:13 PM Esteban Maringolo <[hidden email]> wrote:
Hi,

To feed a Pharo app I need to read from an old MDB file, via ODBC.

So I tested PharoADO and, after adding a missing Variant Type [1], I got my query working (at least read only).

However the time of ADOClient to return a query with 10K rows of 5 columns each is extremely slow (+65000ms) compared, e.g., with Dolphin Smalltalk via ODBC (62ms).

The time of the execute call to return is instantaneous, so I think the issue might be in the creation of the result set.

Is this a known issue?

Should I check something else?

Thanks in advance, 


[1]  https://github.com/tesonep/pharo-com/pull/10  
Esteban A. Maringolo


On Fri, Jan 24, 2020 at 5:59 AM Tomaž Turk <[hidden email]> wrote:

That would be amazing!

I'm a Mac/Unix guy so I don't have access to the other platforms (I suppose I could fire up an AWS Oracle).  I can do mysql/mariadb, posgresql, and sqlite.

I'm pretty close to pushing my ActiveRecord extensions.  I just need to get many to many with link tables done and it is good to go.  I spent a few days "porting" the ruby inflector and its tests.

As for database introspection, I am relying on this new method and the result set format on DatabasePlatform.

printSqlStatementToListColumnsInTable: aDatabaseTable inSchema: schemaString on: aStream
" Format:
      name           |       type        | length | nullable |     default_value     | pk 
-------------------------+-------------------+--------+----------+-----------------------+----
 id                      | character varying |    255 |        0 | ''::character varying |  1 


This is great news about ActiveRecord! 

ADO is, unfortunately, a Windows library, so PharoADO is limited to Windows. Maybe as a next step we should focus into Garage and the addition of SQL Server and Oracle support. If I only had time ...

Stéphane, thanks for the support. 

Best wishes,
Tomaz






Reply | Threaded
Open this post in threaded view
|

Re: Generate class hierarchy from JSON Schema

eftomi
Esteban, if time permits, could you please describe what situations lead to
crashes, and with which images?

Thanks,
Tomaz



--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html