How to avoid image freezing while querying?

On Mon, Mar 11, 2013 at 7:41 AM, Víctor Casado <[hidden email]> wrote:

Hola Mariano, Diogenes & company!

First of all thank you for developing such a great tool like DBXTalk, which I'm using right now in a project and it works great! The only problem is, while executing a query, the whole image freezes until it finishes. Is it normal? Is there some way to avoid this behaviour?

Hi Victor. 

Thanks for the nice words. The problem is that the current FFI plugin is blocking FFI...there is an experimental VM with multi-threaded FFI. Still, it is not working properly. You should wait until that. In the meanwhile.... which database, driver, and OS are you using? because most of them support async queries. OpenDBX tries to do async queries when the database client supports them. And from OpenDBXDriver what we do when the backend supports async queries is to do a lot of "small blocking". So we do a kind of busy waiting from the image side, but that allows other process to continue executing. 

For this, see the method #processNextResultSet: aConnection querySettings: aQuerySettings

This line 

(code = OpenDBX resultTimeout) ifTrue: [ (Delay forMilliseconds: (aQuerySettings timeout asMiliseconds)) wait  

allows (the #wait) other Smalltalk process to run....

but of course, all that is only if the backend supports async.

Finally, can these timeouts be improved? yes!!  but we never have time to implemented...

(I copy paste and idea from a collegue)

You could
a) Use a default timeout for the first call which means it actually completes more queries on the first try yet still returns fast, say 100ms rather than 1ms.
(For later calls just to check if it is possibly finished, you probably want to block for as short a time as possible though)
b) Use an exponentially growing value for the Delay rather than a constant one, starting at 1ms and max some other value
1 2 4 8 16 32 64 128 256 512 1024 for instance, polling once per second shouldn't hurt other processes at all, yet give ok responsiveness for queries > 1 seconds.

This way, you (in the cases where potential is 9k queries /sec) will have a hard cap at 10k queries (due to the 100ms block time), and hurt those above that as little as possible using Delays. What you don't have though, is a cap of around 1k, due to calls never completing in 1ms, and having to wait (at least, I don't know the default value of aQuerySettings timeout) 1ms for each due to minimum delay wait time resolution.

Btw, unless the microseconds and seconds are switched, this could be simpler (as well as misspelled :) ):
DBXQueryTimeout >> asMiliseconds
    ^ (self seconds * 1000) + (((self microseconds / 1000) asFloat) integerPart asInteger )
    ^ (self seconds * 1000) + (self microseconds // 1000)

DBXTimeSpec also has an field called nseconds which contains microseconds, rather confusing :)

 

Thanks in advance! :)

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Hi!

On Tue, Mar 12, 2013 at 3:20 PM, Víctor Casado <[hidden email]> wrote:

Hola Mariano,

Thanks to you for the quick response! Sorry for no give you more information about my scenario... I'm using it on Debian with OpenDBX driver from package libopendbx (which version I assume it's the same as the stable release available on the official website) and the backend is mssql.

So, if I understand correctly, the fact that I'm suffering those blockings is because OpenDBX detects that my database supports async queries and makes them with that feature enabled, right?

No, actually it's most probably the opposite. Since the database driver doesn't support async queries, you perceive large blocks. With async queries the query is done in the background and the image gets blocked only when the results are really ready and transferred to the image side.

 

I was searching and didn't found the method that you mentioned... anyway, is not critical, due I've programmed the queries to be executed while nobody is working on the image, with TaskScheduler. I'll wait to the VM with multithreaded FFI, as you said!

I've another question: I'm trying to install it on a Pharo 2.0 image, but following your instructions and executing the first script it throws me an error: 

 

The symbolic version #stable is not defined in ConfigurationOfOpenDBXDriver for the current platform.

Is OpenDBX compatible on 2.0? If it is, am I doing something wrong? How can I fix it?

It is not yet ported, but I'll do it this week :)

 

Muchas gracias por todo!

De nada! 

Saludos,

Guillermo

 

2013/3/11 Mariano Martinez Peck <[hidden email]>

On Mon, Mar 11, 2013 at 7:41 AM, Víctor Casado <[hidden email]> wrote:

Hola Mariano, Diogenes & company!

First of all thank you for developing such a great tool like DBXTalk, which I'm using right now in a project and it works great! The only problem is, while executing a query, the whole image freezes until it finishes. Is it normal? Is there some way to avoid this behaviour?

Hi Victor. 

Thanks for the nice words. The problem is that the current FFI plugin is blocking FFI...there is an experimental VM with multi-threaded FFI. Still, it is not working properly. You should wait until that. In the meanwhile.... which database, driver, and OS are you using? because most of them support async queries. OpenDBX tries to do async queries when the database client supports them. And from OpenDBXDriver what we do when the backend supports async queries is to do a lot of "small blocking". So we do a kind of busy waiting from the image side, but that allows other process to continue executing. 

For this, see the method #processNextResultSet: aConnection querySettings: aQuerySettings

This line 

(code = OpenDBX resultTimeout) ifTrue: [ (Delay forMilliseconds: (aQuerySettings timeout asMiliseconds)) wait  

allows (the #wait) other Smalltalk process to run....

but of course, all that is only if the backend supports async.

Finally, can these timeouts be improved? yes!!  but we never have time to implemented...

(I copy paste and idea from a collegue)

You could
a) Use a default timeout for the first call which means it actually completes more queries on the first try yet still returns fast, say 100ms rather than 1ms.
(For later calls just to check if it is possibly finished, you probably want to block for as short a time as possible though)
b) Use an exponentially growing value for the Delay rather than a constant one, starting at 1ms and max some other value
1 2 4 8 16 32 64 128 256 512 1024 for instance, polling once per second shouldn't hurt other processes at all, yet give ok responsiveness for queries > 1 seconds.

This way, you (in the cases where potential is 9k queries /sec) will have a hard cap at 10k queries (due to the 100ms block time), and hurt those above that as little as possible using Delays. What you don't have though, is a cap of around 1k, due to calls never completing in 1ms, and having to wait (at least, I don't know the default value of aQuerySettings timeout) 1ms for each due to minimum delay wait time resolution.

Btw, unless the microseconds and seconds are switched, this could be simpler (as well as misspelled :) ):
DBXQueryTimeout >> asMiliseconds
    ^ (self seconds * 1000) + (((self microseconds / 1000) asFloat) integerPart asInteger )
    ^ (self seconds * 1000) + (self microseconds // 1000)

DBXTimeSpec also has an field called nseconds which contains microseconds, rather confusing :)

 

Thanks in advance! :)

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

On Wed, Mar 20, 2013 at 10:57 AM, Víctor Casado <[hidden email]> wrote:

Hola Guillermo!

Sorry for misunderstanding the explanation of Mariano. But, according to this, my backend should support async queries right? Or I need to enable something on my SQL server? Sorry again but I'm pretty new on this.

Hi Victor,

I think that it does not depend on MSSQL itself but the libraries used to communicate to it. In this case it can be with tds or with ODBC. Which one are you using? In other words....which dll of opendbx are you using? or which parameter you used in the ./configure as backend?

I think only the former supports async while the latter (odbc) doesn't.....

 

Good to know you're going to port it to Pharo 2.0 soon... I'm already waiting for it :)

Gracias de nuevo y saludos!

2013/3/17 Guillermo Polito <[hidden email]>

Hi!

On Tue, Mar 12, 2013 at 3:20 PM, Víctor Casado <[hidden email]> wrote:

Hola Mariano,

Thanks to you for the quick response! Sorry for no give you more information about my scenario... I'm using it on Debian with OpenDBX driver from package libopendbx (which version I assume it's the same as the stable release available on the official website) and the backend is mssql.

So, if I understand correctly, the fact that I'm suffering those blockings is because OpenDBX detects that my database supports async queries and makes them with that feature enabled, right?

No, actually it's most probably the opposite. Since the database driver doesn't support async queries, you perceive large blocks. With async queries the query is done in the background and the image gets blocked only when the results are really ready and transferred to the image side.

 

I was searching and didn't found the method that you mentioned... anyway, is not critical, due I've programmed the queries to be executed while nobody is working on the image, with TaskScheduler. I'll wait to the VM with multithreaded FFI, as you said!

I've another question: I'm trying to install it on a Pharo 2.0 image, but following your instructions and executing the first script it throws me an error: 

 

The symbolic version #stable is not defined in ConfigurationOfOpenDBXDriver for the current platform.

Is OpenDBX compatible on 2.0? If it is, am I doing something wrong? How can I fix it?

It is not yet ported, but I'll do it this week :)

 

Muchas gracias por todo!

De nada! 

Saludos,

Guillermo

 

2013/3/11 Mariano Martinez Peck <[hidden email]>

On Mon, Mar 11, 2013 at 7:41 AM, Víctor Casado <[hidden email]> wrote:

Hola Mariano, Diogenes & company!

First of all thank you for developing such a great tool like DBXTalk, which I'm using right now in a project and it works great! The only problem is, while executing a query, the whole image freezes until it finishes. Is it normal? Is there some way to avoid this behaviour?

Hi Victor. 

Thanks for the nice words. The problem is that the current FFI plugin is blocking FFI...there is an experimental VM with multi-threaded FFI. Still, it is not working properly. You should wait until that. In the meanwhile.... which database, driver, and OS are you using? because most of them support async queries. OpenDBX tries to do async queries when the database client supports them. And from OpenDBXDriver what we do when the backend supports async queries is to do a lot of "small blocking". So we do a kind of busy waiting from the image side, but that allows other process to continue executing. 

For this, see the method #processNextResultSet: aConnection querySettings: aQuerySettings

This line 

(code = OpenDBX resultTimeout) ifTrue: [ (Delay forMilliseconds: (aQuerySettings timeout asMiliseconds)) wait  

allows (the #wait) other Smalltalk process to run....

but of course, all that is only if the backend supports async.

Finally, can these timeouts be improved? yes!!  but we never have time to implemented...

(I copy paste and idea from a collegue)

You could
a) Use a default timeout for the first call which means it actually completes more queries on the first try yet still returns fast, say 100ms rather than 1ms.
(For later calls just to check if it is possibly finished, you probably want to block for as short a time as possible though)
b) Use an exponentially growing value for the Delay rather than a constant one, starting at 1ms and max some other value
1 2 4 8 16 32 64 128 256 512 1024 for instance, polling once per second shouldn't hurt other processes at all, yet give ok responsiveness for queries > 1 seconds.

This way, you (in the cases where potential is 9k queries /sec) will have a hard cap at 10k queries (due to the 100ms block time), and hurt those above that as little as possible using Delays. What you don't have though, is a cap of around 1k, due to calls never completing in 1ms, and having to wait (at least, I don't know the default value of aQuerySettings timeout) 1ms for each due to minimum delay wait time resolution.

Btw, unless the microseconds and seconds are switched, this could be simpler (as well as misspelled :) ):
DBXQueryTimeout >> asMiliseconds
    ^ (self seconds * 1000) + (((self microseconds / 1000) asFloat) integerPart asInteger )
    ^ (self seconds * 1000) + (self microseconds // 1000)

DBXTimeSpec also has an field called nseconds which contains microseconds, rather confusing :)

 

Thanks in advance! :)

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Hi Mariano,

I'm using freetds and the opendbx libraries are from debian sid repos (which actual version is 1.4.6), because I was getting troubles compiling it from source... should I try to build it again? Maybe version 1.5.0 is better?

Thanks a lot!

2013/3/20 Mariano Martinez Peck <[hidden email]>

On Wed, Mar 20, 2013 at 10:57 AM, Víctor Casado <[hidden email]> wrote:

Hola Guillermo!

Sorry for misunderstanding the explanation of Mariano. But, according to this, my backend should support async queries right? Or I need to enable something on my SQL server? Sorry again but I'm pretty new on this.

Hi Victor,

I think that it does not depend on MSSQL itself but the libraries used to communicate to it. In this case it can be with tds or with ODBC. Which one are you using? In other words....which dll of opendbx are you using? or which parameter you used in the ./configure as backend?

I think only the former supports async while the latter (odbc) doesn't.....

 

Good to know you're going to port it to Pharo 2.0 soon... I'm already waiting for it :)

Gracias de nuevo y saludos!

2013/3/17 Guillermo Polito <[hidden email]>

Hi!

On Tue, Mar 12, 2013 at 3:20 PM, Víctor Casado <[hidden email]> wrote:

Hola Mariano,

Thanks to you for the quick response! Sorry for no give you more information about my scenario... I'm using it on Debian with OpenDBX driver from package libopendbx (which version I assume it's the same as the stable release available on the official website) and the backend is mssql.

So, if I understand correctly, the fact that I'm suffering those blockings is because OpenDBX detects that my database supports async queries and makes them with that feature enabled, right?

No, actually it's most probably the opposite. Since the database driver doesn't support async queries, you perceive large blocks. With async queries the query is done in the background and the image gets blocked only when the results are really ready and transferred to the image side.

 

I was searching and didn't found the method that you mentioned... anyway, is not critical, due I've programmed the queries to be executed while nobody is working on the image, with TaskScheduler. I'll wait to the VM with multithreaded FFI, as you said!

I've another question: I'm trying to install it on a Pharo 2.0 image, but following your instructions and executing the first script it throws me an error: 

 

The symbolic version #stable is not defined in ConfigurationOfOpenDBXDriver for the current platform.

Is OpenDBX compatible on 2.0? If it is, am I doing something wrong? How can I fix it?

It is not yet ported, but I'll do it this week :)

 

Muchas gracias por todo!

De nada! 

Saludos,

Guillermo

 

2013/3/11 Mariano Martinez Peck <[hidden email]>

On Mon, Mar 11, 2013 at 7:41 AM, Víctor Casado <[hidden email]> wrote:

Hola Mariano, Diogenes & company!

First of all thank you for developing such a great tool like DBXTalk, which I'm using right now in a project and it works great! The only problem is, while executing a query, the whole image freezes until it finishes. Is it normal? Is there some way to avoid this behaviour?

Hi Victor. 

Thanks for the nice words. The problem is that the current FFI plugin is blocking FFI...there is an experimental VM with multi-threaded FFI. Still, it is not working properly. You should wait until that. In the meanwhile.... which database, driver, and OS are you using? because most of them support async queries. OpenDBX tries to do async queries when the database client supports them. And from OpenDBXDriver what we do when the backend supports async queries is to do a lot of "small blocking". So we do a kind of busy waiting from the image side, but that allows other process to continue executing. 

For this, see the method #processNextResultSet: aConnection querySettings: aQuerySettings

This line 

(code = OpenDBX resultTimeout) ifTrue: [ (Delay forMilliseconds: (aQuerySettings timeout asMiliseconds)) wait  

allows (the #wait) other Smalltalk process to run....

but of course, all that is only if the backend supports async.

Finally, can these timeouts be improved? yes!!  but we never have time to implemented...

(I copy paste and idea from a collegue)

You could
a) Use a default timeout for the first call which means it actually completes more queries on the first try yet still returns fast, say 100ms rather than 1ms.
(For later calls just to check if it is possibly finished, you probably want to block for as short a time as possible though)
b) Use an exponentially growing value for the Delay rather than a constant one, starting at 1ms and max some other value
1 2 4 8 16 32 64 128 256 512 1024 for instance, polling once per second shouldn't hurt other processes at all, yet give ok responsiveness for queries > 1 seconds.

This way, you (in the cases where potential is 9k queries /sec) will have a hard cap at 10k queries (due to the 100ms block time), and hurt those above that as little as possible using Delays. What you don't have though, is a cap of around 1k, due to calls never completing in 1ms, and having to wait (at least, I don't know the default value of aQuerySettings timeout) 1ms for each due to minimum delay wait time resolution.

Btw, unless the microseconds and seconds are switched, this could be simpler (as well as misspelled :) ):
DBXQueryTimeout >> asMiliseconds
    ^ (self seconds * 1000) + (((self microseconds / 1000) asFloat) integerPart asInteger )
    ^ (self seconds * 1000) + (self microseconds // 1000)

DBXTimeSpec also has an field called nseconds which contains microseconds, rather confusing :)

 

Thanks in advance! :)

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.