Quantcast

How to avoid image freezing while querying?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to avoid image freezing while querying?

Víctor Casado
Hola Mariano, Diogenes & company!

First of all thank you for developing such a great tool like DBXTalk, which I'm using right now in a project and it works great! The only problem is, while executing a query, the whole image freezes until it finishes. Is it normal? Is there some way to avoid this behaviour?

Thanks in advance! :)

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to avoid image freezing while querying?

Mariano Martinez Peck

On Mon, Mar 11, 2013 at 7:41 AM, Víctor Casado <[hidden email]> wrote:
Hola Mariano, Diogenes & company!

First of all thank you for developing such a great tool like DBXTalk, which I'm using right now in a project and it works great! The only problem is, while executing a query, the whole image freezes until it finishes. Is it normal? Is there some way to avoid this behaviour?


Hi Victor. 

Thanks for the nice words. The problem is that the current FFI plugin is blocking FFI...there is an experimental VM with multi-threaded FFI. Still, it is not working properly. You should wait until that. In the meanwhile.... which database, driver, and OS are you using? because most of them support async queries. OpenDBX tries to do async queries when the database client supports them. And from OpenDBXDriver what we do when the backend supports async queries is to do a lot of "small blocking". So we do a kind of busy waiting from the image side, but that allows other process to continue executing. 

For this, see the method #processNextResultSet: aConnection querySettings: aQuerySettings

This line 

(code = OpenDBX resultTimeout) ifTrue: [ (Delay forMilliseconds: (aQuerySettings timeout asMiliseconds)) wait  

allows (the #wait) other Smalltalk process to run....

but of course, all that is only if the backend supports async.

Finally, can these timeouts be improved? yes!!  but we never have time to implemented...

(I copy paste and idea from a collegue)



You could
a) Use a default timeout for the first call which means it actually completes more queries on the first try yet still returns fast, say 100ms rather than 1ms.
(For later calls just to check if it is possibly finished, you probably want to block for as short a time as possible though)
b) Use an exponentially growing value for the Delay rather than a constant one, starting at 1ms and max some other value
1 2 4 8 16 32 64 128 256 512 1024 for instance, polling once per second shouldn't hurt other processes at all, yet give ok responsiveness for queries > 1 seconds.

This way, you (in the cases where potential is 9k queries /sec) will have a hard cap at 10k queries (due to the 100ms block time), and hurt those above that as little as possible using Delays. What you don't have though, is a cap of around 1k, due to calls never completing in 1ms, and having to wait (at least, I don't know the default value of aQuerySettings timeout) 1ms for each due to minimum delay wait time resolution.

Btw, unless the microseconds and seconds are switched, this could be simpler (as well as misspelled :) ):
DBXQueryTimeout >> asMiliseconds
    ^ (self seconds * 1000) + (((self microseconds / 1000) asFloat) integerPart asInteger )
    ^ (self seconds * 1000) + (self microseconds // 1000)

DBXTimeSpec also has an field called nseconds which contains microseconds, rather confusing :)


 
Thanks in advance! :)

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to avoid image freezing while querying?

Víctor Casado
Hola Mariano,

Thanks to you for the quick response! Sorry for no give you more information about my scenario... I'm using it on Debian with OpenDBX driver from package libopendbx (which version I assume it's the same as the stable release available on the official website) and the backend is mssql.

So, if I understand correctly, the fact that I'm suffering those blockings is because OpenDBX detects that my database supports async queries and makes them with that feature enabled, right?

I was searching and didn't found the method that you mentioned... anyway, is not critical, due I've programmed the queries to be executed while nobody is working on the image, with TaskScheduler. I'll wait to the VM with multithreaded FFI, as you said!

I've another question: I'm trying to install it on a Pharo 2.0 image, but following your instructions and executing the first script it throws me an error:

The symbolic version #stable is not defined in ConfigurationOfOpenDBXDriver for the current platform.

Is OpenDBX compatible on 2.0? If it is, am I doing something wrong? How can I fix it?

Muchas gracias por todo!



2013/3/11 Mariano Martinez Peck <[hidden email]>

On Mon, Mar 11, 2013 at 7:41 AM, Víctor Casado <[hidden email]> wrote:
Hola Mariano, Diogenes & company!

First of all thank you for developing such a great tool like DBXTalk, which I'm using right now in a project and it works great! The only problem is, while executing a query, the whole image freezes until it finishes. Is it normal? Is there some way to avoid this behaviour?


Hi Victor. 

Thanks for the nice words. The problem is that the current FFI plugin is blocking FFI...there is an experimental VM with multi-threaded FFI. Still, it is not working properly. You should wait until that. In the meanwhile.... which database, driver, and OS are you using? because most of them support async queries. OpenDBX tries to do async queries when the database client supports them. And from OpenDBXDriver what we do when the backend supports async queries is to do a lot of "small blocking". So we do a kind of busy waiting from the image side, but that allows other process to continue executing. 

For this, see the method #processNextResultSet: aConnection querySettings: aQuerySettings

This line 

(code = OpenDBX resultTimeout) ifTrue: [ (Delay forMilliseconds: (aQuerySettings timeout asMiliseconds)) wait  

allows (the #wait) other Smalltalk process to run....

but of course, all that is only if the backend supports async.

Finally, can these timeouts be improved? yes!!  but we never have time to implemented...

(I copy paste and idea from a collegue)



You could
a) Use a default timeout for the first call which means it actually completes more queries on the first try yet still returns fast, say 100ms rather than 1ms.
(For later calls just to check if it is possibly finished, you probably want to block for as short a time as possible though)
b) Use an exponentially growing value for the Delay rather than a constant one, starting at 1ms and max some other value
1 2 4 8 16 32 64 128 256 512 1024 for instance, polling once per second shouldn't hurt other processes at all, yet give ok responsiveness for queries > 1 seconds.

This way, you (in the cases where potential is 9k queries /sec) will have a hard cap at 10k queries (due to the 100ms block time), and hurt those above that as little as possible using Delays. What you don't have though, is a cap of around 1k, due to calls never completing in 1ms, and having to wait (at least, I don't know the default value of aQuerySettings timeout) 1ms for each due to minimum delay wait time resolution.

Btw, unless the microseconds and seconds are switched, this could be simpler (as well as misspelled :) ):
DBXQueryTimeout >> asMiliseconds
    ^ (self seconds * 1000) + (((self microseconds / 1000) asFloat) integerPart asInteger )
    ^ (self seconds * 1000) + (self microseconds // 1000)

DBXTimeSpec also has an field called nseconds which contains microseconds, rather confusing :)


 
Thanks in advance! :)

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to avoid image freezing while querying?

Guillermo Polito
Hi!

On Tue, Mar 12, 2013 at 3:20 PM, Víctor Casado <[hidden email]> wrote:
Hola Mariano,

Thanks to you for the quick response! Sorry for no give you more information about my scenario... I'm using it on Debian with OpenDBX driver from package libopendbx (which version I assume it's the same as the stable release available on the official website) and the backend is mssql.

So, if I understand correctly, the fact that I'm suffering those blockings is because OpenDBX detects that my database supports async queries and makes them with that feature enabled, right?

No, actually it's most probably the opposite. Since the database driver doesn't support async queries, you perceive large blocks. With async queries the query is done in the background and the image gets blocked only when the results are really ready and transferred to the image side.
 

I was searching and didn't found the method that you mentioned... anyway, is not critical, due I've programmed the queries to be executed while nobody is working on the image, with TaskScheduler. I'll wait to the VM with multithreaded FFI, as you said!

I've another question: I'm trying to install it on a Pharo 2.0 image, but following your instructions and executing the first script it throws me an error: 
 
The symbolic version #stable is not defined in ConfigurationOfOpenDBXDriver for the current platform.

Is OpenDBX compatible on 2.0? If it is, am I doing something wrong? How can I fix it?


It is not yet ported, but I'll do it this week :)
 
Muchas gracias por todo!

De nada! 
Saludos,
Guillermo
 

2013/3/11 Mariano Martinez Peck <[hidden email]>

On Mon, Mar 11, 2013 at 7:41 AM, Víctor Casado <[hidden email]> wrote:
Hola Mariano, Diogenes & company!

First of all thank you for developing such a great tool like DBXTalk, which I'm using right now in a project and it works great! The only problem is, while executing a query, the whole image freezes until it finishes. Is it normal? Is there some way to avoid this behaviour?


Hi Victor. 

Thanks for the nice words. The problem is that the current FFI plugin is blocking FFI...there is an experimental VM with multi-threaded FFI. Still, it is not working properly. You should wait until that. In the meanwhile.... which database, driver, and OS are you using? because most of them support async queries. OpenDBX tries to do async queries when the database client supports them. And from OpenDBXDriver what we do when the backend supports async queries is to do a lot of "small blocking". So we do a kind of busy waiting from the image side, but that allows other process to continue executing. 

For this, see the method #processNextResultSet: aConnection querySettings: aQuerySettings

This line 

(code = OpenDBX resultTimeout) ifTrue: [ (Delay forMilliseconds: (aQuerySettings timeout asMiliseconds)) wait  

allows (the #wait) other Smalltalk process to run....

but of course, all that is only if the backend supports async.

Finally, can these timeouts be improved? yes!!  but we never have time to implemented...

(I copy paste and idea from a collegue)



You could
a) Use a default timeout for the first call which means it actually completes more queries on the first try yet still returns fast, say 100ms rather than 1ms.
(For later calls just to check if it is possibly finished, you probably want to block for as short a time as possible though)
b) Use an exponentially growing value for the Delay rather than a constant one, starting at 1ms and max some other value
1 2 4 8 16 32 64 128 256 512 1024 for instance, polling once per second shouldn't hurt other processes at all, yet give ok responsiveness for queries > 1 seconds.

This way, you (in the cases where potential is 9k queries /sec) will have a hard cap at 10k queries (due to the 100ms block time), and hurt those above that as little as possible using Delays. What you don't have though, is a cap of around 1k, due to calls never completing in 1ms, and having to wait (at least, I don't know the default value of aQuerySettings timeout) 1ms for each due to minimum delay wait time resolution.

Btw, unless the microseconds and seconds are switched, this could be simpler (as well as misspelled :) ):
DBXQueryTimeout >> asMiliseconds
    ^ (self seconds * 1000) + (((self microseconds / 1000) asFloat) integerPart asInteger )
    ^ (self seconds * 1000) + (self microseconds // 1000)

DBXTimeSpec also has an field called nseconds which contains microseconds, rather confusing :)


 
Thanks in advance! :)

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to avoid image freezing while querying?

Víctor Casado
Hola Guillermo!

Sorry for misunderstanding the explanation of Mariano. But, according to this, my backend should support async queries right? Or I need to enable something on my SQL server? Sorry again but I'm pretty new on this.

Good to know you're going to port it to Pharo 2.0 soon... I'm already waiting for it :)

Gracias de nuevo y saludos!


2013/3/17 Guillermo Polito <[hidden email]>
Hi!

On Tue, Mar 12, 2013 at 3:20 PM, Víctor Casado <[hidden email]> wrote:
Hola Mariano,

Thanks to you for the quick response! Sorry for no give you more information about my scenario... I'm using it on Debian with OpenDBX driver from package libopendbx (which version I assume it's the same as the stable release available on the official website) and the backend is mssql.

So, if I understand correctly, the fact that I'm suffering those blockings is because OpenDBX detects that my database supports async queries and makes them with that feature enabled, right?

No, actually it's most probably the opposite. Since the database driver doesn't support async queries, you perceive large blocks. With async queries the query is done in the background and the image gets blocked only when the results are really ready and transferred to the image side.
 

I was searching and didn't found the method that you mentioned... anyway, is not critical, due I've programmed the queries to be executed while nobody is working on the image, with TaskScheduler. I'll wait to the VM with multithreaded FFI, as you said!

I've another question: I'm trying to install it on a Pharo 2.0 image, but following your instructions and executing the first script it throws me an error: 
 
The symbolic version #stable is not defined in ConfigurationOfOpenDBXDriver for the current platform.

Is OpenDBX compatible on 2.0? If it is, am I doing something wrong? How can I fix it?


It is not yet ported, but I'll do it this week :)
 
Muchas gracias por todo!

De nada! 
Saludos,
Guillermo
 

2013/3/11 Mariano Martinez Peck <[hidden email]>

On Mon, Mar 11, 2013 at 7:41 AM, Víctor Casado <[hidden email]> wrote:
Hola Mariano, Diogenes & company!

First of all thank you for developing such a great tool like DBXTalk, which I'm using right now in a project and it works great! The only problem is, while executing a query, the whole image freezes until it finishes. Is it normal? Is there some way to avoid this behaviour?


Hi Victor. 

Thanks for the nice words. The problem is that the current FFI plugin is blocking FFI...there is an experimental VM with multi-threaded FFI. Still, it is not working properly. You should wait until that. In the meanwhile.... which database, driver, and OS are you using? because most of them support async queries. OpenDBX tries to do async queries when the database client supports them. And from OpenDBXDriver what we do when the backend supports async queries is to do a lot of "small blocking". So we do a kind of busy waiting from the image side, but that allows other process to continue executing. 

For this, see the method #processNextResultSet: aConnection querySettings: aQuerySettings

This line 

(code = OpenDBX resultTimeout) ifTrue: [ (Delay forMilliseconds: (aQuerySettings timeout asMiliseconds)) wait  

allows (the #wait) other Smalltalk process to run....

but of course, all that is only if the backend supports async.

Finally, can these timeouts be improved? yes!!  but we never have time to implemented...

(I copy paste and idea from a collegue)



You could
a) Use a default timeout for the first call which means it actually completes more queries on the first try yet still returns fast, say 100ms rather than 1ms.
(For later calls just to check if it is possibly finished, you probably want to block for as short a time as possible though)
b) Use an exponentially growing value for the Delay rather than a constant one, starting at 1ms and max some other value
1 2 4 8 16 32 64 128 256 512 1024 for instance, polling once per second shouldn't hurt other processes at all, yet give ok responsiveness for queries > 1 seconds.

This way, you (in the cases where potential is 9k queries /sec) will have a hard cap at 10k queries (due to the 100ms block time), and hurt those above that as little as possible using Delays. What you don't have though, is a cap of around 1k, due to calls never completing in 1ms, and having to wait (at least, I don't know the default value of aQuerySettings timeout) 1ms for each due to minimum delay wait time resolution.

Btw, unless the microseconds and seconds are switched, this could be simpler (as well as misspelled :) ):
DBXQueryTimeout >> asMiliseconds
    ^ (self seconds * 1000) + (((self microseconds / 1000) asFloat) integerPart asInteger )
    ^ (self seconds * 1000) + (self microseconds // 1000)

DBXTimeSpec also has an field called nseconds which contains microseconds, rather confusing :)


 
Thanks in advance! :)

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to avoid image freezing while querying?

Mariano Martinez Peck


On Wed, Mar 20, 2013 at 10:57 AM, Víctor Casado <[hidden email]> wrote:
Hola Guillermo!

Sorry for misunderstanding the explanation of Mariano. But, according to this, my backend should support async queries right? Or I need to enable something on my SQL server? Sorry again but I'm pretty new on this.


Hi Victor,

I think that it does not depend on MSSQL itself but the libraries used to communicate to it. In this case it can be with tds or with ODBC. Which one are you using? In other words....which dll of opendbx are you using? or which parameter you used in the ./configure as backend?
I think only the former supports async while the latter (odbc) doesn't.....
 

Good to know you're going to port it to Pharo 2.0 soon... I'm already waiting for it :)

Gracias de nuevo y saludos!


2013/3/17 Guillermo Polito <[hidden email]>
Hi!

On Tue, Mar 12, 2013 at 3:20 PM, Víctor Casado <[hidden email]> wrote:
Hola Mariano,

Thanks to you for the quick response! Sorry for no give you more information about my scenario... I'm using it on Debian with OpenDBX driver from package libopendbx (which version I assume it's the same as the stable release available on the official website) and the backend is mssql.

So, if I understand correctly, the fact that I'm suffering those blockings is because OpenDBX detects that my database supports async queries and makes them with that feature enabled, right?

No, actually it's most probably the opposite. Since the database driver doesn't support async queries, you perceive large blocks. With async queries the query is done in the background and the image gets blocked only when the results are really ready and transferred to the image side.
 

I was searching and didn't found the method that you mentioned... anyway, is not critical, due I've programmed the queries to be executed while nobody is working on the image, with TaskScheduler. I'll wait to the VM with multithreaded FFI, as you said!

I've another question: I'm trying to install it on a Pharo 2.0 image, but following your instructions and executing the first script it throws me an error: 
 
The symbolic version #stable is not defined in ConfigurationOfOpenDBXDriver for the current platform.

Is OpenDBX compatible on 2.0? If it is, am I doing something wrong? How can I fix it?


It is not yet ported, but I'll do it this week :)
 
Muchas gracias por todo!

De nada! 
Saludos,
Guillermo
 

2013/3/11 Mariano Martinez Peck <[hidden email]>

On Mon, Mar 11, 2013 at 7:41 AM, Víctor Casado <[hidden email]> wrote:
Hola Mariano, Diogenes & company!

First of all thank you for developing such a great tool like DBXTalk, which I'm using right now in a project and it works great! The only problem is, while executing a query, the whole image freezes until it finishes. Is it normal? Is there some way to avoid this behaviour?


Hi Victor. 

Thanks for the nice words. The problem is that the current FFI plugin is blocking FFI...there is an experimental VM with multi-threaded FFI. Still, it is not working properly. You should wait until that. In the meanwhile.... which database, driver, and OS are you using? because most of them support async queries. OpenDBX tries to do async queries when the database client supports them. And from OpenDBXDriver what we do when the backend supports async queries is to do a lot of "small blocking". So we do a kind of busy waiting from the image side, but that allows other process to continue executing. 

For this, see the method #processNextResultSet: aConnection querySettings: aQuerySettings

This line 

(code = OpenDBX resultTimeout) ifTrue: [ (Delay forMilliseconds: (aQuerySettings timeout asMiliseconds)) wait  

allows (the #wait) other Smalltalk process to run....

but of course, all that is only if the backend supports async.

Finally, can these timeouts be improved? yes!!  but we never have time to implemented...

(I copy paste and idea from a collegue)



You could
a) Use a default timeout for the first call which means it actually completes more queries on the first try yet still returns fast, say 100ms rather than 1ms.
(For later calls just to check if it is possibly finished, you probably want to block for as short a time as possible though)
b) Use an exponentially growing value for the Delay rather than a constant one, starting at 1ms and max some other value
1 2 4 8 16 32 64 128 256 512 1024 for instance, polling once per second shouldn't hurt other processes at all, yet give ok responsiveness for queries > 1 seconds.

This way, you (in the cases where potential is 9k queries /sec) will have a hard cap at 10k queries (due to the 100ms block time), and hurt those above that as little as possible using Delays. What you don't have though, is a cap of around 1k, due to calls never completing in 1ms, and having to wait (at least, I don't know the default value of aQuerySettings timeout) 1ms for each due to minimum delay wait time resolution.

Btw, unless the microseconds and seconds are switched, this could be simpler (as well as misspelled :) ):
DBXQueryTimeout >> asMiliseconds
    ^ (self seconds * 1000) + (((self microseconds / 1000) asFloat) integerPart asInteger )
    ^ (self seconds * 1000) + (self microseconds // 1000)

DBXTimeSpec also has an field called nseconds which contains microseconds, rather confusing :)


 
Thanks in advance! :)

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to avoid image freezing while querying?

Víctor Casado
Hi Mariano,

I'm using freetds and the opendbx libraries are from debian sid repos (which actual version is 1.4.6), because I was getting troubles compiling it from source... should I try to build it again? Maybe version 1.5.0 is better?

Thanks a lot!


2013/3/20 Mariano Martinez Peck <[hidden email]>


On Wed, Mar 20, 2013 at 10:57 AM, Víctor Casado <[hidden email]> wrote:
Hola Guillermo!

Sorry for misunderstanding the explanation of Mariano. But, according to this, my backend should support async queries right? Or I need to enable something on my SQL server? Sorry again but I'm pretty new on this.


Hi Victor,

I think that it does not depend on MSSQL itself but the libraries used to communicate to it. In this case it can be with tds or with ODBC. Which one are you using? In other words....which dll of opendbx are you using? or which parameter you used in the ./configure as backend?
I think only the former supports async while the latter (odbc) doesn't.....
 

Good to know you're going to port it to Pharo 2.0 soon... I'm already waiting for it :)

Gracias de nuevo y saludos!


2013/3/17 Guillermo Polito <[hidden email]>
Hi!

On Tue, Mar 12, 2013 at 3:20 PM, Víctor Casado <[hidden email]> wrote:
Hola Mariano,

Thanks to you for the quick response! Sorry for no give you more information about my scenario... I'm using it on Debian with OpenDBX driver from package libopendbx (which version I assume it's the same as the stable release available on the official website) and the backend is mssql.

So, if I understand correctly, the fact that I'm suffering those blockings is because OpenDBX detects that my database supports async queries and makes them with that feature enabled, right?

No, actually it's most probably the opposite. Since the database driver doesn't support async queries, you perceive large blocks. With async queries the query is done in the background and the image gets blocked only when the results are really ready and transferred to the image side.
 

I was searching and didn't found the method that you mentioned... anyway, is not critical, due I've programmed the queries to be executed while nobody is working on the image, with TaskScheduler. I'll wait to the VM with multithreaded FFI, as you said!

I've another question: I'm trying to install it on a Pharo 2.0 image, but following your instructions and executing the first script it throws me an error: 
 
The symbolic version #stable is not defined in ConfigurationOfOpenDBXDriver for the current platform.

Is OpenDBX compatible on 2.0? If it is, am I doing something wrong? How can I fix it?


It is not yet ported, but I'll do it this week :)
 
Muchas gracias por todo!

De nada! 
Saludos,
Guillermo
 

2013/3/11 Mariano Martinez Peck <[hidden email]>

On Mon, Mar 11, 2013 at 7:41 AM, Víctor Casado <[hidden email]> wrote:
Hola Mariano, Diogenes & company!

First of all thank you for developing such a great tool like DBXTalk, which I'm using right now in a project and it works great! The only problem is, while executing a query, the whole image freezes until it finishes. Is it normal? Is there some way to avoid this behaviour?


Hi Victor. 

Thanks for the nice words. The problem is that the current FFI plugin is blocking FFI...there is an experimental VM with multi-threaded FFI. Still, it is not working properly. You should wait until that. In the meanwhile.... which database, driver, and OS are you using? because most of them support async queries. OpenDBX tries to do async queries when the database client supports them. And from OpenDBXDriver what we do when the backend supports async queries is to do a lot of "small blocking". So we do a kind of busy waiting from the image side, but that allows other process to continue executing. 

For this, see the method #processNextResultSet: aConnection querySettings: aQuerySettings

This line 

(code = OpenDBX resultTimeout) ifTrue: [ (Delay forMilliseconds: (aQuerySettings timeout asMiliseconds)) wait  

allows (the #wait) other Smalltalk process to run....

but of course, all that is only if the backend supports async.

Finally, can these timeouts be improved? yes!!  but we never have time to implemented...

(I copy paste and idea from a collegue)



You could
a) Use a default timeout for the first call which means it actually completes more queries on the first try yet still returns fast, say 100ms rather than 1ms.
(For later calls just to check if it is possibly finished, you probably want to block for as short a time as possible though)
b) Use an exponentially growing value for the Delay rather than a constant one, starting at 1ms and max some other value
1 2 4 8 16 32 64 128 256 512 1024 for instance, polling once per second shouldn't hurt other processes at all, yet give ok responsiveness for queries > 1 seconds.

This way, you (in the cases where potential is 9k queries /sec) will have a hard cap at 10k queries (due to the 100ms block time), and hurt those above that as little as possible using Delays. What you don't have though, is a cap of around 1k, due to calls never completing in 1ms, and having to wait (at least, I don't know the default value of aQuerySettings timeout) 1ms for each due to minimum delay wait time resolution.

Btw, unless the microseconds and seconds are switched, this could be simpler (as well as misspelled :) ):
DBXQueryTimeout >> asMiliseconds
    ^ (self seconds * 1000) + (((self microseconds / 1000) asFloat) integerPart asInteger )
    ^ (self seconds * 1000) + (self microseconds // 1000)

DBXTimeSpec also has an field called nseconds which contains microseconds, rather confusing :)


 
Thanks in advance! :)

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to avoid image freezing while querying?

Guillermo Polito
Hi, sorry for the delay...

1.4.6 is the current stable version. 1.5 is AFAIK kind of experimental and does not include all the latest fixes.

From an old mail from Norbert (from OpenDBX), I quote...

"This are the combinations that work:

MSSQL/All
PostgreSQL/Linux
SQLite/All
SQLite3/All

PostgreSQL/Windows and Sybase might be possible but aren't implemented yet."

So in theory mssql backend using freetds should work...
But I don't have much knowledge about it... Maybe asking in the opendbx mailing list...

Sorry for not being much helpful :/
Guille


On Mon, Mar 25, 2013 at 10:09 AM, Víctor Casado <[hidden email]> wrote:
Hi Mariano,

I'm using freetds and the opendbx libraries are from debian sid repos (which actual version is 1.4.6), because I was getting troubles compiling it from source... should I try to build it again? Maybe version 1.5.0 is better?

Thanks a lot!


2013/3/20 Mariano Martinez Peck <[hidden email]>


On Wed, Mar 20, 2013 at 10:57 AM, Víctor Casado <[hidden email]> wrote:
Hola Guillermo!

Sorry for misunderstanding the explanation of Mariano. But, according to this, my backend should support async queries right? Or I need to enable something on my SQL server? Sorry again but I'm pretty new on this.


Hi Victor,

I think that it does not depend on MSSQL itself but the libraries used to communicate to it. In this case it can be with tds or with ODBC. Which one are you using? In other words....which dll of opendbx are you using? or which parameter you used in the ./configure as backend?
I think only the former supports async while the latter (odbc) doesn't.....
 

Good to know you're going to port it to Pharo 2.0 soon... I'm already waiting for it :)

Gracias de nuevo y saludos!


2013/3/17 Guillermo Polito <[hidden email]>
Hi!

On Tue, Mar 12, 2013 at 3:20 PM, Víctor Casado <[hidden email]> wrote:
Hola Mariano,

Thanks to you for the quick response! Sorry for no give you more information about my scenario... I'm using it on Debian with OpenDBX driver from package libopendbx (which version I assume it's the same as the stable release available on the official website) and the backend is mssql.

So, if I understand correctly, the fact that I'm suffering those blockings is because OpenDBX detects that my database supports async queries and makes them with that feature enabled, right?

No, actually it's most probably the opposite. Since the database driver doesn't support async queries, you perceive large blocks. With async queries the query is done in the background and the image gets blocked only when the results are really ready and transferred to the image side.
 

I was searching and didn't found the method that you mentioned... anyway, is not critical, due I've programmed the queries to be executed while nobody is working on the image, with TaskScheduler. I'll wait to the VM with multithreaded FFI, as you said!

I've another question: I'm trying to install it on a Pharo 2.0 image, but following your instructions and executing the first script it throws me an error: 
 
The symbolic version #stable is not defined in ConfigurationOfOpenDBXDriver for the current platform.

Is OpenDBX compatible on 2.0? If it is, am I doing something wrong? How can I fix it?


It is not yet ported, but I'll do it this week :)
 
Muchas gracias por todo!

De nada! 
Saludos,
Guillermo
 

2013/3/11 Mariano Martinez Peck <[hidden email]>

On Mon, Mar 11, 2013 at 7:41 AM, Víctor Casado <[hidden email]> wrote:
Hola Mariano, Diogenes & company!

First of all thank you for developing such a great tool like DBXTalk, which I'm using right now in a project and it works great! The only problem is, while executing a query, the whole image freezes until it finishes. Is it normal? Is there some way to avoid this behaviour?


Hi Victor. 

Thanks for the nice words. The problem is that the current FFI plugin is blocking FFI...there is an experimental VM with multi-threaded FFI. Still, it is not working properly. You should wait until that. In the meanwhile.... which database, driver, and OS are you using? because most of them support async queries. OpenDBX tries to do async queries when the database client supports them. And from OpenDBXDriver what we do when the backend supports async queries is to do a lot of "small blocking". So we do a kind of busy waiting from the image side, but that allows other process to continue executing. 

For this, see the method #processNextResultSet: aConnection querySettings: aQuerySettings

This line 

(code = OpenDBX resultTimeout) ifTrue: [ (Delay forMilliseconds: (aQuerySettings timeout asMiliseconds)) wait  

allows (the #wait) other Smalltalk process to run....

but of course, all that is only if the backend supports async.

Finally, can these timeouts be improved? yes!!  but we never have time to implemented...

(I copy paste and idea from a collegue)



You could
a) Use a default timeout for the first call which means it actually completes more queries on the first try yet still returns fast, say 100ms rather than 1ms.
(For later calls just to check if it is possibly finished, you probably want to block for as short a time as possible though)
b) Use an exponentially growing value for the Delay rather than a constant one, starting at 1ms and max some other value
1 2 4 8 16 32 64 128 256 512 1024 for instance, polling once per second shouldn't hurt other processes at all, yet give ok responsiveness for queries > 1 seconds.

This way, you (in the cases where potential is 9k queries /sec) will have a hard cap at 10k queries (due to the 100ms block time), and hurt those above that as little as possible using Delays. What you don't have though, is a cap of around 1k, due to calls never completing in 1ms, and having to wait (at least, I don't know the default value of aQuerySettings timeout) 1ms for each due to minimum delay wait time resolution.

Btw, unless the microseconds and seconds are switched, this could be simpler (as well as misspelled :) ):
DBXQueryTimeout >> asMiliseconds
    ^ (self seconds * 1000) + (((self microseconds / 1000) asFloat) integerPart asInteger )
    ^ (self seconds * 1000) + (self microseconds // 1000)

DBXTimeSpec also has an field called nseconds which contains microseconds, rather confusing :)


 
Thanks in advance! :)

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Víctor Casado

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "DBXTalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Loading...