You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Slow query performance (2-5 seconds per query) on shared compute #459
Description:
I am experiencing a consistent delay of 2-5 seconds for each query execution using the databricks-sql-python library, regardless of whether the query is run on a large or small table. Below are the details of my setup and the issue:
Issue:
Every query executed, whether it's on a large or small table, takes between 2-5 seconds to return results.
This occurs even with very basic queries on small datasets, which raises performance concerns.
Question:
Is this behavior normal when connected through this library? Are there any known ways to improve performance or mitigate the delay ?
Thank you! So, as I understand, your script uses the following pattern: for each query, it does sql.connect, connection.cursor, cursor.execute, and then close everything. If yes - have you tried to reuse connection? Also, please double-check that you properly release all the resources - periodic lags may happen because server has to cleanup orphaned objects
Activity
kravets-levko commentedon Oct 24, 2024
Hi @zabbix-byte! Sometimes delays may happen, but if they're regular - it's suspicious. Can you please share more details:
Also, if possible - please enable debug logging and share output of your script. Add these lines at a very beginning of your script:
Thankk you!
zabbix-byte commentedon Oct 24, 2024
Python version: 3.12

library version: 3.4.0
are you using SQL warehouse or Compute cluster: Compute cluster
DBR version

if possible - share your script (or a simplified version of it which still has the same behavior)
This it lasts 4 seconds in this case.
And make this querys
SELECT * FROM pro_eu_west_1_ireland.x.productos WHERE id=826SELECT * FROM pro_eu_west_1_ireland.x.almacen WHERE id=2SELECT ds, y FROM pro_eu_west_1_ireland.x.forecasting__demand WHERE codigo_producto='1463703' and codigo_tienda=1kravets-levko commentedon Oct 24, 2024
Thank you! So, as I understand, your script uses the following pattern: for each query, it does
sql.connect,connection.cursor,cursor.execute, and then close everything. If yes - have you tried to reuse connection? Also, please double-check that you properly release all the resources - periodic lags may happen because server has to cleanup orphaned objectszabbix-byte commentedon Oct 25, 2024
@kravets-levko If it seems that this was the cause, now the requests practically take half the time.
ci: post Python Integration Tests check as the driver-test app (fix a…
ci: post Python Integration Tests check as the driver-test app (fix a…
ci(bots): add daily retrospective (rolling learning PR) + bump engine (…