How to improve the speed of a loop containing a sqlalchemy query statement as conditional
- by LtPinback
This loop checks if a record is in the sqlite database and builds a list of dictionaries for those records that are missing and then executes a multiple insert statement with the list. This works but it is very slow (at least i think it is slow) as it takes 5 minutes to loop over 3500 queries. I am a complete newbie in python, sqlite and sqlalchemy so I wonder if there is a faster way of doing this.
list_dict = []
session = Session()
for data in data_list:
if session.query(Class_object).filter(Class_object.column_name_01 == data[2]).filter(Class_object.column_name_00 == an_id).count() == 0:
list_dict.append({'column_name_00':a_id,
'column_name_01':data[2]})
conn = engine.connect()
conn.execute(prices.insert(),list_dict)
conn.close()
session.close()
edit: I moved session = Session() outside the loop. Did not make a difference.
