How to improve the speed of a loop containing a sqlalchemy query statement as conditional

Posted by LtPinback on Stack Overflow See other posts from Stack Overflow or by LtPinback
Published on 2010-04-08T22:32:30Z Indexed on 2010/04/08 22:43 UTC
Read the original article Hit count: 170

Filed under:
|
|
|

This loop checks if a record is in the sqlite database and builds a list of dictionaries for those records that are missing and then executes a multiple insert statement with the list. This works but it is very slow (at least i think it is slow) as it takes 5 minutes to loop over 3500 queries. I am a complete newbie in python, sqlite and sqlalchemy so I wonder if there is a faster way of doing this.

list_dict = []

session = Session()

for data in data_list:
    if session.query(Class_object).filter(Class_object.column_name_01 == data[2]).filter(Class_object.column_name_00 == an_id).count() == 0:
        list_dict.append({'column_name_00':a_id,
                          'column_name_01':data[2]})

conn = engine.connect()
conn.execute(prices.insert(),list_dict)
conn.close()
session.close()

edit: I moved session = Session() outside the loop. Did not make a difference.

© Stack Overflow or respective owner

Related posts about python

Related posts about sqlite