Efficient update of SQLite table with many records

Posted by blackrim on Stack Overflow See other posts from Stack Overflow or by blackrim
Published on 2010-05-15T03:18:11Z Indexed on 2010/05/15 3:24 UTC
Read the original article Hit count: 216

Filed under:
|
|
|
|

I am trying to use sqlite (sqlite3) for a project to store hundreds of thousands of records (would like sqlite so users of the program don't have to run a [my]sql server).

I have to update hundreds of thousands of records sometimes to enter left right values (they are hierarchical), but have found the standard

update table set left_value = 4, right_value = 5 where id = 12340;

to be very slow. I have tried surrounding every thousand or so with

begin;
....
update...
update table set left_value = 4, right_value = 5 where id = 12340;
update...
....
commit;

but again, very slow. Odd, because when I populate it with a few hundred thousand (with inserts), it finishes in seconds.

I am currently trying to test the speed in python (the slowness is at the command line and python) before I move it to the C++ implementation, but right now this is way to slow and I need to find a new solution unless I am doing something wrong. Thoughts? (would take open source alternative to SQLite that is portable as well)

© Stack Overflow or respective owner

Related posts about sqlite

Related posts about python