how can I speed up insertion of many rows to a table via ADO.NET?

Posted by jcollum on Stack Overflow See other posts from Stack Overflow or by jcollum
Published on 2010-04-16T18:09:44Z Indexed on 2010/04/16 18:13 UTC
Read the original article Hit count: 338

Filed under:
|

I have a table that has 5 columns: AcctId (int), Address1 (varchar), Address2 (varchar), Person1 (varchar), Person2 (varchar) . I'm generating random data to insert into this table via a C# console application. I've tried doing this random data insert via SQL-Server and decided it was not a good solution -- SQL is not good at random on an each-row basis. Generating the random data -- 975k rows of it -- takes a minimal amount of time. It's in a List of custom objects.

I need to take this random data and update many rows in the database with the new random data. I tried updating the rows one at a time, very slow because of the repeated searching of the List object in code. So I think the best approach is to put all the randomized data into a table in the database, then update all the other tables that use this data. I.e. UPDATE t SET t.Address1=d.Address1 FROM Table1 t INNER JOIN RandomizedData d ON d.AcctId = t.Acct_ID. The database is very un-normalized so this Acct data is sprinkled all over the place. I've got no control of the normalization.

So, having decided to insert all of the randomized data into a single table, I set out to create insert scripts:

USE TheDatabase
Insert tmp_RandomizedData
 SELECT 1,'4392 EIGHTH AVE','','JENNIFER CARTER','BARBARA CARTER'  UNION ALL  
 SELECT 2,'2168 MAIN ST','HNGR F','DANIEL HERNANDEZ','SUSAN MARTIN'
// etc another 98 times... 
// FYI, this is not real data! 

I'm building this INSERT script in batches of 100. It's taking on average 175 ms to run each insert. Does this seem like a long time? It's going to take about 35 mins to run the whole insert.

The table doesn't have a primary key or any indexes. I was planning on adding those after all the data in inserted (thinking that that would be faster).

Is there a better way to do this?

© Stack Overflow or respective owner

Related posts about c#

Related posts about sql-server