Search Results

Search found 55 results on 3 pages for 'sqlbulkcopy'.

Page 3/3 | < Previous Page | 1 2 3 

  • What is causing this SQL 2005 Primary Key Deadlock between two real-time bulk upserts?

    - by skimania
    Here's the scenario: I've got a table called MarketDataCurrent (MDC) that has live updating stock prices. I've got one process called 'LiveFeed' which reads prices streaming from the wire, queues up inserts, and uses a 'bulk upload to temp table then insert/update to MDC table.' (BulkUpsert) I've got another process which then reads this data, computes other data, and then saves the results back into the same table, using a similar BulkUpsert stored proc. Thirdly, there are a multitude of users running a C# Gui polling the MDC table and reading updates from it. Now, during the day when the data is changing rapidly, things run pretty smoothly, but then, after market hours, we've recently started seeing an increasing number of Deadlock exceptions coming out of the database, nowadays we see 10-20 a day. The imporant thing to note here is that these happen when the values are NOT changing. Here's all the relevant info: Table Def: CREATE TABLE [dbo].[MarketDataCurrent]( [MDID] [int] NOT NULL, [LastUpdate] [datetime] NOT NULL, [Value] [float] NOT NULL, [Source] [varchar](20) NULL, CONSTRAINT [PK_MarketDataCurrent] PRIMARY KEY CLUSTERED ( [MDID] ASC )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY] - stackoverflow wont let me post images until my reputation goes up to 10, so i'll add them as soon as you bump me up, hopefully as a result of this question. ![alt text][1] [1]: http://farm5.static.flickr.com/4049/4690759452_6b94ff7b34.jpg I've got a Sql Profiler Trace Running, catching the deadlocks, and here's what all the graphs look like. stackoverflow wont let me post images until my reputation goes up to 10, so i'll add them as soon as you bump me up, hopefully as a result of this question. ![alt text][2] [2]: http://farm5.static.flickr.com/4035/4690125231_78d84c9e15_b.jpg Process 258 is called the following 'BulkUpsert' stored proc, repeatedly, while 73 is calling the next one: ALTER proc [dbo].[MarketDataCurrent_BulkUpload] @updateTime datetime, @source varchar(10) as begin transaction update c with (rowlock) set LastUpdate = getdate(), Value = t.Value, Source = @source from MarketDataCurrent c INNER JOIN #MDTUP t ON c.MDID = t.mdid where c.lastUpdate < @updateTime and c.mdid not in (select mdid from MarketData where LiveFeedTicker is not null and PriceSource like 'LiveFeed.%') and c.value <> t.value insert into MarketDataCurrent with (rowlock) select MDID, getdate(), Value, @source from #MDTUP where mdid not in (select mdid from MarketDataCurrent with (nolock)) and mdid not in (select mdid from MarketData where LiveFeedTicker is not null and PriceSource like 'LiveFeed.%') commit And the other one: ALTER PROCEDURE [dbo].[MarketDataCurrent_LiveFeedUpload] AS begin transaction -- Update existing mdid UPDATE c WITH (ROWLOCK) SET LastUpdate = t.LastUpdate, Value = t.Value, Source = t.Source FROM MarketDataCurrent c INNER JOIN #TEMPTABLE2 t ON c.MDID = t.mdid; -- Insert new MDID INSERT INTO MarketDataCurrent with (ROWLOCK) SELECT * FROM #TEMPTABLE2 WHERE MDID NOT IN (SELECT MDID FROM MarketDataCurrent with (NOLOCK)) -- Clean up the temp table DELETE #TEMPTABLE2 commit To clarify, those Temp Tables are being created by the C# code on the same connection and are populated using the C# SqlBulkCopy class. To me it looks like it's deadlocking on the PK of the table, so I tried removing that PK and switching to a Unique Constraint instead but that increased the number of deadlocks 10-fold. I'm totally lost as to what to do about this situation and am open to just about any suggestion. HELP!!

    Read the article

  • Getting this error while reading excel

    - by Lalit
    I have code string conn = @"Provider=Microsoft.ACE.OLEDB.12.0;DSN=Excel Files;DBQ=" + strSheetPath + @";DriverId=1046;FIL=excel 12.0;MaxBufferSize=2048;PageTimeout=5;"; //string conn = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + strSheetPath +";Extended Properties=Excel 12.0 Xml;HDR=YES;IMEX=1"; string destinationConnectionString = ConfigurationManager.ConnectionStrings["KGD6DBConnectionString"].ConnectionString; string query = "Select * From [Sheet1$]"; using (var myConnection = new OleDbConnection(conn)) using (var destinationConnection = new SqlConnection(destinationConnectionString)) using (var bulkCopy = new SqlBulkCopy(destinationConnection)) { //Map first column in source to second column in sql table (skipping the ID column). //Excel schema[CompanyName,Phone] Table schema[ShipperID, CompanyName, Phone] bulkCopy.ColumnMappings.Add(0, 1); bulkCopy.ColumnMappings.Add(1, 2); bulkCopy.ColumnMappings.Add(2, 3); bulkCopy.ColumnMappings.Add(3, 4); bulkCopy.ColumnMappings.Add(4, 5); bulkCopy.ColumnMappings.Add(5, 6); bulkCopy.ColumnMappings.Add(6, 7); bulkCopy.ColumnMappings.Add(7, 8); bulkCopy.ColumnMappings.Add(8, 9); bulkCopy.ColumnMappings.Add(9, 10); bulkCopy.ColumnMappings.Add(10, 11); bulkCopy.ColumnMappings.Add(11, 12); bulkCopy.ColumnMappings.Add(12, 13); bulkCopy.ColumnMappings.Add(13, 14); bulkCopy.ColumnMappings.Add(14, 15); bulkCopy.ColumnMappings.Add(15, 16); bulkCopy.ColumnMappings.Add(16, 17); bulkCopy.ColumnMappings.Add(17, 18); bulkCopy.ColumnMappings.Add(18, 19); bulkCopy.ColumnMappings.Add(19, 20); bulkCopy.ColumnMappings.Add(20, 21); bulkCopy.ColumnMappings.Add(21, 22); bulkCopy.ColumnMappings.Add(22, 23); bulkCopy.ColumnMappings.Add(23, 24); bulkCopy.DestinationTableName = "dbo.sampleInventory"; using (var myCommand = new OleDbCommand(query, myConnection)) { myConnection.Open(); destinationConnection.Open(); var myReader = myCommand.ExecuteReader(); while (myReader.Read()) { bulkCopy.WriteToServer(myReader); } } } return (new DataTable()); getting error on connection opening line. Error: Could not find installable ISAM.

    Read the article

  • Getting this error while reading excel from c#

    - by Lalit
    Hello all, I have code string conn = @"Provider=Microsoft.ACE.OLEDB.12.0;DSN=Excel Files;DBQ=" + strSheetPath + @";DriverId=1046;FIL=excel 12.0;MaxBufferSize=2048;PageTimeout=5;"; //string conn = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + strSheetPath +";Extended Properties=Excel 12.0 Xml;HDR=YES;IMEX=1"; string destinationConnectionString = ConfigurationManager.ConnectionStrings["KGD6DBConnectionString"].ConnectionString; string query = "Select * From [Sheet1$]"; using (var myConnection = new OleDbConnection(conn)) using (var destinationConnection = new SqlConnection(destinationConnectionString)) using (var bulkCopy = new SqlBulkCopy(destinationConnection)) { //Map first column in source to second column in sql table (skipping the ID column). //Excel schema[CompanyName,Phone] Table schema[ShipperID, CompanyName, Phone] bulkCopy.ColumnMappings.Add(0, 1); bulkCopy.ColumnMappings.Add(1, 2); bulkCopy.ColumnMappings.Add(2, 3); bulkCopy.ColumnMappings.Add(3, 4); bulkCopy.ColumnMappings.Add(4, 5); bulkCopy.ColumnMappings.Add(5, 6); bulkCopy.ColumnMappings.Add(6, 7); bulkCopy.ColumnMappings.Add(7, 8); bulkCopy.ColumnMappings.Add(8, 9); bulkCopy.ColumnMappings.Add(9, 10); bulkCopy.ColumnMappings.Add(10, 11); bulkCopy.ColumnMappings.Add(11, 12); bulkCopy.ColumnMappings.Add(12, 13); bulkCopy.ColumnMappings.Add(13, 14); bulkCopy.ColumnMappings.Add(14, 15); bulkCopy.ColumnMappings.Add(15, 16); bulkCopy.ColumnMappings.Add(16, 17); bulkCopy.ColumnMappings.Add(17, 18); bulkCopy.ColumnMappings.Add(18, 19); bulkCopy.ColumnMappings.Add(19, 20); bulkCopy.ColumnMappings.Add(20, 21); bulkCopy.ColumnMappings.Add(21, 22); bulkCopy.ColumnMappings.Add(22, 23); bulkCopy.ColumnMappings.Add(23, 24); bulkCopy.DestinationTableName = "dbo.sampleInventory"; using (var myCommand = new OleDbCommand(query, myConnection)) { myConnection.Open(); destinationConnection.Open(); var myReader = myCommand.ExecuteReader(); while (myReader.Read()) { bulkCopy.WriteToServer(myReader); } } } return (new DataTable()); getting error on connection opening line. Error: Could not find installable ISAM.

    Read the article

  • Thread locking issue with FileHelpers between calling engine.ReadNext() method and readign engine.Li

    - by Rad
    I use producer/consumer pattern with FileHelpers library to import data from one file (which can be huge) using multiple threads. Each thread is supposed to import a chunk of that file and I would like to use LineNumber property of the FileHelperAsyncEngine instance that is reading the file as primary key for imported rows. FileHelperAsyncEngine internally has an IEnumerator IEnumerable.GetEnumerator(); which is iterated over using engine.ReadNext() method. That internally sets LineNumber property (which seems is not thread safe). Consumers will have Producers assiciated with them that will supply DataTables to Consumers which will consume them via SqlBulkLoad class which will use IDataReader implementation which will iterate over a collection of DataTables which are internal to a Consumer instance. Each instance of will have one SqlBulkCopy instance associate with it. I have thread locking issue. Below is how I create multiple Producer threads. I start each thread afterwords. Produce method on a producer instance will be called determining which chunk of input file will be processed. It seems that engine.LineNumber is not thread safe and I doesn't import a proper LineNumber in the database. It seems that by the time engine.LineNumber is read some other thread called engine.ReadNext() and changed engine.LineNumber property. I don't want to lock the loop that is supposed to process a chunk of input file because I loose parallelism. How to reorganize the code to solve this threading issue? Thanks Rad for (int i = 0; i < numberOfProducerThreads; i++) DataConsumer consumer = dataConsumers[i]; //create a new producer DataProducer producer = new DataProducer(); //consumer has already being created consumer.Subscribe(producer); FileHelperAsyncEngine orderDetailEngine = new FileHelperAsyncEngine(recordType); orderDetailEngine.Options.RecordCondition.Condition = RecordCondition.ExcludeIfBegins; orderDetailEngine.Options.RecordCondition.Selector = STR_ORDR; int skipLines = i * numberOfBufferTablesToProcess * DataBuffer.MaxBufferRowCount; Thread newThread = new Thread(() => { producer.Produce(consumer, inputFilePath, lineNumberFieldName, dict, orderDetailEngine, skipLines, numberOfBufferTablesToProcess); consumer.SetEndOfData(producer); }); producerThreads.Add(newThread); thread.Start();} public void Produce(DataConsumer consumer, string inputFilePath, string lineNumberFieldName, Dictionary<string, object> dict, FileHelperAsyncEngine engine, int skipLines, int numberOfBufferTablesToProcess) { lock (this) { engine.Options.IgnoreFirstLines = skipLines; engine.BeginReadFile(inputFilePath); } int rowCount = 1; DataTable buffer = consumer.BufferDataTable; while (engine.ReadNext() != null) { lock (this) { dict[lineNumberFieldName] = engine.LineNumber; buffer.Rows.Add(ObjectFieldsDataRowMapper.MapObjectFieldsToDataRow(engine.LastRecord, dict, buffer)); if (rowCount % DataBuffer.MaxBufferRowCount == 0) { consumer.AddBufferDataTable(buffer); buffer = consumer.BufferDataTable; } if (rowCount % (numberOfBufferTablesToProcess * DataBuffer.MaxBufferRowCount) == 0) { break; } rowCount++; } } if (buffer.Rows.Count > 0) { consumer.AddBufferDataTable(buffer); } engine.Close(); }

    Read the article

  • Deadlock in SQL Server 2005! Two real-time bulk upserts are fighting. WHY?

    - by skimania
    Here's the scenario: I've got a table called MarketDataCurrent (MDC) that has live updating stock prices. I've got one process called 'LiveFeed' which reads prices streaming from the wire, queues up inserts, and uses a 'bulk upload to temp table then insert/update to MDC table.' (BulkUpsert) I've got another process which then reads this data, computes other data, and then saves the results back into the same table, using a similar BulkUpsert stored proc. Thirdly, there are a multitude of users running a C# Gui polling the MDC table and reading updates from it. Now, during the day when the data is changing rapidly, things run pretty smoothly, but then, after market hours, we've recently started seeing an increasing number of Deadlock exceptions coming out of the database, nowadays we see 10-20 a day. The imporant thing to note here is that these happen when the values are NOT changing. Here's all the relevant info: Table Def: CREATE TABLE [dbo].[MarketDataCurrent]( [MDID] [int] NOT NULL, [LastUpdate] [datetime] NOT NULL, [Value] [float] NOT NULL, [Source] [varchar](20) NULL, CONSTRAINT [PK_MarketDataCurrent] PRIMARY KEY CLUSTERED ( [MDID] ASC )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY] - I've got a Sql Profiler Trace Running, catching the deadlocks, and here's what all the graphs look like. Process 258 is called the following 'BulkUpsert' stored proc, repeatedly, while 73 is calling the next one: ALTER proc [dbo].[MarketDataCurrent_BulkUpload] @updateTime datetime, @source varchar(10) as begin transaction update c with (rowlock) set LastUpdate = getdate(), Value = t.Value, Source = @source from MarketDataCurrent c INNER JOIN #MDTUP t ON c.MDID = t.mdid where c.lastUpdate < @updateTime and c.mdid not in (select mdid from MarketData where LiveFeedTicker is not null and PriceSource like 'LiveFeed.%') and c.value <> t.value insert into MarketDataCurrent with (rowlock) select MDID, getdate(), Value, @source from #MDTUP where mdid not in (select mdid from MarketDataCurrent with (nolock)) and mdid not in (select mdid from MarketData where LiveFeedTicker is not null and PriceSource like 'LiveFeed.%') commit And the other one: ALTER PROCEDURE [dbo].[MarketDataCurrent_LiveFeedUpload] AS begin transaction -- Update existing mdid UPDATE c WITH (ROWLOCK) SET LastUpdate = t.LastUpdate, Value = t.Value, Source = t.Source FROM MarketDataCurrent c INNER JOIN #TEMPTABLE2 t ON c.MDID = t.mdid; -- Insert new MDID INSERT INTO MarketDataCurrent with (ROWLOCK) SELECT * FROM #TEMPTABLE2 WHERE MDID NOT IN (SELECT MDID FROM MarketDataCurrent with (NOLOCK)) -- Clean up the temp table DELETE #TEMPTABLE2 commit To clarify, those Temp Tables are being created by the C# code on the same connection and are populated using the C# SqlBulkCopy class. To me it looks like it's deadlocking on the PK of the table, so I tried removing that PK and switching to a Unique Constraint instead but that increased the number of deadlocks 10-fold. I'm totally lost as to what to do about this situation and am open to just about any suggestion. HELP!! In response to the request for the XDL, here it is: <deadlock-list> <deadlock victim="processc19978"> <process-list> <process id="processaf0b68" taskpriority="0" logused="0" waitresource="KEY: 6:72057594090487808 (d900ed5a6cc6)" waittime="718" ownerId="1102128174" transactionname="user_transaction" lasttranstarted="2010-06-11T16:30:44.750" XDES="0xffffffff817f9a40" lockMode="U" schedulerid="3" kpid="8228" status="suspended" spid="73" sbid="0" ecid="0" priority="0" transcount="2" lastbatchstarted="2010-06-11T16:30:44.750" lastbatchcompleted="2010-06-11T16:30:44.750" clientapp=".Net SqlClient Data Provider" hostname="RISKAPPS_VM" hostpid="3836" loginname="RiskOpt" isolationlevel="read committed (2)" xactid="1102128174" currentdb="6" lockTimeout="4294967295" clientoption1="671088672" clientoption2="128056"> <executionStack> <frame procname="MKP_RISKDB.dbo.MarketDataCurrent_BulkUpload" line="28" stmtstart="1062" stmtend="1720" sqlhandle="0x03000600a28e5e4ef4fd8e00849d00000100000000000000"> UPDATE c WITH (ROWLOCK) SET LastUpdate = getdate(), Value = t.Value, Source = @source FROM MarketDataCurrent c INNER JOIN #MDTUP t ON c.MDID = t.mdid WHERE c.lastUpdate &lt; @updateTime and c.mdid not in (select mdid from MarketData where BloombergTicker is not null and PriceSource like &apos;Blbg.%&apos;) and c.value &lt;&gt; t.value </frame> <frame procname="adhoc" line="1" stmtstart="88" sqlhandle="0x01000600c1653d0598706ca7000000000000000000000000"> exec MarketDataCurrent_BulkUpload @clearBefore, @source </frame> <frame procname="unknown" line="1" sqlhandle="0x000000000000000000000000000000000000000000000000"> unknown </frame> </executionStack> <inputbuf> (@clearBefore datetime,@source nvarchar(10))exec MarketDataCurrent_BulkUpload @clearBefore, @source </inputbuf> </process> <process id="processc19978" taskpriority="0" logused="0" waitresource="KEY: 6:72057594090487808 (74008e31572b)" waittime="718" ownerId="1102128228" transactionname="user_transaction" lasttranstarted="2010-06-11T16:30:44.780" XDES="0x380be9d8" lockMode="U" schedulerid="5" kpid="8464" status="suspended" spid="248" sbid="0" ecid="0" priority="0" transcount="2" lastbatchstarted="2010-06-11T16:30:44.780" lastbatchcompleted="2010-06-11T16:30:44.780" clientapp=".Net SqlClient Data Provider" hostname="RISKBBG_VM" hostpid="4480" loginname="RiskOpt" isolationlevel="read committed (2)" xactid="1102128228" currentdb="6" lockTimeout="4294967295" clientoption1="671088672" clientoption2="128056"> <executionStack> <frame procname="MKP_RISKDB.dbo.MarketDataCurrentBlbgRtUpload" line="14" stmtstart="840" stmtend="1220" sqlhandle="0x03000600005f9d24c8878f00849d00000100000000000000"> UPDATE c WITH (ROWLOCK) SET LastUpdate = t.LastUpdate, Value = t.Value, Source = t.Source FROM MarketDataCurrent c INNER JOIN #TEMPTABLE2 t ON c.MDID = t.mdid; -- Insert new MDID </frame> <frame procname="adhoc" line="1" sqlhandle="0x010006004a58132228bf8d73000000000000000000000000"> MarketDataCurrentBlbgRtUpload </frame> </executionStack> <inputbuf> MarketDataCurrentBlbgRtUpload </inputbuf> </process> </process-list> <resource-list> <keylock hobtid="72057594090487808" dbid="6" objectname="MKP_RISKDB.dbo.MarketDataCurrent" indexname="PK_MarketDataCurrent" id="lock5ba77b00" mode="U" associatedObjectId="72057594090487808"> <owner-list> <owner id="processc19978" mode="U"/> </owner-list> <waiter-list> <waiter id="processaf0b68" mode="U" requestType="wait"/> </waiter-list> </keylock> <keylock hobtid="72057594090487808" dbid="6" objectname="MKP_RISKDB.dbo.MarketDataCurrent" indexname="PK_MarketDataCurrent" id="lock65dca340" mode="U" associatedObjectId="72057594090487808"> <owner-list> <owner id="processaf0b68" mode="U"/> </owner-list> <waiter-list> <waiter id="processc19978" mode="U" requestType="wait"/> </waiter-list> </keylock> </resource-list> </deadlock> </deadlock-list>

    Read the article

< Previous Page | 1 2 3