Improving the performance of a db import process

Posted by mmr on Programmers See other posts from Programmers or by mmr
Published on 2013-11-05T06:57:24Z Indexed on 2013/11/05 10:11 UTC
Read the original article Hit count: 516

I have a program in Microsoft Access that processes text and also inserts data in MySQL database. This operation takes 30 mins or less to finished.

I translated it into VB.NET and it takes 2 hours to finish.

The program goes like this:

  1. A text file contains individual swipe from a corresponding person, it contains their id, time and date of swipe in the machine, and an indicator if it is a time-in or a time-out.
  2. I process this text, segregate the information and insert the time-in and time-out per row.
  3. I also check if there are double occurrences in the database.
  4. After checking, I simply merge the time-in and time-out of the corresponding person into one row only.

This process takes 2 hours to finished in VB.NET considering I have a table to compare which contains 600,000+ rows.

Now, I read in the internet that python is best in text processing, i already have a test but i doubt in database operation.

What do you think is the best programming language for this kind of problem?

How can I speed up the process? My first idea was using python instead of VB.NET, but since people here telling me here on SO that this most probably won't help I am searching for different solutions.

© Programmers or respective owner

Related posts about .NET

Related posts about python