Improving the performance of a db import process
Posted
by
mmr
on Programmers
See other posts from Programmers
or by mmr
Published on 2013-11-05T06:57:24Z
Indexed on
2013/11/05
10:11 UTC
Read the original article
Hit count: 511
I have a program in Microsoft Access that processes text and also inserts data in MySQL database. This operation takes 30 mins or less to finished.
I translated it into VB.NET and it takes 2 hours to finish.
The program goes like this:
- A text file contains individual swipe from a corresponding person, it contains their id, time and date of swipe in the machine, and an indicator if it is a time-in or a time-out.
- I process this text, segregate the information and insert the time-in and time-out per row.
- I also check if there are double occurrences in the database.
- After checking, I simply merge the time-in and time-out of the corresponding person into one row only.
This process takes 2 hours to finished in VB.NET considering I have a table to compare which contains 600,000+ rows.
Now, I read in the internet that python is best in text processing, i already have a test but i doubt in database operation.
What do you think is the best programming language for this kind of problem?
How can I speed up the process? My first idea was using python instead of VB.NET, but since people here telling me here on SO that this most probably won't help I am searching for different solutions.
© Programmers or respective owner