Parsing large txt files in ruby taking a lot of time?

Posted by hershey92 on Stack Overflow See other posts from Stack Overflow or by hershey92
Published on 2012-07-01T11:59:01Z Indexed on 2012/07/01 21:16 UTC
Read the original article Hit count: 188

Filed under:

below is the code to download a txt file from internet approx 9000 lines and populate the database, I have tried a lot but it takes a lot of time more than 7 minutes. I am using win 7 64 bit and ruby 1.9.3. Is there a way to do it faster ??

require 'open-uri'
require 'dbi'
dbh = DBI.connect("DBI:Mysql:mfmodel:localhost","root","")
#file = open('http://www.amfiindia.com/spages/NAV0.txt')
file = File.open('test.txt','r')
lines = file.lines
2.times { lines.next }
curSubType = ''
curType = ''
curCompName = ''
lines.each do |line|
    line.strip!
if line[-1] == ')'
    curType,curSubType = line.split('(')
    curSubType.chop!
elsif line[-4..-1] == 'Fund'
    curCompName = line.split(" Mutual Fund")[0] 
elsif line == ''
    next
else
    sCode,isin_div,isin_re,sName,nav,rePrice,salePrice,date = line.split(';')
    sCode = Integer(sCode)
    sth = dbh.prepare "call mfmodel.populate(?,?,?,?,?,?,?)"
    sth.execute curCompName,curSubType,curType,sCode,isin_div,isin_re,sName
end
end
dbh.do "commit"
dbh.disconnect
file.close


106799;-;-;HDFC ARBITRAGE FUND RETAIL PLAN DIVIDEND OPTION;10.352;10.3;10.352;29-Jun-2012

This is the format of data to be inserted in the table. Now there are 8000 such lines and how can I do an insert by combining all that and call the procedure just once. Also, does mysql support arrays and iteration to do such a thing inside the routine. Please give your suggestions.Thanks.

© Stack Overflow or respective owner

Related posts about ruby