Memory efficient import many data files into panda DataFrame in Python
- by richardh
I import into a panda DataFrame a directory of |-delimited.dat files. The following code works, but I eventually run out of RAM with a MemoryError:.
import pandas as pd
import glob
temp = []
dataDir = 'C:/users/richard/research/data/edgar/masterfiles'
for dataFile in glob.glob(dataDir + '/master_*.dat'):
print dataFile
…
