Bad performance function in PHP. With large files memory blows up! How can I refactor?
Posted
by
André
on Stack Overflow
See other posts from Stack Overflow
or by André
Published on 2010-12-21T16:50:53Z
Indexed on
2010/12/21
16:54 UTC
Read the original article
Hit count: 276
Hi
I have a function that strips out lines from files. I'm handling with large files(more than 100Mb). I have the PHP Memory with 256MB but the function that handles with the strip out of lines blows up with a 100MB CSV File.
What the function must do is this:
Originally I have the CSV like:
Copyright (c) 2007 MaxMind LLC. All Rights Reserved. locId,country,region,city,postalCode,latitude,longitude,metroCode,areaCode 1,"O1","","","",0.0000,0.0000,, 2,"AP","","","",35.0000,105.0000,, 3,"EU","","","",47.0000,8.0000,, 4,"AD","","","",42.5000,1.5000,, 5,"AE","","","",24.0000,54.0000,, 6,"AF","","","",33.0000,65.0000,, 7,"AG","","","",17.0500,-61.8000,, 8,"AI","","","",18.2500,-63.1667,, 9,"AL","","","",41.0000,20.0000,,
When I pass the CSV file to this function I got:
locId,country,region,city,postalCode,latitude,longitude,metroCode,areaCode 1,"O1","","","",0.0000,0.0000,, 2,"AP","","","",35.0000,105.0000,, 3,"EU","","","",47.0000,8.0000,, 4,"AD","","","",42.5000,1.5000,, 5,"AE","","","",24.0000,54.0000,, 6,"AF","","","",33.0000,65.0000,, 7,"AG","","","",17.0500,-61.8000,, 8,"AI","","","",18.2500,-63.1667,, 9,"AL","","","",41.0000,20.0000,,
It only strips out the first line, nothing more. The problem is the performance of this function with large files, it blows up the memory.
The function is:
public function deleteLine($line_no, $csvFileName) {
// this function strips a specific line from a file
// if a line is stripped, functions returns True else false
//
// e.g.
// deleteLine(-1, xyz.csv); // strip last line
// deleteLine(1, xyz.csv); // strip first line
// Assigna o nome do ficheiro
$filename = $csvFileName;
$strip_return=FALSE;
$data=file($filename);
$pipe=fopen($filename,'w');
$size=count($data);
if($line_no==-1) $skip=$size-1;
else $skip=$line_no-1;
for($line=0;$line<$size;$line++)
if($line!=$skip)
fputs($pipe,$data[$line]);
else
$strip_return=TRUE;
return $strip_return;
}
It is possible to refactor this function to not blow up with the 256MB PHP Memory?
Give me some clues.
Best Regards,
© Stack Overflow or respective owner