Bad performance function in PHP. With large files memory blows up! How can I refactor?

Posted by André on Stack Overflow See other posts from Stack Overflow or by André
Published on 2010-12-21T16:50:53Z Indexed on 2010/12/21 16:54 UTC
Read the original article Hit count: 280

Filed under:
|
|

Hi

I have a function that strips out lines from files. I'm handling with large files(more than 100Mb). I have the PHP Memory with 256MB but the function that handles with the strip out of lines blows up with a 100MB CSV File.

What the function must do is this:

Originally I have the CSV like:

Copyright (c) 2007 MaxMind LLC. All Rights Reserved. locId,country,region,city,postalCode,latitude,longitude,metroCode,areaCode 1,"O1","","","",0.0000,0.0000,, 2,"AP","","","",35.0000,105.0000,, 3,"EU","","","",47.0000,8.0000,, 4,"AD","","","",42.5000,1.5000,, 5,"AE","","","",24.0000,54.0000,, 6,"AF","","","",33.0000,65.0000,, 7,"AG","","","",17.0500,-61.8000,, 8,"AI","","","",18.2500,-63.1667,, 9,"AL","","","",41.0000,20.0000,,

When I pass the CSV file to this function I got:

locId,country,region,city,postalCode,latitude,longitude,metroCode,areaCode 1,"O1","","","",0.0000,0.0000,, 2,"AP","","","",35.0000,105.0000,, 3,"EU","","","",47.0000,8.0000,, 4,"AD","","","",42.5000,1.5000,, 5,"AE","","","",24.0000,54.0000,, 6,"AF","","","",33.0000,65.0000,, 7,"AG","","","",17.0500,-61.8000,, 8,"AI","","","",18.2500,-63.1667,, 9,"AL","","","",41.0000,20.0000,,

It only strips out the first line, nothing more. The problem is the performance of this function with large files, it blows up the memory.

The function is:

 public function deleteLine($line_no, $csvFileName) {

  // this function strips a specific line from a file
  // if a line is stripped, functions returns True else false
  //
  // e.g.
  // deleteLine(-1, xyz.csv); // strip last line
  // deleteLine(1, xyz.csv); // strip first line

  // Assigna o nome do ficheiro
  $filename = $csvFileName;

  $strip_return=FALSE;

  $data=file($filename);
  $pipe=fopen($filename,'w');
  $size=count($data);

  if($line_no==-1) $skip=$size-1;
  else $skip=$line_no-1;

  for($line=0;$line<$size;$line++)
   if($line!=$skip)
    fputs($pipe,$data[$line]);
   else
    $strip_return=TRUE;

  return $strip_return;
 }

It is possible to refactor this function to not blow up with the 256MB PHP Memory?

Give me some clues.

Best Regards,

© Stack Overflow or respective owner

Related posts about php

Related posts about Performance