I am using IBM Websphere Application Server v6 and Java 1.4 and am trying to write large CSV files to the ServletOutputStream for a user to download. Files are ranging from a 50-750MB at the moment.

The smaller files aren't causing too much of a problem but with the larger files it appears that it is being written into the heap which is then causing an OutOfMemory error and bringing down the entire server.

These files can only be served out to authenticated users over https which is why I am serving them through a Servlet instead of just sticking them in Apache.

The code I am using is (some fluff removed around this):

    resp.setHeader("Content-length", "" + fileLength);
    resp.setContentType("application/vnd.ms-excel");
    resp.setHeader("Content-Disposition","attachment; filename=\"export.csv\"");

    FileInputStream inputStream = null;

    try
    {
        inputStream = new FileInputStream(path);
        byte[] buffer = new byte[1024];
        int bytesRead = 0;

        do
        {
        	bytesRead = inputStream.read(buffer, offset, buffer.length);
        	resp.getOutputStream().write(buffer, 0, bytesRead);
        }
        while (bytesRead == buffer.length);

        resp.getOutputStream().flush();
    }
    finally
    {
        if(inputStream != null)
	        inputStream.close();
    }

The FileInputStream doesn't seem to be causing a problem as if I write to another file or just remove the write completly the memory usage doesn't appear to be a problem.

What I am thinking is that the resp.getOutputStream().write is being stored in memory until the data can be sent through to the client. So the entire file might be read and stored in the resp.getOutputStream() causing my memory issues and crashing!

I have tried Buffering these streams and also tried using Channels from java.nio, none of which seems to make any bit of difference to my memory issues. I have also flushed the outputstream once per iteration of the loop and after the loop, which didn't help.