This is a general design question about how to make a web application that will receive a large amount of uploaded data, process it, and return a result, all without the dreaded spinning beach-ball for 5 minutes or a possible HTTP timeout.

Here's the requirements:

• make a web form where you can upload a CSV file containing a list of URLs
• when the user clicks "submit", the server fetches the file, and checks each URL to see if its alive, and what the title tag of the page is.
• the result is a downloadable CSV file containing the URL, and the result HTTP code
• the input CSV can be very large ( > 100000 rows), so the fetch process might take 5-30 minutes.

My solution so far is to have a spinning javascript loop on the client site, which queries the server every second to determine the overall progress of the job. This seems kludgy to me, and I'm hesitant to accept this as the best solution.

I'm using perl, template toolkit, and jquery, but any solution using any web technology would be acceptable.