I am currently using CURL via a php script running as daily cron to export product data in csv format from a site's admin area.

The normal way of exporting data will be to go to the Export page in a browser, and set the configuration, then click on "export data" button. But as the number of products I am exporting is very large, and it takes more than 5-10 mins to export the data, I've decided to use php's curl function to mimic this on a daily basis via cron.

Previously, it is working fine, but recently as I increased the number of products in the store by 500+, the script fails to return the exported data. Testing it manually via clicking on the "export" button in a browser, does return the data correctly. Thus there is no "timeout" issue with running the export in a browser manually.

I've tested and by removing/decreasing the number of products (thus the time needed), the php-curl script works fine again when run from cron.

So I suspect that it has something to do with timeouts issue, specifically with the curl function in php.

I've set both CURLOPT_TIMEOUT and CURLOPT_CONNECTTIMEOUT to '0' respectively to try. In the php-curl script, I've also set "set_time_limit(3000)". But still it does not work, and the request will timeout, with the script failing to return with a complete set of csv data.

Any help in helping me resolve/understand this issue will be much appreciated!