rcurl web scraping timeout exits program
Posted
by
user1742368
on Stack Overflow
See other posts from Stack Overflow
or by user1742368
Published on 2012-10-12T21:33:48Z
Indexed on
2012/10/12
21:36 UTC
Read the original article
Hit count: 230
I am using a loop and rcurl scrape data from multiple pages which seems to work fine at certain times but fails when there is a timeout due to the server not responding. I am using a timeout=30 which traps the timeout error however the program stops after the timeout. i would like the progrm to continue to the next page when the timeout occurrs but cant figureout how to do this? url = getCurlHandle(cookiefile = "", verbose = TRUE)
Here is the statement I am using that causes the timeout. I am happy to share the code if there is interest. webpage = getURLContent(url, followlocation=TRUE, curl = curl,.opts=list( verbose = TRUE, timeout=90, maxredirs = 2))
woodwardjj
© Stack Overflow or respective owner