curl: downloading from dynamic url

Posted by adam n on Stack Overflow See other posts from Stack Overflow or by adam n
Published on 2010-03-31T19:01:33Z Indexed on 2010/04/01 4:53 UTC
Read the original article Hit count: 335

Filed under:
|
|
|
|

I'm trying to download an html file with curl in bash. Like this site: http://www.registrar.ucla.edu/schedule/detselect.aspx?termsel=10S&subareasel=PHYSICS&idxcrs=0001B+++

When I download it manually, it works fine. However, when i try and run my script through crontab, the output html file is very small and just says "Object moved to here." with a broken link. Does this have something to do with the sparse environment the crontab commands run it? I found this question:

http://stackoverflow.com/questions/1279340/php-ssl-curl-object-moved-error

but i'm using bash, not php. What are the equivalent command line options or variables to set to fix this problem in bash?

(I want to do this with curl, not wget)

Edit: well, sometimes downloading the file manually (via interactive shell) works, but sometimes it doesn't (I still get the "Object moved here" message). So it may not be a a specifically be a problem with cron's environment, but with curl itself.

the cron entry:
* * * * * ~/.class/test.sh >> ~/.class/test_out 2>&1

test.sh:

#! /bin/bash
PATH=/usr/local/bin:/usr/bin:/bin:/sbin
cd ~/.class

course="physics 1b"
url="http://www.registrar.ucla.edu/schedule/detselect.aspx?termsel=10S<URL>subareasel=PHYSICS<URL>idxcrs=0001B+++"

curl "$url" -sLo "$course".html  --max-redirs 5

As I was searching around on google, someone suggested that the problem might happen because there are parameters in the url. (Because it is a dynamic url?)

© Stack Overflow or respective owner

Related posts about curl

Related posts about bash