Can EC2 instances be set up to come from different IP ranges?
Posted
by Joshua Frank
on Stack Overflow
See other posts from Stack Overflow
or by Joshua Frank
Published on 2010-06-03T12:52:51Z
Indexed on
2010/06/03
13:24 UTC
Read the original article
Hit count: 311
I need to run a web crawler and I want to do it from EC2 because I want the HTTP requests to come from different IP ranges so I don't get blocked. So I thought distributing this on EC2 instances might help, but I can't find any information about what the outbound IP range will be. I don't want to go to the trouble of figuring out the extra complexity of EC2 and distributed data, only to find that all the instances use the same address block and I get blocked by the server anyway.
NOTE: This isn't for a DoS attack or anything. I'm trying to harvest data for a legitimate business purpose, I'm respecting robots.txt, and I'm only making one request per second, but the host is still shutting me down.
Edit: Commenter Paul Dixon suggests that the act of blocking even my modest crawl indicates that the host doesn't want me to crawl them and therefore that I shouldn't do it (even assuming I can work around the blocking). Do people agree with this?
© Stack Overflow or respective owner