Load balanced proxies to avoid an API request limit

Posted by ClickClickClick on Server Fault See other posts from Server Fault or by ClickClickClick
Published on 2013-11-02T15:25:24Z Indexed on 2013/11/02 15:54 UTC
Read the original article Hit count: 161

Filed under:

proxy

|

load-balancing

|

transparent-proxy

There is a certain API out there which limits the number of requests per day per IP. My plan is to create a bunch of EC2 instances with elastic IPs to sidestep the limitation. I'm familiar with EC2 and am just interested in the configuration of the proxies and a software load balancer.

I think I want to run a simple TCP Proxy on each instance and a software load balancer on the machine I will be requesting from. Something that allows the following to return a response from a different IP (round robin, availability, doesn't really matter..)

eg.

curl http://www.bbc.co.uk -x http://myproxyloadbalancer:port

Could anyone recommend a combination of software or even a link to an article that details a pleasing way to pull it off?

(My client won't be curl but is proxy aware.. I'll be making the requests from a Ruby script..)

© Server Fault or respective owner

Related posts about proxy

DNS lookup failures while accessing my website some proxy error

as seen on Server Fault - Search for 'Server Fault'
Here is a situation until today morning,every thing has been working perfectly fine with me. From past 6 months many of my domains wer accessible as http://site1.myserver.com http://site2.myserver.com http://site3.myserver.com http://site4.myserver.com All these were Reverse Proxy configurations… >>> More
Apache2 mod_proxy to remote Tomcat7 - slow response

as seen on Server Fault - Search for 'Server Fault'
Been stuck with this one for a few days. Will try to provide as much information as possible, but please feel free to ask for extra detail. I have 2 VMs behind a NAT, 192.168.0.100 and 192.168.0.102, both running Ubuntu 11.04 x64. The first one is mapped to the exterior and is our webserver, has… >>> More
How to set a proxy for Webbrowser Control without effecting the SYSTEM/IE proxy

as seen on Stack Overflow - Search for 'Stack Overflow'
How can I set a proxy for a Webbrowser Control without setting a proxy for IE. Basically I want my application to use a certain proxy however obviously I don't want to change user's IE proxy settings. >>> More
Difference between proxy server and reverse proxy server

as seen on Stack Overflow - Search for 'Stack Overflow'
What is the difference between proxy server and reverse proxy server? >>> More
[MAJ] Transformer APACHE 2.0 en Proxy HTTP - Comment utiliser le Module Proxy d'Apache

as seen on ASP-PHP.net - Search for 'ASP-PHP.net'
Modification du script suite ? l'alerte concernant la possibilit? offerte ? n'importe qui d'utiliser le moteur APACHE comme proxy sans aucun contr?le. >>> More

Related posts about load-balancing

Failover or load balancing configuration on Apple Xserve and Mac OS X Server Snow Leopard (10.6)

as seen on Server Fault - Search for 'Server Fault'
I'm currently installing a number of Apple Xserve boxes running Mac OS X Server (10.6). After poking and prodding for a while I can't find any obvious way, or documentation telling me how, to set up the 2 ethernet ports in a failover or load-balancing configuration. There seems to be an obvious… >>> More
Conceptually how does load-balancing on the EJB tier work in Glassfish/any ejb container

as seen on Stack Overflow - Search for 'Stack Overflow'
I am wondering conceptually how load-balancing works on the EJB-level (not web session replication) with Java EE containers like Glassfish. From what I have gleaned your remote interface is a proxy that delegates your call to one of many servers you may have in an environment. If things fail are… >>> More
Do i need a dedicated server for load balancing?

as seen on Server Fault - Search for 'Server Fault'
I'm completely new to the concept of load balancing so i hope this question isn't a "stupid question" because i've been searching around and im having a hard time understanding this. So to my understanding, in order to load balance, i need a separate machine with an ip address i can direct all traffic… >>> More
Load balancing and sessions

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi there, What is the better approach for load balancing on web servers? My services run in .NET and Mono, so they could be hosted on IIS or Apache2, and the will have to provide SSL connection. I've read two main approaches, store the state in a common server and use sticky sessions, there is any… >>> More
Loadbalancing outbound traffic while using openbgpd on freebsd

as seen on Server Fault - Search for 'Server Fault'
Hi, I am using openbgpd in freeBSD with 2 ISP connections. I have my own AS number and a /22 network. Currently I am advertising entire /22 to both networks. Inbound traffic comes in But my outbound traffic goes via a single link. I would like to either distribute my outbound traffic via both links… >>> More