Rate limiting a ruby file stream
Posted
by Matthew Savage
on Stack Overflow
See other posts from Stack Overflow
or by Matthew Savage
Published on 2010-03-15T14:03:56Z
Indexed on
2010/03/15
17:19 UTC
Read the original article
Hit count: 350
I am working on a project which involves uploading flash video files to a S3 bucket from a number of geographically distributed nodes.
The video files are about 2-3mb each, and we are only sending one file (per node) every ten minutes, however the bandwidth we consume needs to be rate limited to ~20k/s, as these nodes are delivering streaming media to a CDN, and due to the locations we are only able to get 512k max upload.
I have been looking into the ASW-S3 gem and while it doesn't offer any kind of rate limiting I am aware that you can pass in a IO Stream. Given this I am wondering if it might be possible to create a rate-limited stream which overrides the read
method, adds in the rate limiting logic (e.g. in its simplest form a call to sleep
between reads) and then call out to the super of the overridden method.
Another option I considered is hacking the code for Net::HTTP and putting the rate limiting into the send_request_with_body_stream
method which is using a while
loop, but I'm not entirely sure which would be the best option.
I have attempted at extending the IO class, however that didn't work at all, simply inheriting from the class with class ThrottledIO < IO
didn't do anything.
Any suggestions will be greatly appreciated.
© Stack Overflow or respective owner