Homebrew large data cluster access for 2 user levels?

Posted by Yegor on Server Fault See other posts from Server Fault or by Yegor
Published on 2010-03-30T05:37:48Z Indexed on 2010/03/30 5:43 UTC
Read the original article Hit count: 310

Filed under:
|
|
|

The title probably makes little sense, so here is an example.

I have a file hosting site, that serves a large amount of semi-randomly accessed files.

The setup is as follows:

  • High horsepower front-end +DB server that also does encoding for files that need encoding
  • Fresh file server, which stores newly uploaded content, thats probably (and usually) rapidly accessible, which has 500GB of raided SSD storage, that can push over 3GBit of traffic.
  • 3 cheap node servers, containing 2 x 750GB SATA drives in raid1, where files older than 2 weeks are archived, from the SSD server (mentioned above).

Files on each server are accessed via subdomains (via modsec) in a straight forward fashion (server1.domain.com, server2.domain.com, etc)

Where I have the problem is this. I introduced a "premium" service where people pay a small fee every month, and get ad-free, quick accesses to stuff on the site. Once they are logged in, they access same files via premium.server1.domain.com via a different modsec script, with a different pass phrase. That all works fine and dandy.... except the cheap node servers are all IO bound, so accessing the files on them via a different, unsaturated network makes no difference, since it cannot read off the drive fast enough.

What would be a good way to make files on the site be accessible via 2 different network routes, 1 of which will be saturated (the "free network") while all other files are on an un-saturated "premium" network?

© Server Fault or respective owner

Related posts about file-hosting

Related posts about hosting