best filesystem for an aws s3 like service

Posted by gucki on Server Fault See other posts from Server Fault or by gucki
Published on 2011-03-10T16:52:51Z Indexed on 2011/03/11 0:12 UTC
Read the original article Hit count: 468

Filed under:

storage

Hi!

I need to build a fault tolerant, highly available key/value storage (no posix, only same functionaluty as S3) using cheap existing hardware. The storage should be able to handle several billions of items. The maximum size of items is around 1GB, most are only several KB. What's the best software/ filesystem for this task?

I already had a brief look at mogilefs, mongodb (grid-fs) & glusterfs but I'm not really sure which is stable & fault tolerant enough. The simpler the setup and later expansion the better :).

Corin

Related posts about storage

Oracle Storage Implementation Boot Camp: ZFS Storage Appliance and Flash

as seen on Oracle Blogs - Search for 'Oracle Blogs'
Oracle Storage Implementation Boot Camp: ZFS Storage Appliance and Flash Thursday 20th September 9.30 – 16.30 … >>> More
Slow NFS and GFS2 performance

as seen on Server Fault - Search for 'Server Fault'
Recently I've designed and configured a 4 node cluster for a webapp that does lots of file handling. The cluster have been broken down into 2 main roles, webserver and storage. Each role is replicated to a second server using drbd in active/passive mode. The webserver does a NFS mount of the data… >>> More
Moving data files failing

as seen on Super User - Search for 'Super User'
Trying to migrate data from C: to D: via the SBS console is failing. The wizard starts running but drops out in the first few seconds. I'll post the full logs, but the important lines appear to be as follows: An exception of type 'Type: System.IO.FileNotFoundException, mscorlib, Version=2… >>> More
Mounting Replicated Gluster Multi-AZ Storage

as seen on Server Fault - Search for 'Server Fault'
I have Replicated Gluster Storage which is used by Auto scaling Servers. Both, Auto scaling and Storage are allocated in two Availability zones. Gluster: Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: gluster01:/storage/1a # Zone A Brick2: gluster02:/storage/1b # Zone B Brick3:… >>> More
On Server Disk Storage VS SAN Storage

as seen on Server Fault - Search for 'Server Fault'
Hello, I am looking at buying three servers, and trying to figure out which storage solution makes the most sense in terms of performance and cost. Total budget is around: $10,000. OPTION 1: Dell servers with RAID 10 (4 Drives) each 7200RPM SAS 500GB, for a total capacity of 1TB. Each server is… >>> More

Developer IT