Distributed, Parallel, Fault-tolerant File System

Posted by Eddified on Server Fault See other posts from Server Fault or by Eddified
Published on 2009-11-05T00:55:12Z Indexed on 2010/06/09 16:42 UTC
Read the original article Hit count: 316

Filed under:

linux

|

filesystem

|

distributed

There are so many choices that it's hard to know where to start. My requirements are these:

Runs on Linux
Most of the files will be between 5-9 MB in size. There will also be a significant number of small-ish jpgs (100px x 100px).
All of the files need to be available over http.
Redundancy -- ideally it would provide the space efficiency similar to RAID 5 of 75% (in RAID 5 this would be calculated thus: with 4 identical disks, 25% of the space is used for parity => 75% efficent)
Must support several petabytes of data
scalable
runs on commodity hardware

In addition, I look for these qualities, though they are not "requirements":

Stable, mature file system
Lots of momentum and support
etc

I would like some input as to which file system works best for the given requirements. Some people at my organization are leaning towards MogileFS, but I'm not convinced of the stability and momentum of that project. GlusterFS and Lustre, based on my limited research, appear to be better supported...

Thoughts?

© Server Fault or respective owner

Related posts about linux

apt-get install and update fail

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I've got a problem with apt-get update and apt-get install ... commands . every time update or installing fails and errors are : Get:1 http://dl.google.com stable Release.gpg [198B] Ign http://dl.google.com/linux/chrome/deb/ stable/main Translation-en_US Get:2 http://dl… >>> More
kernel module compiling error

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
sh@ubuntu:/home/ccpp/helloworld$ make gcc-4.6 -O2 -DMODULE -D_KERNEL_ -W -Wall -Wstrict-prototypes -Wmissing-prototypes -isystem /lib/modules/`uname -r`/build/include -c -o hello-1.o hello-1.c hello-1.c:4:0: warning: "MODULE" redefined [enabled by default] <command-line>:0:0: note: this is… >>> More
Build-Essentials installation failing

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I am having trouble accessing the several critical header files that show to be a part of the build process. The "Ubuntu Software Center" shows "Build Essentials" as installed: Next I did the following two commands, which did not improve the problem: ~$ sudo apt-get install build-essential [sudo]… >>> More
Updating Debian kernel

as seen on Super User - Search for 'Super User'
I'm trying to update my Debian machine to 2.6.32-46 (which is the new stable). However, after doing apt-get update my apt-cache search linux-image shows me: linux-headers-2.6.32-5-486 - Header files for Linux 2.6.32-5-486 linux-headers-2.6.32-5-686-bigmem - Header files for Linux 2.6.32-5-686-bigmem linux-headers-2… >>> More
Serial connection over a single USB cable (Windows to linux, or linux to linux)

as seen on Server Fault - Search for 'Server Fault'
I'm helping out with a project for an embedded device that only has USB and no serial. This device is running Linux. These days, when we need to connect to a serial port on a device we typically use a USB to serial adapter (on something like a phone system or a load balancing device, etc). I would… >>> More

Related posts about filesystem

How to delete/edit files from readonly filesystem

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I am having problem with my memory device (actually a memory card that act external memory device like pendrive). experimentx@workmateX:/var/www/zendtest$ sudo rm /media/A88F-8788/python-2.7.1-docs-html.zip rm: cannot remove `/media/A88F-8788/python-2.7.1-docs-html.zip': Read-only file system I… >>> More
Open source embedded filesystem (or single file virtual filesystem, or structured storage) library f

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm not sure what the "general" name of something like this might be. I'm looking for a library that gives me a file format to store different types of binary data in an expanding single file. open source, non-GPL (LGPL ok) C interface the file format is a single file multiple files within using… >>> More
Wubi and Vista 64 bits can't work

as seen on Super User - Search for 'Super User'
First of all, I have posted this issue at Ubuntu Forum without success yet. Hello, I have downloaded "kubuntu-9.10-desktop-amd64.iso" and I have mounted it on my Windows Vista 64 bits Ultimate. I have downloaded wubi 9.10. The problem is when installing, it crash after few time. Here is the log… >>> More
Why didn't 12.04 install?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
Ok, so I've installed Ubuntu many times on my computer.. Normally on the same partition, and WIndows would always delete Ubuntu(I don't know how.. it just happens) if i go away from keyboard during boot and it chooses Windows automatically because I took to long. So i tried to reinstall again, but… >>> More
faking a filesystem / virtual filesystem

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a web service to which users upload python scripts that are run on a server. Those scripts process files that are on the server and I want them to be able to see only a certain hierarchy of the server's filesystem (best: a temporary folder on which I copy the files I want processed and the… >>> More