Fast distributed filesystem for a large amounts of data with metadata in database
Posted
by
undefined hero
on Server Fault
See other posts from Server Fault
or by undefined hero
Published on 2014-08-12T11:59:46Z
Indexed on
2014/08/18
22:23 UTC
Read the original article
Hit count: 337
My project uses several processing machines and one storage machine. Currently storage organized with a MSSQL filetable shared folder. Every file in storage have some metadata in database.
Processing machines executes tasks for which they needed files from storage and their metadata. After completing task, processing machine puts resulting data back in storage. From there its taken by another processing machine, which also generates some file and put it back in storage. And etc.
Everything was fine, but as number of processing machines increases, I found myself bottlenecked myself with storage machines hard drive performance.
So I want processing machines to put files in distributed FS. to lift load from storage machines, from which they can take data from each other, not only storage machine.
Can You suggest a particular distributed FS which meets my needs? Or there is another way to solve this problem, without it?
Amounts of data in FS in one time are like several terabytes. (storage can handle this, but processors cannot). Data consistence is critical. Read write policy is: once file is written - its constant and may be only removed, but not modified.
My current platform is Windows, but I'm ready to switch it, if there is a substantially more convenient solution on another one.
© Server Fault or respective owner