optimal folder structure for storing 100k files on a USB drive

Posted by cherouvim on Server Fault See other posts from Server Fault or by cherouvim
Published on 2010-12-29T13:52:00Z Indexed on 2010/12/29 13:55 UTC
Read the original article Hit count: 240

Filed under:
|
|
|
|

I need to store 100k files (around 40GB) in a USB drive. Each file has a unique int id (e.g 45000).

Option one is to put all files in a single folder:

root/
root/1.pdf
root/2.pdf
root/3.pdf
...
root/567.pdf
root/568.pdf
root/569.pdf
...
root/10001.pdf
root/10002.pdf
root/10003.pdf
...
root/99998.pdf
root/99999.pdf
root/100000.pdf

Option two is to create a [1-9][0-9]* folder hierarchy based on that id:

root/
root/1/file.pdf
root/2/file.pdf
root/3/file.pdf
...
root/5/6/7/file.pdf
root/5/6/8/file.pdf
root/5/6/9/file.pdf
...
root/1/0/0/0/1/file.pdf
root/1/0/0/0/2/file.pdf
root/1/0/0/0/3/file.pdf
...
root/9/9/9/9/8/file.pdf
root/9/9/9/9/9/file.pdf
root/1/0/0/0/0/0/file.pdf

Which option will scale better? I can understand that the second option will require tons of folders but each folder will at most contain 10 folders and 1 file. Maintenance will not be an issue since everything will be controlled by an application.

Note that this is a USB drive on linux and based on the above I'd also like to know whether I should go with FAT32 or NTFS.

© Server Fault or respective owner

Related posts about linux

Related posts about filesystems