How to organize a large number of objects

Posted by shane on Stack Overflow See other posts from Stack Overflow or by shane
Published on 2010-04-29T02:35:38Z Indexed on 2010/04/29 2:37 UTC
Read the original article Hit count: 520

Filed under:

files

|

best

|

best-practices

We have a large number of documents and metadata (xml files) associated with these documents. What is the best way to organize them?

Currently we put them into a series of nested folders:

/repository/category/date(when they were loaded into our db)/document_number.pdf and .xml

We use the path as a unique identifier for the document in our system. This is more versatile than putting them all in a single flat folder. also it is independent from our database/application, so we can reload them in case of failure. Yet, it introduces some limitations. for example we can't move the files once they've been placed in this structure, also it takes work to put them this way. What is the best practice? How websites such as Scribd deal with this problem?

© Stack Overflow or respective owner

Related posts about files

Unzip individual files from multiple zip files and extract those individual files to home directory

as seen on Super User - Search for 'Super User'
I would like to unzip individual files. These files have a .txt extension. These files also live within multiple zipped files. Here is the command I'm trying to use. unzip -jn /path/to/zipped/files/zipArchiveFile2011\*.zip /path/to/specific/individual/files/myfiles2011*.txt -d /path/to/home/directory/for/extract/ From… >>> More
Storing image files, psd files, ai files, flash in subversion

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, Can I store large amounts of image files in subversion. My designers usually create these designs and store them anywhere on their pc and there's no system. Can I store the files in an svn repository. That way I can also protect my data against unauthorized access and its also easier to archive… >>> More
Copy files in Linux, avoid the copy if files do exist in destination

as seen on Super User - Search for 'Super User'
Hi, I need to copy a /home/user folder from one hardisk to another one. It has 100000 files and around 10G size. I use cp -r /origin /destination sometines I get some errors due to broken links, permissions and so on. So I fix the error, and need to start again the copy. I wonder how could… >>> More
Best way to convert pdf files to tiff files

as seen on Stack Overflow - Search for 'Stack Overflow'
I have around 1000 pdf filesand I need to convert them to 300 dpi tiff files. What is the best way to do this? If there is an SDK or something or a tool that can be scripted that would be ideal. >>> More
what is the difference between ELF files and bin files

as seen on Stack Overflow - Search for 'Stack Overflow'
The final images produced by compliers contain both bin file and extended loader format ELf file ,what is the difference between too , especially the utility of ELF file. >>> More

Related posts about best

Oracle WebCenter: The Best of the Best

as seen on Oracle Blogs - Search for 'Oracle Blogs'
You may remember that the key goals of the new release of WebCenter are providing a Modern User Experience, unparalleled Application Integration, converging all the best of the existing portal platforms into WebCenter and delivering a Common User Experience Architecture. Last week, we provided… >>> More
Best Upper Bound & Best Lower Bound of an Algorithm

as seen on Programmers - Search for 'Programmers'
I am studying for a final exam and I came past a question I had on an earlier test. The questions asks us to find the minimum value in an unsorted array of integers. We must provide the best upper bound and the best lower bound that you can for the problem in the worst case. First, in such an example… >>> More
Best way to compress avi files for web viewing and best format for view

as seen on Super User - Search for 'Super User'
What is the best tool for reducing the file size of AVI files to post for viewing on the web? Also what is the recommended format? >>> More
Are there any good references coparing Software Development CM best practices to IT CM best practice

as seen on Server Fault - Search for 'Server Fault'
I have spent my career on the software development side of things and in the latter part have become more and more involved in the realm of Software Configuration Management. Now I am moving into an IT group and need to ramp up on CM practices from that standpoint. Are there any good references (books… >>> More
Best online photoeditor & Best iphone wireframe Creator

as seen on Stack Overflow - Search for 'Stack Overflow'
Is there any Best online photo editor & Best iphone wireframe Creator? iphone wireframe creator that is useful to draw preview of any iphone application. >>> More