Our company has 100,000s+ photos, how to store and browse/find these efficiently?

Posted by tobefound on Server Fault See other posts from Server Fault or by tobefound
Published on 2010-12-10T13:49:48Z Indexed on 2011/01/04 19:55 UTC
Read the original article Hit count: 180

Filed under:

storage

|

nas

We currently store our photos in a structure like this:

folder\1\10000 - 19999.JPG|ORF|TIF (10 000 files)

folder\2\20000 - 29999.JPG|ORF|TIF (10 000 files) etc...

They are stored on 4 different 2TB D-link NASes attached and shared on our office network (\\nas1, \\nas2, and so on...)

Problems:

1) When a client (Windows only, Vista and 7) wishes to browse the let's say \\nas1\folder\1\ folder, performance is quite poor. A problem. List takes a long time to generate in explorer window. Even with icons turned off.

2) Initial access to the NAS itself is sometimes slow. Problem.

SAN disks too expensive for us. Even with iSCSI interface/switch technology.

I've read a lot of tech pages saying that storing 100 000+ files in one single folder shouldn't be a problem. But we don't dare go there now that we experience problems on a 10K level.

All input greatly appreciated,

/T

© Server Fault or respective owner

Related posts about storage

Oracle Storage Implementation Boot Camp: ZFS Storage Appliance and Flash

as seen on Oracle Blogs - Search for 'Oracle Blogs'
Oracle Storage Implementation Boot Camp: ZFS Storage Appliance and Flash Thursday 20th September 9.30 – 16.30 … >>> More
Slow NFS and GFS2 performance

as seen on Server Fault - Search for 'Server Fault'
Recently I've designed and configured a 4 node cluster for a webapp that does lots of file handling. The cluster have been broken down into 2 main roles, webserver and storage. Each role is replicated to a second server using drbd in active/passive mode. The webserver does a NFS mount of the data… >>> More
Moving data files failing

as seen on Super User - Search for 'Super User'
Trying to migrate data from C: to D: via the SBS console is failing. The wizard starts running but drops out in the first few seconds. I'll post the full logs, but the important lines appear to be as follows: An exception of type 'Type: System.IO.FileNotFoundException, mscorlib, Version=2… >>> More
Mounting Replicated Gluster Multi-AZ Storage

as seen on Server Fault - Search for 'Server Fault'
I have Replicated Gluster Storage which is used by Auto scaling Servers. Both, Auto scaling and Storage are allocated in two Availability zones. Gluster: Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: gluster01:/storage/1a # Zone A Brick2: gluster02:/storage/1b # Zone B Brick3:… >>> More
On Server Disk Storage VS SAN Storage

as seen on Server Fault - Search for 'Server Fault'
Hello, I am looking at buying three servers, and trying to figure out which storage solution makes the most sense in terms of performance and cost. Total budget is around: $10,000. OPTION 1: Dell servers with RAID 10 (4 Drives) each 7200RPM SAS 500GB, for a total capacity of 1TB. Each server is… >>> More

Related posts about nas

Backup NAS to another NAS?

as seen on Server Fault - Search for 'Server Fault'
hey there, is there a way to backup / transfer a NAS (it's like a big harddrive with some access to a webinterface, but closed linux etc.... only two ethernet ports) to another one? or is the only way to connect both NASs to my computer and copy the data from one to the other? it's like 750gb of… >>> More
Copying files from NAS to NAS drives

as seen on Super User - Search for 'Super User'
Very simple question. I've got 2 NAS drive that are "wire" connected via a router. If I have a wireless laptop and request a large amount of data be copied from one NAS drive to the other, does the network traffic go direct from the one drive to the other, going via the wired network, or does the… >>> More
cPanel web servers mounting home partition to a NAS or SAN

as seen on Server Fault - Search for 'Server Fault'
Hello, I currently have 2 cPanel web servers that are little 1RU dual cpu quad core xeons. They have a lot of resources for processing and handling web requests, and never exceed more than 10% cpu usage. They also have plenty of RAM. The problem is though that they both have RAID 1 160Gb SAS hard… >>> More
cant find nas mediaplayer on vista

as seen on Super User - Search for 'Super User'
i have a icy box ib mp3010hw media player i have it on wireless and have it connected to my tp link wr641g router and can access the radio on the net the problem i have is vista sp2 does not see it and i don't know how to go about making it find it. i ran the ping test i think it is by going through… >>> More
Qnap TS-809 NAS

as seen on Server Fault - Search for 'Server Fault'
I'm looking for a portable NAS solution for around 30 servers to back up to. I require 8-16TB of space. Does anyone have any experience of the Qnap TS-809 and know if it would be suitable for this many servers to be backing up to? >>> More