SAS Expanders vs Direct Attached (SAS)?
- by jemmille
I have a storage unit with 2 backplanes. One backplane holds 24 disks, one backplane holds 12 disks. Each backplane is independently connected to a SFF-8087 port (4 channel/12Gbit) to the raid card.
Here is where my question really comes in. Can or how easily can a backplane be overloaded? All the disks in the machine are WD RE4 WD1003FBYX (black) drives that have average writes at 115MB/sec and average read of 125 MB/sec
I know things would vary based on the raid or filesystem we put on top of that but it seems to be that a 24 disk backplane with only one SFF-8087 connector should be able to overload the bus to a point that might actually slow it down?
Based on my math, if I had a RAID0 across all 24 disks and asked for a large file, I should, in theory should get 24*115 MB/sec wich translates to 22.08 GBit/sec of total throughput.
Either I'm confused or this backplane is horribly designed, at least in a perfomance environment.
I'm looking at switching to a model where each drive has it's own channel from the backplane (and new HBA's or raid card).
EDIT: more details
We have used both pure linux (centos), open solaris, software raid, hardware raid, EXT3/4, ZFS.
Here are some examples using bonnie++
4 Disk RAID-0, ZFS
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
194MB/s 19% 92MB/s 11% 200MB/s 8% 310/sec
194MB/s 19% 93MB/s 11% 201MB/s 8% 312/sec
--------- ---- --------- ---- --------- ---- ---------
389MB/s 19% 186MB/s 11% 402MB/s 8% 311/sec
8 Disk RAID-0, ZFS
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
324MB/s 32% 164MB/s 19% 346MB/s 13% 466/sec
324MB/s 32% 164MB/s 19% 348MB/s 14% 465/sec
--------- ---- --------- ---- --------- ---- ---------
648MB/s 32% 328MB/s 19% 694MB/s 13% 465/sec
12 Disk RAID-0, ZFS
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
377MB/s 38% 191MB/s 22% 429MB/s 17% 537/sec
376MB/s 38% 191MB/s 22% 427MB/s 17% 546/sec
--------- ---- --------- ---- --------- ---- ---------
753MB/s 38% 382MB/s 22% 857MB/s 17% 541/sec
Now 16 Disk RAID-0, it's gets interesting
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
359MB/s 34% 186MB/s 22% 407MB/s 18% 1397/sec
358MB/s 33% 186MB/s 22% 407MB/s 18% 1340/sec
--------- ---- --------- ---- --------- ---- ---------
717MB/s 33% 373MB/s 22% 814MB/s 18% 1368/sec
20 Disk RAID-0, ZFS
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
371MB/s 37% 188MB/s 22% 450MB/s 19% 775/sec
370MB/s 37% 188MB/s 22% 447MB/s 19% 797/sec
--------- ---- --------- ---- --------- ---- ---------
741MB/s 37% 376MB/s 22% 898MB/s 19% 786/sec
24 Disk RAID-1, ZFS
WRITE CPU RE-WRITE CPU READ CPU RND-SEEKS
347MB/s 34% 193MB/s 22% 447MB/s 19% 907/sec
347MB/s 34% 192MB/s 23% 446MB/s 19% 933/sec
--------- ---- --------- ---- --------- ---- ---------
694MB/s 34% 386MB/s 22% 894MB/s 19% 920/sec
28 Disk RAID-0, ZFS
32 Disk RAID-0, ZFS
36 Disk RAID-0, ZFS
More details:
Here is the exact unit:
http://www.supermicro.com/products/chassis/4U/847/SC847E1-R1400U.cfm
