I have been doing lots of I/O testing on a ZFS system I will eventually use to serve virtual machines. I thought I would try adding SSD's for use as cache to see how much faster I can get the read speed. I also have 24GB of RAM in the machine that acts as ARC. vol0 is 6.4TB and the cache disks are 60GB SSD's. The zvol is as follows:

pool: vol0
 state: ONLINE
 scrub: none requested
config:

        NAME                     STATE     READ WRITE CKSUM
        vol0                     ONLINE       0     0     0
          c1t8d0                 ONLINE       0     0     0
        cache
          c3t5001517958D80533d0  ONLINE       0     0     0
          c3t5001517959092566d0  ONLINE       0     0     0

The issue is I'm not seeing any difference with the SSD's installed. I've tried bonnie++ benchmarks and some simple dd commands to write a file then read the file. I have run benchmarks before and after adding the SSD's.

I've ensured the file sizes are at least double my RAM so there is no way it can all get cached locally.

Am I missing something here? When am I going to see benefits of having all that cache? Am I simply not under these circumstances? Are the benchmark programs not good for testing the effect of cache because of the the way (and what) it writes and reads?