I ran a NAS/SAN box on Solaris 11 Express before Solaris 11 was released. The box is an HP X1600 with an attached D2700. In all, 12x 1TB 7200 SATA disks, 12x 300GB 10k SAS disks in separate zpools. Total RAM is 30GB. Services provided are CIFS, NFS and iSCSI.

All was well, and I had a ZFS memory usage graph looking like this:

A fairly healthy Arc size of around 23GB - making use of the available memory for caching.

However, I then upgraded to Solaris 11 when that came out. Now, my graph looks like this:

Partial output of arc_summary.pl is:

System Memory:
     Physical RAM:  30701 MB
     Free Memory :  26719 MB
     LotsFree:      479 MB

ZFS Tunables (/etc/system):

ARC Size:
     Current Size:             915 MB (arcsize)
     Target Size (Adaptive):   119 MB (c)
     Min Size (Hard Limit):    64 MB (zfs_arc_min)
     Max Size (Hard Limit):    29677 MB (zfs_arc_max)

It's targetting 119MB while sitting at 915MB. It's got 30GB to play with. Why? Did they change something?

Edit

To clarify, arc_summary.pl is Ben Rockwood's, and the relevent lines generating the above stats are:

my $mru_size = ${Kstat}->{zfs}->{0}->{arcstats}->{p};
my $target_size = ${Kstat}->{zfs}->{0}->{arcstats}->{c};
my $arc_min_size = ${Kstat}->{zfs}->{0}->{arcstats}->{c_min};
my $arc_max_size = ${Kstat}->{zfs}->{0}->{arcstats}->{c_max};
my $arc_size = ${Kstat}->{zfs}->{0}->{arcstats}->{size};

The Kstat entries are there, I'm just getting odd values out of them.

Edit 2

I've just re-measured the arc size with arc_summary.pl - I've verified these numbers with kstat:

System Memory:
     Physical RAM:  30701 MB
     Free Memory :  26697 MB
     LotsFree:      479 MB

ZFS Tunables (/etc/system):

ARC Size:
     Current Size:             744 MB (arcsize)
     Target Size (Adaptive):   119 MB (c)
     Min Size (Hard Limit):    64 MB (zfs_arc_min)
     Max Size (Hard Limit):    29677 MB (zfs_arc_max)

The thing that strikes me is that the Target Size is 119MB. Looking at the graph, it's targeted the exact same value (124.91M according to cacti, 119M according to arc_summary.pl - think the difference is just 1024/1000 rounding issues) ever since Solaris 11 was installed. It looks like the kernel's making zero effort to shift the target size to anything different. The current size is fluctuating as the needs of the system (large) fight with the target size, and it appears equilibrium is between 700 and 1000MB.

So the question is now a little more pointed - why is Solaris 11 hard setting my ARC target size to 119MB, and how do I change it? Should I raise the min size to see what happens?

I've stuck the output of kstat -n arcstats over at http://pastebin.com/WHPimhfg

Edit 3

Ok, weirdness now. I know flibflob mentioned that there was a patch to fix this. I haven't applied this patch yet (still sorting out internal support issues) and I've not applied any other software updates.

Last thursday, the box crashed. As in, completely stopped responding to everything. When I rebooted it, it came back up fine, but here's what my graph now looks like.

It seems to have fixed the problem.

This is proper la la land stuff now. I've literally no idea what's going on. :(