While poking around trying to figure out why a Linux - Linux file transfer is running slower than I think it should, I stumbled across something I'm not familiar with. /dev/dm-0 seems to be my bottleneck, but I have no idea what it is.
On my destination server, the iostat command shows a device at the bottom, /dev/dm-0, as being 100% utilized. This server has 6 disks in a mdadm raid5 set, with LVM running on top of it. Each of the underlying disks are sitting around 50% util. The transfer is writing to a logical volume located on this raidset.
What is this /dev/dm-0 thing? Once I know what it is, maybe I can find how to increase its speed, or at least understand why its the speed that it is.