Why is piping dd through gzip so much faster than a direct copy?
- by Foo Bar
I wanted to backup a path from a computer in my network to another computer in the same network over a 100MBit/s line. For this I did
dd if=/local/path of=/remote/path/in/local/network/backup.img
which gave me a very low network transfer speed of something about 50 to 100 kB/s, which would have taken forever. So I stopped it and decided to try gzipping it on the fly to make it much smaller so that the amount to transfer is less. So I did
dd if=/local/folder | gzip > /remote/path/in/local/network/backup.img.gz
But now I get something like 1 MB/s network transfer speed, so a factor of 10 to 20 faster. After noticing this, I tested this on several paths and files and it was always the same.
Why does piping dd through gzip also increase the transfer rates by a large factor instead of only reducing the bytelength of the stream by a large factor? I'd expected even a small decrease in transfer rates instead, due to the higher CPU consumption while compressing, but now I get a double plus. Not that I'm not happy, but just wondering. ;)
