Why is piping dd through gzip so much faster than a direct copy?
Posted
by
Foo Bar
on Super User
See other posts from Super User
or by Foo Bar
Published on 2014-05-29T08:35:55Z
Indexed on
2014/05/29
9:32 UTC
Read the original article
Hit count: 212
I wanted to backup a path from a computer in my network to another computer in the same network over a 100MBit/s line. For this I did
dd if=/local/path of=/remote/path/in/local/network/backup.img
which gave me a very low network transfer speed of something about 50 to 100 kB/s, which would have taken forever. So I stopped it and decided to try gzipping it on the fly to make it much smaller so that the amount to transfer is less. So I did
dd if=/local/folder | gzip > /remote/path/in/local/network/backup.img.gz
But now I get something like 1 MB/s network transfer speed, so a factor of 10 to 20 faster. After noticing this, I tested this on several paths and files and it was always the same.
Why does piping dd
through gzip
also increase the transfer rates by a large factor instead of only reducing the bytelength of the stream by a large factor? I'd expected even a small decrease in transfer rates instead, due to the higher CPU consumption while compressing, but now I get a double plus. Not that I'm not happy, but just wondering. ;)
© Super User or respective owner