Multi Thread Rsync Transfer

Posted by reefine on Server Fault See other posts from Server Fault or by reefine
Published on 2012-10-04T07:42:57Z Indexed on 2012/10/04 9:40 UTC
Read the original article Hit count: 377

Filed under:

rsync

For some reason when running a single rsync command I am getting 1 MB/sec to 2 MB/sec even when I connecting 2 servers both connected to 1 Gbps ports.

rsync -v --progress -e ssh /backup/mysqldata/mysql-bin.000199 [email protected]:/secondary/mysqldata/mysqldata/mysql-bin.000199

I have over 800 GB of data to transfer split among 500 or so files all starting with:

mysql-bin.000*

I've found that running 25-30 rsync simultaneously from seperate SSH windows gets me upwards of 25 MB/sec but it will take me hours to run these all manually.

Is there anyway to get the 25 MB/sec from a single rsync command?

Related posts about rsync

Rsync: how to mount truecrypt on-the-fly on the receiving side?

as seen on Super User - Search for 'Super User'
The short version: how can I keep an rsync backup on a truecrypt volume? The hard part is to mount/unmount this volume on the fly when it is needed for rsync. Details This is my current backup configuration (which works fairly well for the most part): backup source is on Win7 64 bit, destination… >>> More
Two Questions on for Rsync - rsync by date and by file name

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I have two questions with respect to rsync: 1: I have a bunch of files which are incremented by day of the year. Ex: file.txt.81, file.txt.82, etc. Now, these files are in different directories: data1/file.txt.81 data1/file.txt.82 data2/file2.txt.81 data2/file2.txt.82 How can I have rsync… >>> More
Cygwin rsync broken

as seen on Super User - Search for 'Super User'
I get an error with cygwin rsync trying to transfer files between local - remote host. Any ideas? C:\>rsync user@host:~/file newfile Password: rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender= 3… >>> More
macport selfupdate not working

as seen on Super User - Search for 'Super User'
macbookpro:~ eistrati$ port -v MacPorts 2.1.2 macbookpro:~ eistrati$ xcodebuild -version Xcode 4.5.2 Build version 4G2008a macbookpro:~ eistrati$ sudo port -d selfupdate DEBUG: Copying /Users/eistrati/Library/Preferences/com.apple.dt.Xcode.plist to /opt/local/var/macports/home/Library/Preferences DEBUG:… >>> More
Rsync from godaddy to OS X

as seen on Super User - Search for 'Super User'
I would like to use rsync to backup my website to my local computer (OS X). I started of with this guide and got pretty far. I use the following rsync-line: rsync -PzrlptgD --del --delete-excluded -r --rsync-path=~/bin/rsync user@server:~/ /local/backup/folder/ I wanted to use the -a option (same… >>> More

Developer IT

Multi Thread Rsync Transfer - Developer IT

Multi Thread Rsync Transfer

rsync

Related posts about rsync

Rsync: how to mount truecrypt on-the-fly on the receiving side?

Two Questions on for Rsync - rsync by date and by file name

Cygwin rsync broken

macport selfupdate not working

Rsync from godaddy to OS X

Categories cloud