Moving a lot of small files between servers using rsync

Posted by Adirael on Server Fault See other posts from Server Fault or by Adirael
Published on 2011-02-13T22:23:05Z Indexed on 2011/02/13 23:26 UTC
Read the original article Hit count: 240

Filed under:
|
|

Hello guys,

I'm moving a lot of files (about 2 millions) between two servers on different locations using rsync over ssh, it seems to work fine but I just realised I'm losing some files on the process.

I got server 1, with the original data, and server 2, with the copy. Server 1 runs CentOS 5 and Server 2 runs on Ubuntu 10.

I'm doing the transfer on the Server's 2 command line like this: rsync -e ssh -avzn usr@server1:/remote/path /local/path

The first file movement I did using tar, but I didn't though of piping it through ssh and it failed cause the disk on server 1 was almost full, so I transfered it anyways (it was about 200GB) and got about 80% of the files.

Then I piped another tar with the rest of the files (they're in folders, I got 100 folders with about 30 subfolders each, with files inside) and now I got everything on server 2.

I wanted to be sure, so I my two options are getting the md5sum of all the files and check them or running an rsync on server 2 against server 1, that's what I did. It got some missing stuff and now it says there's nothing more to do (DRY RUN).

But I got at least two files that are missing inside a subfolder. I ran that same rsync on that folder, but still dry run.

Am I doing something wrong?

Thanks, and sorry for the wall of text.

© Server Fault or respective owner

Related posts about linux

Related posts about ssh