Fast (non-blocking) way to transfer many files to another server

Posted by Nyxynyx on Server Fault See other posts from Server Fault or by Nyxynyx
Published on 2012-10-10T14:55:29Z Indexed on 2012/10/10 15:39 UTC
Read the original article Hit count: 280

Filed under:

linux

|

ubuntu

|

centos

|

ftp

|

wget

I am currently attempting to transfer over 1 million files from one server to another. Using wget, it seems to be extremely slow, probably because it starts a new transfer after the previous one has been completed.

Question: Is there a faster non-blocking (asynchronous) way to do the transfer? I do not have enough space on the first server to compress the files into tar.gz and transferring them over. Thanks!

© Server Fault or respective owner

Related posts about linux

apt-get install and update fail

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I've got a problem with apt-get update and apt-get install ... commands . every time update or installing fails and errors are : Get:1 http://dl.google.com stable Release.gpg [198B] Ign http://dl.google.com/linux/chrome/deb/ stable/main Translation-en_US Get:2 http://dl… >>> More
kernel module compiling error

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
sh@ubuntu:/home/ccpp/helloworld$ make gcc-4.6 -O2 -DMODULE -D_KERNEL_ -W -Wall -Wstrict-prototypes -Wmissing-prototypes -isystem /lib/modules/`uname -r`/build/include -c -o hello-1.o hello-1.c hello-1.c:4:0: warning: "MODULE" redefined [enabled by default] <command-line>:0:0: note: this is… >>> More
Build-Essentials installation failing

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I am having trouble accessing the several critical header files that show to be a part of the build process. The "Ubuntu Software Center" shows "Build Essentials" as installed: Next I did the following two commands, which did not improve the problem: ~$ sudo apt-get install build-essential [sudo]… >>> More
Updating Debian kernel

as seen on Super User - Search for 'Super User'
I'm trying to update my Debian machine to 2.6.32-46 (which is the new stable). However, after doing apt-get update my apt-cache search linux-image shows me: linux-headers-2.6.32-5-486 - Header files for Linux 2.6.32-5-486 linux-headers-2.6.32-5-686-bigmem - Header files for Linux 2.6.32-5-686-bigmem linux-headers-2… >>> More
Serial connection over a single USB cable (Windows to linux, or linux to linux)

as seen on Server Fault - Search for 'Server Fault'
I'm helping out with a project for an embedded device that only has USB and no serial. This device is running Linux. These days, when we need to connect to a serial port on a device we typically use a USB to serial adapter (on something like a phone system or a load balancing device, etc). I would… >>> More

Related posts about ubuntu

Cannot update, apt-get cannot fetch index files

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I have a fresh install of Ubuntu 11.10 from the iso 'ubuntu-11.10-desktop-amd64.iso'. I installed this in VMWare Fusion 4.1.1 running on OSX 10.7.3. When setting up the VM, I allowed easy install to take care of creating my user and installing VMWare tools. No problems during installation, everything… >>> More
'sudo apt-get update' error on Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
When I type in 'sudo apt-get update' I get this mohd-arafat-hossain@TUD:~$ sudo apt-get update [sudo] password for mohd-arafat-hossain: Ign http://bd.archive.ubuntu.com precise InRelease Ign http://bd.archive.ubuntu.com precise-updates InRelease Ign http://bd.archive.ubuntu… >>> More
Opening Skype, Opera, OpenOffice logs me off

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
Whats common among Skype, Opera, OpenOffice in Ubuntu ? Whenever I open these applications I get logged off and shows back me the login screen. This started happening since the 10.10 upgrade. Forgot to mention : Yes, its x64.Each time I open these applications, the UI shows and then crashes. I… >>> More
Update information outdated

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I have a warning triangle that my update information is outdated, the last update was 12 days ago. I use Ubuntu 11.10. A run of sudo apt-get update produces the following output: Ign http://ppa.launchpad.net oneiric InRelease Ign http://de.archive.ubuntu.com oneiric InRelease … >>> More
ubuntu/apt-get update said "Failed to Fetch http:// .... 404 not found"

as seen on Server Fault - Search for 'Server Fault'
Hi all, I'm trying to run apt-get update on ubuntu 9.10 I've configured my proxy server and I can access the internet without any problem: /etc/apt# wget "http://www.google.com" Resolving (...) Proxy request sent, awaiting response... 200 OK Length: 292 [text/html] Saving to: `index.html' 100%[=================================================================================================================================>]… >>> More