How to speed up rsync/tar of large Maildir

Posted by psusi on Ask Ubuntu See other posts from Ask Ubuntu or by psusi
Published on 2011-03-10T20:55:15Z Indexed on 2011/03/11 0:18 UTC
Read the original article Hit count: 247

Filed under:

filesystem

|

Performance

|

disk-management

I have a very large Maildir I am copying to a new machine ( over 100 BaseT ) with rsync. The progress is slow. VERY SLOW. Like 1 MB/s slow. I think this is because it is a lot of small files that are being read in an order that essentially is random with respect to where the blocks are stored on disk, causing a massive seek storm. I get similar results when trying to tar the directory. Is there a way to get rsync/tar to read in disk block order, or otherwise overcome this problem?

© Ask Ubuntu or respective owner

Related posts about filesystem

How to delete/edit files from readonly filesystem

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I am having problem with my memory device (actually a memory card that act external memory device like pendrive). experimentx@workmateX:/var/www/zendtest$ sudo rm /media/A88F-8788/python-2.7.1-docs-html.zip rm: cannot remove `/media/A88F-8788/python-2.7.1-docs-html.zip': Read-only file system I… >>> More
Open source embedded filesystem (or single file virtual filesystem, or structured storage) library f

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm not sure what the "general" name of something like this might be. I'm looking for a library that gives me a file format to store different types of binary data in an expanding single file. open source, non-GPL (LGPL ok) C interface the file format is a single file multiple files within using… >>> More
Wubi and Vista 64 bits can't work

as seen on Super User - Search for 'Super User'
First of all, I have posted this issue at Ubuntu Forum without success yet. Hello, I have downloaded "kubuntu-9.10-desktop-amd64.iso" and I have mounted it on my Windows Vista 64 bits Ultimate. I have downloaded wubi 9.10. The problem is when installing, it crash after few time. Here is the log… >>> More
Why didn't 12.04 install?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
Ok, so I've installed Ubuntu many times on my computer.. Normally on the same partition, and WIndows would always delete Ubuntu(I don't know how.. it just happens) if i go away from keyboard during boot and it chooses Windows automatically because I took to long. So i tried to reinstall again, but… >>> More
faking a filesystem / virtual filesystem

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a web service to which users upload python scripts that are run on a server. Those scripts process files that are on the server and I want them to be able to see only a certain hierarchy of the server's filesystem (best: a temporary folder on which I copy the files I want processed and the… >>> More

Related posts about Performance

Improving VPN performance - stronger encryption = more performance?

as seen on Server Fault - Search for 'Server Fault'
I have a site-to-site VPN set up with two SonicWall's (a TZ170 and a Pro1260). It was suggested to me that turning off encryption (so the VPN is tunneling only) would improve performance. (I'm not concerned with security, because the VPN is running over a trusted line.) Using FTP and HTTP transfers… >>> More
Inaccurate performance counter timer values in Windows Performance Monitor

as seen on Stack Overflow - Search for 'Stack Overflow'
I am implementing instrumentation within an application and have encountered an issue where the value that is displayed in Windows Performance Monitor from a PerformanceCounter is incongruent with the value that is recorded. I am using a Stopwatch to record the duration of a method execution, then… >>> More
Excel-based Performance Reviews transformed into Web Application for Performance Management

as seen on Geeks with Blogs - Search for 'Geeks with Blogs'
HR TMS provides enterprise talent management solutions for healthcare, retail and corporate customers, focusing on performance management, compensation management and succession planning. As the competency of nurses and other healthcare workers is critical, the government, via the Joint Commission… >>> More
How to save a perfmon Performance Counter as a textfile (Reliability and Performance Monitor Version

as seen on Server Fault - Search for 'Server Fault'
Now the file gets saved as blg, but I would like a txt versin to import in Excel. >>> More
SQLAuthority News – A Successful Performance Tuning Seminar at Pune – Dec 4-5, 2010

as seen on SQL Authority - Search for 'SQL Authority'
This is report to my third of very successful seminar event on SQL Server Performance Tuning. SQL Server Performance Tuning Seminar in Colombo was oversubscribed with total of 35 attendees. You can read the details over here SQLAuthority News – SQL Server Performance Optimizations Seminar – Grand… >>> More