Synchronize large objects to S3 efficiently

Posted by emk on Server Fault See other posts from Server Fault or by emk
Published on 2011-02-04T15:03:39Z Indexed on 2011/02/04 15:27 UTC
Read the original article Hit count: 264

Filed under:

backup

|

synchronization

|

amazon-s3

I need to synchronize about 30GB of git repositories to S3. These repos may contain some very large pack files, on the rough order of 2GB.

I know that S3 has recently added support for large objects, and has new APIs that allow the objects to be uploaded as several parallel chunks. Is there a good command-line tool for Linux that allows me to efficiently synchronize large objects with S3 in a fashion similar to s3sync?

© Server Fault or respective owner

Related posts about backup

backup exec - backup to disk offline

as seen on Server Fault - Search for 'Server Fault'
Hi We are running backup exec 9.1 doing a backup to disk to portable hard disk drives. When we run the backup manually it works fine. But when the backup is setup to run in the evening on a schedule it does not run as the backup to disk folders goes offline and therefore has to be switched back… >>> More
Ideal backup appliance for backup software like Bacula?

as seen on Server Fault - Search for 'Server Fault'
I'm at a small company and we (the IT department of two) manage <100 client computers and a handful of servers. Currently we're using a company's appliance to handle backup; it does a small backup every night and a full backup every weekend, and a guy comes on Wednesday to take an offsite backup… >>> More
Symantec Backup Exec Error on backup

as seen on Server Fault - Search for 'Server Fault'
Recently we have moved some of servers from real servers- into virtual servers. Since then, we are getting errors like the following: Error category : Resource ErrorsError : e000fed1 - A failure occurred querying the Writer status. For additional information regarding this error refer… >>> More
Windows Server Backup - Recover only shows the latest backup

as seen on Server Fault - Search for 'Server Fault'
We're having quite some trouble at work using Windows Server Backup. We have a HyperV server (Win 2008) running 8 virtual web servers, these are running a variety of OS'es: Win 2003, Win 2008 and a lone Debian. Each virtual server has a separate partition on the physical HyperV server, so e.g. E:… >>> More
Failed Backup Job With Backup Exec 12 and AOFO

as seen on Server Fault - Search for 'Server Fault'
I am backing up a Windows 2003 Small Business Server with SP2. We are running Backup Exec 12 with SP4. Recently the backup job started failing on backing up the system state with the following error: V-79-57344-34110 - AOFO: Initialization failure on: "System?State". Advanced Open File Option… >>> More

Related posts about synchronization

Thread Synchronization and Synchronization Primitives

as seen on DotNetBlocks - Search for 'DotNetBlocks'
When considering synchronization in an application, the decision truly depends on what the application and its worker threads are going to do. I would use synchronization if two or more threads could possibly manipulate the same instance of an object at the same time. An example of this in C# can… >>> More
Inner synchronization on the same object as the outer synchronization

as seen on Stack Overflow - Search for 'Stack Overflow'
Recently I attended a lecture concerning some design patterns: The following code had been displayed: public static Singleton getInstance() { if (instance == null) { synchronized(Singleton.class) { //1 Singleton inst = instance; //2 if (inst == null) { … >>> More
SugarSync: Online Backup and Synchronization

as seen on Internet.com - Search for 'Internet.com'
Despite a few flaws, SugarSync offers small businesses a way to synchronize files across multiple computers and smartphones. Plus, it includes online data backup. >>> More
SugarSync: Online Backup and Synchronization

as seen on Internet.com - Search for 'Internet.com'
Despite a few flaws, SugarSync offers small businesses a way to synchronize files across multiple computers and smartphones. Plus, it includes online data backup. >>> More
Send regular keyboard samples OR keyboard state changes over network

as seen on Game Development - Search for 'Game Development'
Building a multi player asteroids game where ships compete with each other. Using UDP. Wanted to minimize traffic sent to server. Which would you do: Send periodic keyboard state samples every from client every to match server physics update rate e.g. 50 times per second. Highly resilient to packet… >>> More