s3cmd fails too many times

Posted by alfish on Server Fault See other posts from Server Fault or by alfish
Published on 2011-11-12T23:41:05Z Indexed on 2011/11/13 1:54 UTC
Read the original article Hit count: 643

Filed under:
|
|

It used to be my favorite backup transport agent but now I frequently get this result from s3cmd on the very same Ubuntu server/network:

root@server:/home/backups# s3cmd put bkup.tgz s3://mybucket/
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      36864 of 2711541519     0% in    1s    20.95 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      36864 of 2711541519     0% in    1s    23.96 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      28672 of 2711541519     0% in    1s    18.71 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      28672 of 2711541519     0% in    1s    18.86 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.25)
WARNING: Waiting 12 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      28672 of 2711541519     0% in    1s    15.79 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=1.25)
WARNING: Waiting 15 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      12288 of 2711541519     0% in    2s     4.78 kB/s  failed
ERROR: Upload of 'bkup.tgz' failed too many times. Skipping that file.

This happens even for files as small as 100MB, so I suppose it's not a size issue. It also happens when I use put with --acl-private flag (s3cmd version 1.0.1)

I appreciate if you suggest some solution or a lightweight alternative to s3cmd.

Thanks

© Server Fault or respective owner

Related posts about linux

Related posts about backup