Cannot upload files bigger than 8GB to Amazon S3 by multi-part upload due to broken pipe

Posted by spencerho on Server Fault See other posts from Server Fault or by spencerho
Published on 2011-03-11T21:59:47Z Indexed on 2011/03/12 0:12 UTC
Read the original article Hit count: 557

I implemented S3 multi-part upload, both high level and low level version, based on the sample code from http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?HLuploadFileJava.html and http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?llJavaUploadFile.html

When I uploaded files of size less than 4 GB, the upload processes completed without any problem. When I uploaded a file of size 13 GB, the code started to show IO exception, broken pipes. After retries, it still failed.

Here is the way to repeat the scenario. Take 1.1.7.1 release,

  1. create a new bucket in US standard region
  2. create a large EC2 instance as the client to upload file
  3. create a file of 13GB in size on the EC2 instance.
  4. run the sample code on either one of the high-level or low-level API S3 documentation pages from the EC2 instance
  5. test either one of the three part size: default part size (5 MB) or set the part size to 100,000,000 or 200,000,000 bytes.

So far the problem shows up consistently. I attached here a tcpdump file for you to compare. In there, the host on the S3 side kept resetting the socket.

© Server Fault or respective owner

Related posts about amazon-s3

Related posts about large-file-upload