Cannot upload files bigger than 8GB to Amazon S3 by multi-part upload due to broken pipe
Posted
by
spencerho
on Server Fault
See other posts from Server Fault
or by spencerho
Published on 2011-03-11T21:59:47Z
Indexed on
2011/03/12
0:12 UTC
Read the original article
Hit count: 557
amazon-s3
|large-file-upload
I implemented S3 multi-part upload, both high level and low level version, based on the sample code from http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?HLuploadFileJava.html and http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?llJavaUploadFile.html
When I uploaded files of size less than 4 GB, the upload processes completed without any problem. When I uploaded a file of size 13 GB, the code started to show IO exception, broken pipes. After retries, it still failed.
Here is the way to repeat the scenario. Take 1.1.7.1 release,
- create a new bucket in US standard region
- create a large EC2 instance as the client to upload file
- create a file of 13GB in size on the EC2 instance.
- run the sample code on either one of the high-level or low-level API S3 documentation pages from the EC2 instance
- test either one of the three part size: default part size (5 MB) or set the part size to 100,000,000 or 200,000,000 bytes.
So far the problem shows up consistently. I attached here a tcpdump file for you to compare. In there, the host on the S3 side kept resetting the socket.
© Server Fault or respective owner