I am uploading files to Amazon S3 bucket. The files are being uploaded but i get the following Warning.
WARNING: No content length specified for stream data. Stream contents will be buffered in memory and could result in out of memory errors.
So I added the following line to my code
metaData.setContentLength(IOUtils.toByteArray(input).length);
but then i got the following message. I don't even know if it is a warning or what.
Data read has a different length than the expected: dataLength=0; expectedLength=111992; includeSkipped=false; in.getClass()=class sun.net.httpserver.FixedLengthInputStream; markedSupported=false; marked=0; resetSinceLastMarked=false; markCount=0; resetCount=0
How can i set contentLength to the metaData of InputSteam? Any help would be greatly appreciated.
When you read the data with IOUtils.toByteArray
, this consumes the InputStream. When the AWS API tries to read it, it's zero length.
Read the contents into a byte array and provide an InputStream wrapping that array to the API:
byte[] bytes = IOUtils.toByteArray(input);
metaData.setContentLength(bytes.length);
ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes);
PutObjectRequest putObjectRequest = new PutObjectRequest(bucket, key, byteArrayInputStream, metadata);
client.putObject(putObjectRequest);
You should consider using the multipart upload API to avoid loading the whole InputStream into memory. For example:
byte[] bytes = new byte[BUFFER_SIZE];
String uploadId = client.initiateMultipartUpload(new InitiateMultipartUploadRequest(bucket, key)).getUploadId();
int bytesRead = 0;
int partNumber = 1;
List<UploadPartResult> results = new ArrayList<>();
bytesRead = input.read(bytes);
while (bytesRead >= 0) {
UploadPartRequest part = new UploadPartRequest()
.withBucketName(bucket)
.withKey(key)
.withUploadId(uploadId)
.withPartNumber(partNumber)
.withInputStream(new ByteArrayInputStream(bytes, 0, bytesRead))
.withPartSize(bytesRead);
results.add(client.uploadPart(part));
bytesRead = input.read(bytes);
partNumber++;
}
CompleteMultipartUploadRequest completeRequest = new CompleteMultipartUploadRequest()
.withBucketName(bucket)
.withKey(key)
.withUploadId(uploadId)
.withPartETags(results);
client.completeMultipartUpload(completeRequest);