Ggtcf

I am very much new to AWS S3 and trying to upload large file by chunked process. From UI, i am sending the file's chunked data(blob) to WCF service from where i will upload it to S3 using MultiPartAPI. Note that, the file can be in GB's. That's why i am making chunks of the file and uploading it to S3.

public UploadPartResponse UploadChunk(Stream stream, string fileName, string uploadId, List<PartETag> eTags, int partNumber, bool lastPart)

 stream.Position = 0; // Throwing Exceptions

 //Step 1: build and send a multi upload request
 if (partNumber == 1)
 
 var initiateRequest = new InitiateMultipartUploadRequest
 
 BucketName = _settings.Bucket,
 Key = fileName
 ;

 var initResponse = _s3Client.InitiateMultipartUpload(initiateRequest);
 uploadId = initResponse.UploadId;
 

 //Step 2: upload each chunk (this is run for every chunk unlike the other steps which are run once)
 var uploadRequest = new UploadPartRequest
 
 BucketName = _settings.Bucket,
 Key = fileName,
 UploadId = uploadId,
 PartNumber = partNumber,
 InputStream = stream,
 IsLastPart = lastPart,
 PartSize = stream.Length // Throwing Exceptions
 ;

 var response = _s3Client.UploadPart(uploadRequest);

 //Step 3: build and send the multipart complete request
 if (lastPart)
 
 eTags.Add(new PartETag
 
 PartNumber = partNumber,
 ETag = response.ETag
 );

 var completeRequest = new CompleteMultipartUploadRequest
 
 BucketName = _settings.Bucket,
 Key = fileName,
 UploadId = uploadId,
 PartETags = eTags
 ;

 try
 
 _s3Client.CompleteMultipartUpload(completeRequest);
 
 catch
 
 //do some logging and return null response
 return null;
 
 

 response.ResponseMetadata.Metadata["uploadid"] = uploadRequest.UploadId;
 return response;

Here, stream.Position = 0 and stream.Length throwing exceptions like below:

at
System.ServiceModel.Dispatcher.StreamFormatter.MessageBodyStream.get_Length()

Then i saw that stream.CanSeek is false.

Do i need to actually buffer the entire stream, loading it into memory in advance to make it working?

Update: I am doing below and it's working but don't know whether it is efficient or not.

 var ms = new MemoryStream();
 stream.CopyTo(ms);
 ms.Position = 0;

Is there any other way to do this? Thanks in advance.

That's a fair way of doing it, but I opted for a different approach by uploading directly to S3 using signed URLs. This has the benefit of taking some load off of your server, and reducing data transfer.

Depending on your application, it may be worth considering this:

In C# get the Presigned URL:

public string GetPreSignedUrl(string bucketName, string keyPrefix, string fileName)

 var client = new AmazonS3Client(_credentials, _region);
 var keyName = $"keyPrefix/fileName";
 var preSignedUrlRequest = new GetPreSignedUrlRequest()
 
 BucketName = bucketName,
 Key = keyName,
 Expires = DateTime.Now.AddMinutes(5),
 Protocol = (Protocol.HTTPS)
 ;
 return client.GetPreSignedURL(preSignedUrlRequest);

This creates a URL for a client to upload directly to S3, which you need to pass to the UI. Then you can use a multipart upload to the presigned url.

Here is a good example of multipart upload using axious: https://github.com/prestonlimlianjie/aws-s3-multipart-presigned-upload/blob/master/frontend/pages/index.js