How to Upload a String to a File on S3

In web and mobile applications, information technology's common to provide users with the ability to upload data. Your awarding may let users to upload PDFs and documents, or media such equally photos or videos. Every mod web server technology has mechanisms to allow this functionality. Typically, in the server-based environs, the procedure follows this flow:

1. The user uploads the file to the application server.
2. The application server saves the upload to a temporary space for processing.
3. The application transfers the file to a database, file server, or object store for persistent storage.

While the procedure is simple, it can have significant side-furnishings on the performance of the web-server in busier applications. Media uploads are typically big, and so transferring these can represent a big share of network I/O and server CPU fourth dimension. You must too manage the land of the transfer to ensure that the unabridged object is successfully uploaded, and manage retries and errors.

This is challenging for applications with spiky traffic patterns. For example, in a web awarding that specializes in sending holiday greetings, it may experience almost traffic simply around holidays. If thousands of users attempt to upload media around the same time, this requires you lot to scale out the application server and ensure that at that place is sufficient network bandwidth bachelor.

Past straight uploading these files to Amazon S3, y'all tin avoid proxying these requests through your awarding server. This can significantly reduce network traffic and server CPU usage, and enable your awarding server to handle other requests during busy periods. S3 also is highly bachelor and durable, making it an platonic persistent shop for user uploads.

In this blog post, I walk through how to implement serverless uploads and show the benefits of this approach. This design is used in the Happy Path web awarding. You can download the lawmaking from this blog postal service in this GitHub repo.

Overview of serverless uploading to S3

When y'all upload directly to an S3 bucket, you lot must first asking a signed URL from the Amazon S3 service. You can then upload directly using the signed URL. This is 2-pace procedure for your application front stop:

1. Call an Amazon API Gateway endpoint, which invokes the getSignedURL Lambda function. This gets a signed URL from the S3 saucepan.
2. Directly upload the file from the awarding to the S3 bucket.

To deploy the S3 uploader case in your AWS account:

1. Navigate to the S3 uploader repo and install the prerequisites listed in the README.md.
2. In a concluding window, run:
   git clone https://github.com/aws-samples/amazon-s3-presigned-urls-aws-sam
cd amazon-s3-presigned-urls-aws-sam
sam deploy --guided

3. At the prompts, enter s3uploader for Stack Name and select your preferred Region. Once the deployment is consummate, note the APIendpoint output.The API endpoint value is the base of operations URL. The upload URL is the API endpoint with /uploads appended. For instance: https://ab123345677.execute-api.us-west-2.amazonaws.com/uploads.

Testing the application

I show two ways to test this application. The first is with Postman, which allows you to directly call the API and upload a binary file with the signed URL. The second is with a bones frontend application that demonstrates how to integrate the API.

To test using Postman:

1. Beginning, copy the API endpoint from the output of the deployment.
2. In the Postman interface, paste the API endpoint into the box labeled Enter request URL.
3. Choose Ship.
4. After the request is complete, the Trunk section shows a JSON response. The uploadURL attribute contains the signed URL. Re-create this attribute to the clipboard.
5. Select the + icon adjacent to the tabs to create a new request.
6. Using the dropdown, change the method from Get to PUT. Paste the URL into the Enter asking URL box.
7. Choose the Trunk tab, then the binary radio button.
8. Cull Select file and choose a JPG file to upload.
   Cull Send. Yous run into a 200 OK response after the file is uploaded.
9. Navigate to the S3 console, and open up the S3 bucket created by the deployment. In the bucket, you run into the JPG file uploaded via Postman.

To test with the sample frontend application:

1. Copy index.html from the instance'south repo to an S3 bucket.
2. Update the object'due south permissions to arrive publicly readable.
3. In a browser, navigate to the public URL of index.html file.
4. Select Choose file and then select a JPG file to upload in the file picker. Choose Upload paradigm. When the upload completes, a confirmation bulletin is displayed.
5. Navigate to the S3 console, and open up the S3 bucket created by the deployment. In the saucepan, you see the second JPG file you uploaded from the browser.

Understanding the S3 uploading process

When uploading objects to S3 from a web application, you must configure S3 for Cantankerous-Origin Resource Sharing (CORS). CORS rules are defined as an XML document on the bucket. Using AWS SAM, you can configure CORS equally part of the resource definition in the AWS SAM template:

                      S3UploadBucket:     Type: AWS::S3::Bucket     Properties:       CorsConfiguration:         CorsRules:         - AllowedHeaders:             - "*"           AllowedMethods:             - GET             - PUT             - Head           AllowedOrigins:             - "*"                  

The preceding policy allows all headers and origins – it's recommended that you apply a more restrictive policy for production workloads.

In the get-go pace of the procedure, the API endpoint invokes the Lambda role to make the signed URL request. The Lambda part contains the post-obit code:

          const AWS = require('aws-sdk') AWS.config.update({ region: process.env.AWS_REGION }) const s3 = new AWS.S3() const URL_EXPIRATION_SECONDS = 300  // Chief Lambda entry point exports.handler = async (event) => {   return await getUploadURL(issue) }  const getUploadURL = async function(event) {   const randomID = parseInt(Math.random() * 10000000)   const Fundamental = `${randomID}.jpg`    // Go signed URL from S3   const s3Params = {     Bucket: process.env.UploadBucket,     Key,     Expires: URL_EXPIRATION_SECONDS,     ContentType: 'paradigm/jpeg'   }   const uploadURL = wait s3.getSignedUrlPromise('putObject', s3Params)   render JSON.stringify({     uploadURL: uploadURL,     Primal   }) }                  

This function determines the proper name, or key, of the uploaded object, using a random number. The s3Params object defines the accepted content type and also specifies the expiration of the central. In this case, the key is valid for 300 seconds. The signed URL is returned as role of a JSON object including the key for the calling application.

The signed URL contains a security token with permissions to upload this single object to this bucket. To successfully generate this token, the code calling getSignedUrlPromise must take s3:putObject permissions for the bucket. This Lambda part is granted the S3WritePolicy policy to the saucepan by the AWS SAM template.

The uploaded object must match the same file proper name and content blazon every bit defined in the parameters. An object matching the parameters may be uploaded multiple times, providing that the upload process starts before the token expires. The default expiration is 15 minutes only you may want to specify shorter expirations depending upon your use case.

One time the frontend application receives the API endpoint response, it has the signed URL. The frontend application then uses the PUT method to upload binary data directly to the signed URL:

          let blobData = new Blob([new Uint8Array(assortment)], {type: 'prototype/jpeg'}) const result = await fetch(signedURL, {   method: 'PUT',   body: blobData })                  

At this betoken, the caller awarding is interacting directly with the S3 service and not with your API endpoint or Lambda function. S3 returns a 200 HTML condition code once the upload is consummate.

For applications expecting a large number of user uploads, this provides a elementary way to offload a big amount of network traffic to S3, away from your backend infrastructure.

Adding authentication to the upload process

The current API endpoint is open, bachelor to any service on the net. This means that anyone tin can upload a JPG file once they receive the signed URL. In most product systems, developers want to use authentication to control who has access to the API, and who can upload files to your S3 buckets.

Y'all can restrict access to this API past using an authorizer. This sample uses HTTP APIs, which support JWT authorizers. This allows you lot to control access to the API via an identity provider, which could exist a service such as Amazon Cognito or Auth0.

The Happy Path application just allows signed-in users to upload files, using Auth0 every bit the identity provider. The sample repo contains a 2d AWS SAM template, templateWithAuth.yaml, which shows how you can add an authorizer to the API:

                      MyApi:     Type: AWS::Serverless::HttpApi     Backdrop:       Auth:         Authorizers:           MyAuthorizer:             JwtConfiguration:               issuer: !Ref Auth0issuer               audition:                 - https://auth0-jwt-authorizer             IdentitySource: "$request.header.Potency"         DefaultAuthorizer: MyAuthorizer                  

Both the issuer and audience attributes are provided by the Auth0 configuration. Past specifying this authorizer as the default authorizer, it is used automatically for all routes using this API. Read part one of the Ask Around Me series to acquire more about configuring Auth0 and authorizers with HTTP APIs.

After authentication is added, the calling spider web application provides a JWT token in the headers of the asking:

          const response = await axios.get(API_ENDPOINT_URL, {   headers: {     Potency: `Bearer ${token}`         } })                  

API Gateway evaluates this token before invoking the getUploadURL Lambda function. This ensures that but authenticated users can upload objects to the S3 bucket.

Modifying ACLs and creating publicly readable objects

In the electric current implementation, the uploaded object is not publicly attainable. To brand an uploaded object publicly readable, you must set up its access control list (ACL). In that location are preconfigured ACLs bachelor in S3, including a public-read selection, which makes an object readable by anyone on the internet. Fix the appropriate ACL in the params object before calling s3.getSignedUrl:

          const s3Params = {   Saucepan: procedure.env.UploadBucket,   Key,   Expires: URL_EXPIRATION_SECONDS,   ContentType: 'image/jpeg',   ACL: 'public-read' }                  

Since the Lambda part must have the appropriate bucket permissions to sign the request, y'all must also ensure that the role has PutObjectAcl permission. In AWS SAM, you can add the permission to the Lambda office with this policy:

                      - Statement:           - Result: Allow             Resource: !Sub 'arn:aws:s3:::${S3UploadBucket}/'             Action:               - s3:putObjectAcl                  

Decision

Many web and mobile applications allow users to upload data, including big media files similar images and videos. In a traditional server-based application, this tin can create heavy load on the application server, and as well use a considerable amount of network bandwidth.

Past enabling users to upload files to Amazon S3, this serverless pattern moves the network load away from your service. This can make your application much more scalable, and capable of handling spiky traffic.

This blog post walks through a sample application repo and explains the process for retrieving a signed URL from S3. Information technology explains how to the test the URLs in both Postman and in a spider web application. Finally, I explicate how to add authentication and make uploaded objects publicly accessible.

To learn more, see this video walkthrough that shows how to upload straight to S3 from a frontend spider web application. For more than serverless learning resources, visit https://serverlessland.com.

deatonpilly1991.blogspot.com

Source: https://aws.amazon.com/blogs/compute/uploading-to-amazon-s3-directly-from-a-web-or-mobile-application/