Abílio Azevedo.

Upload to AWS S3 using Django, Boto3 and S3Direct

Cover Image for Upload to AWS S3 using Django, Boto3 and S3Direct
Abílio Azevedo
Abílio Azevedo

The Challenge

One of the primary challenges when uploading files directly from a client-side application (such as a React Native mobile app) to an AWS S3 bucket is security. AWS S3 buckets are private by default, and you don't want to expose your AWS credentials (access key and secret key) directly in your client-side code, as this would pose a significant security risk.

To address this challenge, you can use pre-signed URLs, which provide a secure way to grant temporary access to your S3 bucket without exposing your credentials. A pre-signed URL is a URL that has been signed with your AWS credentials and includes a specific expiration time. This URL can be shared with a client application, allowing it to upload files directly to your S3 bucket without needing your AWS credentials.

S3Direct

S3 Direct is a Django library that provides a simple interface for uploading files directly to AWS S3 from the client-side. It aims to simplify the process of getting the necessary upload parameters and generating the required AWS Signature Version 4 (SigV4) signature for authenticating the upload request.

You can register destinations following the docs, and base on these destinations it provides two views:

1. get_upload_params: It returns the params for upload, such as: object_key, access_key_id, session_token, region, bucket, endpoint, acl and allow_existence_optimization.

2. generate_aws_v4_signature: It returns the signature key for the upload.

AWS Signature

AWS Signature Version 4 is a process for authenticating inbound API requests to AWS services using an HMAC-SHA256 signature. The signature is calculated based on the request data, including the HTTP method, request URI, request body, and other headers. This signature is then included in the request headers, allowing AWS to verify the authenticity and integrity of the request. The AWS documentation provides a detailed example of how to calculate the SigV4 signature for an S3 request

This example walks through the step-by-step process of calculating the signature, including creating the canonical request, creating the string to sign, calculating the signature key, and finally calculating the signature itself. You can use EvaporateJS library on web to handle the calculation using JS.

Since, it was complicated for me to portable this signature process for mobile, I used a pre signed url to upload the video to S3.

Pre Signed URL

Pre-signed URLs are a secure way to provide temporary access to AWS S3 resources without exposing your AWS credentials. A pre-signed URL is a URL that is constructed using your AWS credentials (access key and secret key), and it includes a specific expiration time. When a client accesses an S3 object using a pre-signed URL, AWS authenticates the request by verifying the signature and ensuring that the request is made within the specified time window.

Boto3

Since, S3 Direct has a limitation in that it doesn't provide a way to generate pre-signed URLs. This is where Boto3 comes into play.

Boto3 is the AWS SDK for Python, and it provides a convenient way to interact with various AWS services, including S3. In your code, you're using Boto3 to generate a pre-signed URL for uploading files to your S3 bucket.

The Boto3 documentation provides more details on generating pre-signed URLs for S3.

Implementation

Here's a high-level overview of the implementation:

First, we have the routes register in the urls.py file:

Embedded content: https://gist.github.com/kibolho/99cd18c72baf56780ce0423a2fa7be94?file=urls.py

Then, the create_presigned_post function in s3helper.py uses the generate_presigned_url method from the Boto3 S3 client to create a pre-signed URL for uploading an object to your S3 bucket. The pre-signed URL is then returned to the client application, which can use it to upload the file directly to S3.

Embedded content: https://gist.github.com/kibolho/99cd18c72baf56780ce0423a2fa7be94?file=s3helper.py

And finally, in the view we have the get_presigned_url function that uses the params from S3Direct and Generate the presigned url with Boto3:

Embedded content: https://gist.github.com/kibolho/99cd18c72baf56780ce0423a2fa7be94?file=view.py

By combining S3 Direct and Boto3, you're able to leverage the convenience of S3 Direct for handling the upload process while also taking advantage of Boto3's ability to generate secure pre-signed URLs for uploading files directly to your S3 bucket from a client-side application.undefined


More posts

Cover Image for Factory Health Monitor

Factory Health Monitor

The application is a tool for evaluating the health of various machines in an automobile manufacturing plant. This plant features a range of machines, each with specific data points that you will use to assess the condition of the production process.

Abílio Azevedo
Abílio Azevedo
Cover Image for Tech Documents

Tech Documents

Learn the vital importance of comprehensive technical documentation for growing software projects. Discover best practices like Requests for Comments (RFCs) and Architectural Decision Records (ADRs) that promote transparency, collaboration, and architectural decision logging. Explore powerful tools like wiki.js and Backstage to create effective documentation hubs. Keep your project organized, comprehensible, and sustainable with this approach to technical documentation.

Abílio Azevedo
Abílio Azevedo

NewsLetter

I will send the content posted here. No Spam =)

Experienced Software Engineer with degree in Electrical Engineering with over 8 years of hands-on expertise in building robust and scalable mobile, web and backend applications across various projects mainly in the fintech sector. Mobile (React Native), Web (React and Next.JS) and Backend (Node.JS, PHP and DJANGO). My goal is to create products that add value to people. - © 2024, Abílio Azevedo