Upload to AWS S3 using Django, Boto3 and S3Direct
The Challenge
One of the primary challenges when uploading files directly from a client-side application (such as a React Native mobile app) to an AWS S3 bucket is security. AWS S3 buckets are private by default, and you don't want to expose your AWS credentials (access key and secret key) directly in your client-side code, as this would pose a significant security risk.
To address this challenge, you can use pre-signed URLs, which provide a secure way to grant temporary access to your S3 bucket without exposing your credentials. A pre-signed URL is a URL that has been signed with your AWS credentials and includes a specific expiration time. This URL can be shared with a client application, allowing it to upload files directly to your S3 bucket without needing your AWS credentials.
S3Direct
S3 Direct is a Django library that provides a simple interface for uploading files directly to AWS S3 from the client-side. It aims to simplify the process of getting the necessary upload parameters and generating the required AWS Signature Version 4 (SigV4) signature for authenticating the upload request.
You can register destinations following the docs, and base on these destinations it provides two views:
1. get_upload_params: It returns the params for upload, such as: object_key, access_key_id, session_token, region, bucket, endpoint, acl and allow_existence_optimization.
2. generate_aws_v4_signature: It returns the signature key for the upload.
AWS Signature
AWS Signature Version 4 is a process for authenticating inbound API requests to AWS services using an HMAC-SHA256 signature. The signature is calculated based on the request data, including the HTTP method, request URI, request body, and other headers. This signature is then included in the request headers, allowing AWS to verify the authenticity and integrity of the request. The AWS documentation provides a detailed example of how to calculate the SigV4 signature for an S3 request
This example walks through the step-by-step process of calculating the signature, including creating the canonical request, creating the string to sign, calculating the signature key, and finally calculating the signature itself. You can use EvaporateJS library on web to handle the calculation using JS.
Since, it was complicated for me to portable this signature process for mobile, I used a pre signed url to upload the video to S3.
Pre Signed URL
Pre-signed URLs are a secure way to provide temporary access to AWS S3 resources without exposing your AWS credentials. A pre-signed URL is a URL that is constructed using your AWS credentials (access key and secret key), and it includes a specific expiration time. When a client accesses an S3 object using a pre-signed URL, AWS authenticates the request by verifying the signature and ensuring that the request is made within the specified time window.
Boto3
Since, S3 Direct has a limitation in that it doesn't provide a way to generate pre-signed URLs. This is where Boto3 comes into play.
Boto3 is the AWS SDK for Python, and it provides a convenient way to interact with various AWS services, including S3. In your code, you're using Boto3 to generate a pre-signed URL for uploading files to your S3 bucket.
The Boto3 documentation provides more details on generating pre-signed URLs for S3.
Implementation
Here's a high-level overview of the implementation:
First, we have the routes register in the urls.py file:
Embedded content: https://gist.github.com/kibolho/99cd18c72baf56780ce0423a2fa7be94?file=urls.py
Then, the create_presigned_post function in s3helper.py uses the generate_presigned_url method from the Boto3 S3 client to create a pre-signed URL for uploading an object to your S3 bucket. The pre-signed URL is then returned to the client application, which can use it to upload the file directly to S3.
Embedded content: https://gist.github.com/kibolho/99cd18c72baf56780ce0423a2fa7be94?file=s3helper.py
And finally, in the view we have the get_presigned_url function that uses the params from S3Direct and Generate the presigned url with Boto3:
Embedded content: https://gist.github.com/kibolho/99cd18c72baf56780ce0423a2fa7be94?file=view.py
By combining S3 Direct and Boto3, you're able to leverage the convenience of S3 Direct for handling the upload process while also taking advantage of Boto3's ability to generate secure pre-signed URLs for uploading files directly to your S3 bucket from a client-side application.
