Leverage Graviton2 and CloudFront to dynamically generate thumbnails for S3 object storage

As a developer, it is often necessary to provide images of various sizes to ensure a great access experience across devices of different screen sizes and resolutions. In this way, the management of pictures will become very complicated. Images stored on S3 are often processed into various sizes to suit websites and apps.

This article will explain a way, when the device accesses the picture on S3, it will generate a picture of appropriate size and return it to the device.

In fact, in 2017, Amazon Cloud Technology released a solution - Serverless Image Handler

https://aws.amazon.com/solutions/implementations/serverless-image-handler/

This solution leverages Lambda and API Gateway to dynamically generate thumbnails. This paper implemented functions and Serverless Image Handler exactly the same, but this paper, a new release of Graviton2 models have in massive workload and changes gently premise can be reached on substantial cost optimization . You can see the different scenarios where Serverless Image Handler and Graviton2 handle thumbnails from the table below:

this solution advantage

handles pictures flexibly: can dynamically generate pictures by simply adding the size of the pictures to be generated in the URL, and with the upgraded floating-point operation performance of Graviton2, pictures can be processed more quickly.

Optimization cost: is only triggered and generated quickly when an image needs to change its size. This saves as much storage space as possible. And use Graviton2 to process images, which is 20% cheaper than an Intel chip EC2 instance of the same specification.

High reliability: This solution uses the Origin Group function of CloudFront to ensure that 404 will not occur when accessing S3 objects. And using Auto Scaling Group, Graviton2 EC2 instances that process images also maintain high reliability.

architecture overview

If the user accesses an original image through the APP, the thumbnail of the original image will be requested first, so that the image can be loaded faster. This request is forwarded to CloudFront. CloudFront has the function of Origin Group. This function allows a CloudFront distribution to have two origin sites. If the first origin site returns 400 or 500, the CloudFront distribution will forward the request to the set second origin site. Using CloudFront's Origin Group function, we make the S3 bucket the first origin site. When the requested thumbnail is not found, CloudFront forwards the request to the second origin site we set up: an Auto Scaling Group Gravition2 cluster, this The cluster will extract the information in the URL, the bucket name and original image name provided in the URL, and the length and width that need to be modified, find the original image from the S3 bucket, return the thumbnail and save it back to the S3 bucket.

1. The user requests a 300-pixel by 300-pixel thumbnail of the original image thumber-test.example.com/test.jpg, and the URL of this thumbnail is thumber-test.example.com/test.jpg/thumb_300_300.jpg

2. Through the Origin Group distributed by CloudFront, forward the request to the first origin server S3 bucket.

3. If the thumbnail S3://thumber-test/test.jpg/thumb_300_300.jpg exists, return directly and the whole process ends. If the thumbnail doesn't exist, the S3 bucket returns 400.

4. Since the S3 bucket returns 400, according to the settings of the CloudFront Origin Group, the request is forwarded to the second origin: an Auto Scaling Group Gravition2 cluster.

5. The application on the Gravition2 cluster analyzes the request URL, extracts the original image name jpg, and the thumbnail size is 300 X 300. Download the original image S3://thumber-test/test.jpg from the S3 bucket and resize it.

6. The program creates a folder in the S3 bucket named thumb-test: "jpg/", and uploads the thumbnail thumb_300_300.jpg to it.

7. Also returns the thumbnail file to CloudFront.

8. CloudFront returns the file to the user.

Deployment Guide

1. First we refer to this blog:

https://aws.amazon.com/cn/blogs/china/use-graviton2-example-to-build-a-cost-effective-php-load-operating-environment/

Among them, 2.1, 2.2 and 2.3, build a php running environment based on Gravition2 EC2 instance. If it is used for production, you can build an Auto Scaling Group, but building ASG is not the focus of this article, so only a single EC2 instance is used as an example.

2. Put the https://github.com/nwcd-samples/thumber/blob/main/s3thumber.php file into the /usr/share/nginx/html/ directory on the EC2 instance and set it as the default home page.

3. Create an S3 bucket named thumber-test, fill in a second-level domain name in "Alternate domain names (CNAMEs)", and ensure that the subdomain name is the same as the S3 bucket name, such as: thumber-test.example.com, And upload the image jpg to the storage bucket.

4. Open the CloudFront service and create a distribution based on the thumber-test bucket. Then click to enter the newly created distribution, select "Source and Source Group", click "Create Source", and fill in the domain name of the newly created Gravition2 EC2 instance in "Source Domain Name".

1. On the same page, select "Create Source Group" in "Source Group", and add an S3 bucket source and an EC2 instance source in turn. And tick 404 and 403 in "Failover Condition".

2. Select "Behavior" in the tab bar, select the default behavior, click "Edit", select the newly created source group ID in "Source or source group", and then save.

3. Point the CNAME of the second-level domain name thumber-test.example.com to the distributed domain name.

4. At this point, you can verify, enter https://thumber-test.example.com/test.jpg/thumb_300_300 in the browser, and a picture with a length and width of 300 pixels will be returned. Enter the S3 bucket named thumber-test, you can see an additional folder named "jpg/" with a file named thumb_300_300. The pixel size can be freely modified as required.

Conclusion

After the deployment is completed, we use the same size models, c6g.large, c5.large, c5a.large, to simulate the customer's production environment for testing. The test data set is 10% below 5M, 20% above 20M, and 70 % of 5M to 20M size pictures. After testing, the following results can be obtained (due to the fact that the actual system parameters, data sets and other details may be different, the final test results will be different, so the specific values of the test results are hidden):

It can be seen that among the models of the same size, c6g.large has the best performance and the lowest price when running php to modify the image size. In addition, c6g also has smaller instances. You can use the c6g instance to build an auto scaling group, and its elastic granularity is better than the other two instances.

The author of this

Amazon Cloud Technology Solution Architect

Currently focusing on containerized solutions. Before joining Amazon Cloud Technology, he worked for HP, IBM and other technology companies, engaged in data center infrastructure related work, and has more than ten years of technical service experience.