Using Amazon PrivateLink - Easy Access to Private Endpoint Amazon S3

Summary

Amazon Cloud Technology officially released Amazon PrivateLink for Amazon S3 in February 2021. Amazon PrivateLink Amazon Simple Storage Service (S3) and on-premises resources using a private IP in a virtual network. With this feature, you can now use Virtual Private Cloud to directly access Amazon S3 as a private endpoint in a secure virtual network. This extends the functionality of existing gateway endpoints by allowing you to access Amazon S3 using a private IP address. API and HTTPS requests to Amazon S3 from on-premises applications are automatically directed through interface endpoints that connect to Amazon S3 in a secure and private manner via PrivateLink.

• Amazon PrivateLink for Amazon S3
  https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html
• Amazon PrivateLink
  https://aws.amazon.com/privatelink/
• Amazon Simple Storage Service (S3)
  https://aws.amazon.com/s3/
• Virtual Private Cloud
  https://aws.amazon.com/vpc/

The solution described in this document is to achieve cross-region private access to Amazon S3 by utilizing cross-region VPC peering connections and Amazon S3 interface endpoints. Customers can privately access Amazon S3 from on-premises applications via secure connections provided Amazon Direct Connect or Amazon VPN , depending on their needs.

• Amazon Direct Connect
  https://aws.amazon.com/directconnect/
• Amazon VPN
  https://aws.amazon.com/vpn/

📢 To learn more about the latest technology releases and practical innovations of Amazon Cloud Technology, please pay attention to the 2021 Amazon Cloud Technology China Summit! Click on the picture to sign up~ For more exciting content, please look forward to the 8.19-20 Beijing and 9.15 Shenzhen chapters!

Service Introduction

Back in 2015, Amazon S3 was the first service to support adding VPC endpoints ; provides a secure connection to Amazon S3 through a VPC gateway endpoint without requiring a gateway or NAT instance. The principle is to create a route to the Amazon S3 service in the specified routing table, each subnet associated with the routing table can access the endpoint, and then route the traffic from these subnet instances to the Amazon S3 service through the endpoint. . In the following diagram, instances in subnet 2 can access Amazon S3 through the gateway endpoint.

• Amazon S3 is the first service to support adding VPC endpoints
  https://aws.amazon.com/blogs/aws/new-vpc-endpoint-for-amazon-s3/

This new flexibility is well received by customers, but it also has its limitations – private access to Amazon S3 across regions and on-premises datacenters is not supported, which the newly introduced Amazon S3 interface endpoints can do by choosing , create an Amazon S3 interface endpoint with an ENI that assigns a private IP, enabling a secure connection to Amazon S3. In-VPC applications, or Amazon Direct Connect or Amazon VPN , can privately access Amazon S3 through the interface endpoint. Interface endpoints simplify your network architecture by eliminating the need to configure firewall rules or internet gateways when connecting to Amazon S3 from on-premises applications.

• Amazon Direct Connect
  https://aws.amazon.com/directconnect/
• Amazon VPN
  https://aws.amazon.com/vpn/

Comparison of two Amazon S3 endpoints:

overall architecture of the demo environment

  

VPC setup for the demo environment:

Notice:

• The CIDR blocks of 2 VPCs connected by VPC peering cannot overlap
• Using private DNS (Amazon PrivateLink) with an interface VPC endpoint, you must enableDnsHostnames
• and enableDnsSupport property set to true
• DNS resolution needs to be enabled in both VPC peering connections
• The security group for the interface endpoint must allow inbound HTTPS (port 443) traffic

Demonstration environment

4.1 Create a demo VPC environment

Create a VPC with a public subnet and a private subnet in each region through the wizard of the VPC Console, and create an Amazon S3 bucket in the cn-northwest-1 region (the specific steps are omitted). Below is a screenshot from my demo environment:

cn-north-1 regional VPC and subnet

cn-northwest-1 regional VPC and subnet

Enable DNS hostnames and DNS resolution in both VPCs:

Amazon S3 bucket in the cn-northwest-1 region

  

4.2 Establish a cross-region VPC

Create a peering connection between the VPCs in the two regions and update the routing table (the specific steps are omitted). The following is a screenshot of my demo environment:

VPC peering connection

Enable DNS resolution in both VPC peering connections

cn-north-1 area routing table

cn-northwest-1 regional routing table

4.3 Verifying cross-region VPC

Create an EC2 instance in each of the public subnet and private subnet of cn-north-1. The EC2 in the public subnet is used as a springboard to connect to the EC2 in the private subnet. At the same time, create an EC2 instance on the private subnet of cn-northwest-1.

cn-north-1 region EC2

cn-northwest-1 region EC2

EC2 Security Group Inbound

EC2 Security Group Outbound

With the help of the EC2 jumper in the public subnet, verify that the EC2 (IP: 172.31.30.22) in the private subnet of cn-north-1 has successfully SSHed to the EC2 instance (IP: 10.10.1.94) of cn-northwest-1.

4.4 Create Amazon S3 interface endpoint

First, in order to ensure that Amazon Cloud Technology command line tools (such as Amazon CLI) can make requests from resources in the VPC to Amazon Cloud Technology services over HTTPS, the security group of the interface endpoint must allow inbound HTTPS (port 443) traffic, so in cn The -northwest-1 region creates a security group with the following inbound rules:

Create an Amazon S3 interface endpoint in the VPC in the cn-northwest-1 region, and associate the corresponding VPC and subnet to implement cross-region private access to Amazon S3.

Next, associate the security group created earlier with the interface endpoint, and then create the interface endpoint for Amazon S3.

Once created, two types of DNS for Amazon S3 interface endpoints are generated:

• Regional DNS – includes the unique VPC endpoint ID, service identifier, Amazon region and amazonaws.com, in this instance vpce-0257c85882d96cc95-vj9syim9.s3.cn-northwest-1.vpce.amazonaws.com.cn
• Zone DNS – includes availability zones, such as vpce-0257c85882d96cc95-vj9syim9-cn-northwest-1a.s3.cn-northwest-1.vpce.amazonaws.com.cn, customers use this type of DNS to specify availability zones and reduce Availability Zone Data Transfer Cost

Amazon S3 Interface Endpoint Create an Amazon Amazon S3 Interface Endpoint with an ENI that assigns a private IP in the subnet of your choice, enabling a secure connection to Amazon S3.

 

4.5 Test Verification

First, create a role for the EC2 service in cn-north-1 and give Amazon S3 read and write permissions.

Assign the role of the created EC2 service to EC2 (IP: 172.31.30.22) in the private subnet of cn-north-1.

Verify that the Amazon s3 bucket of cn-northwest-1 cannot be directly accessed from EC2, because EC2 is in a private subnet and cannot access the public network to access Amazon S3 Endpoint. The command will hang until it times out.

Note: You need to upgrade the Amazon CLI to the latest version first.

Verify that EC2 in the private subnet of cn-north-1 can successfully access the Amazon s3 bucket located at cn-northwest-1 and upload files through the VPC peering connection and Amazon S3 interface endpoint DNS.

use recommended

The Amazon S3 interface endpoint and the previous gateway endpoint can coexist. It is recommended that applications in the VPC in this region secure Amazon S3 privately through the gateway endpoint, which will not incur additional charges. For other regions and customer data centers with interface endpoints, connect to Amazon S3 in a secure and private manner via Amazon PrivateLink. The recommended architecture is as follows:

For the local data center to access the DNS of the Amazon S3 interface endpoint, the Amazon Cloud Technology Global area can be combined with Route53 Resolver (refer to Document 1); the Amazon Cloud Technology China area can build a DNS Proxy to implement DNS resolution (refer to Document 2).

summary

This article introduces the use of Amazon S3's interface endpoints and cross-region VPC peering to achieve cross-region private access to Amazon S3, and network traffic is maintained on Amazon's network through Amazon PrivateLink. Customers can privately access Amazon S3 from on-premises applications via secure connections provided Amazon Direct Connect or Amazon VPN , depending on their needs. API and HTTPS requests from on-premises applications to Amazon S3 are automatically directed through interface endpoints that connect to Amazon S3 in a secure and private manner via PrivateLink.

• Amazon Direct Connect
  https://aws.amazon.com/directconnect/
• Amazon VPN
  https://aws.amazon.com/vpn/

Reference Document

• https://docs.aws.amazon.com/zh_cn/Route53/latest/DeveloperGuide/resolver.html
• https://aws.amazon.com/blogs/security/how-to-set-up-dns-resolution-between-on-premises-networks-and-aws-by-using-unbound/
• https://aws.amazon.com/cn/blogs/china/aws-privatelink-for-amazon-s3-now-available/
• https://docs.aws.amazon.com/zh_cn/vpc/latest/peering/working-with-vpc-peering.html
• https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/userguide/privatelink-interface-endpoints.html

Author of this article

Zhu

Amazon Cloud Technology Account Manager

Responsible for the structure and cost optimization, technical support and other work of enterprise-level customers. He is committed to AdTech and other industries. He has worked for IBM and has 15+ years of experience in product design, development/testing, and technical support.