What will cloud computing look like without the SDK?

Foreword

Recently, I spent some time compiling and optimizing one of my routers running OpenWRT, a Linksys WRT 1900ACS. In order to expand the function of the router, some new functions need to be developed on OpenWRT, especially some interesting services on Amazon cloud technology need to be used. When I started using the Amazon SDK habitually, I suddenly realized that using these SDKs in a system with low hardware configuration and extremely thin software is undoubtedly an extremely luxurious idea. Even the router with high hardware configuration like my hand has only 128M of storage and 512M of memory resources.

This may make my work more interesting, allowing me to study the API calling mechanism of Amazon Cloud Technology and how to use it more effectively, instead of relying on a highly packaged SDK. The main challenge of this task is to successfully execute authenticated Amazon REST API requests, such as EC2 security groups, VPC ACL rules, Amazon S3 file access and other interesting features.

?

My job is not to use the system illegally like a "hacker". In fact, Amazon Cloud Technology has long provided a standard calling interface for REST API calls, and the most critical link is the signature process of the API request header named Signatura Version 4. This process is described in detail in the documentation of Amazon Cloud Technology, but I believe that only a few people should be able to read this document, because this process is really cumbersome. Reasonable developers usually ignore these APIs and are more accustomed to using various Amazon cloud technology SDKs. Even some simple tasks can be solved by Amazon-Cli using a script.

However, as in my case. SDKs suitable for working platforms or programming languages may not be available in some scenarios. These requirements include but are not limited to these

1. Resource constraints. For example in an embedded environment

2. Performance requirements. For example, the CPU with lower performance is a simple performance comparison I did. The scenario is for a file on S3 to be downloaded to the local

3. SDK is missing. For example Amazon SDK

4. Missing language-specific SDKs. For example Rust, etc. (Note: rusoto is an unofficial SDK package)

5. The existing SDK function is missing. For example Amazon Transcribe live transcription

5. Reduce dependencies. For example, using Python's boto3, you need to install some dependencies like python3, python3-yaml, python3-pyasn1, python3-botocore, python3-rsa, python3-colorama, python3-docutils, python3-s3transfer, etc.

In addition, understanding and mastering the Amazon REST API will be of great benefit to developers in system optimization, architecture design, and system security improvement.

tools we need

For this task, we will use:

1. python3+ (python2 can theoretically also be implemented, but I didn't try it) 2. Python's requests package (pip3 install requests) can be installed. It is also possible to use Python's built-in urllib instead of requests.

2. Text editor (such as my commonly used vim)

3. curl (command line tool for requesting web services)

4. openssl (basic software package for secure communication)

5.sed (a stream editor commonly used in Linux scripts)

We will use these tools to implement calls to the Amazon API in Python programs and shell scripts, respectively. Usually, the SDK of Amazon cloud technology (such as boto3 for Python) will help our application to automatically complete the signing of the request, so this link is transparent to the developer. For today's task, we will need to do the most important signature operation by ourselves.

Related reference implementations

Similar to my idea, someone has practiced and shared it for a long time. Among the more well-known ones are:

1.requests-amazon4auth
https://github.com/sam-washington/requests-aws4auth
Python Request library for Amazon Web Service Authentication Version 4

2.amazon-requests-auth
https://github.com/DavidMuller/aws-requests-auth
Python requests module for Amazon Cloud Tech Signature Version 4 signing process

3.amazon-request-signer
https://github.com/iksteen/aws-request-signer
Python library for signing Amazon Cloud Tech requests using Amazon Cloud Tech Signature V4

Of the above three open source Python libraries, except for the last one that was updated 4 months ago, the other two have not been updated for more than 2 years, so it is difficult to use them with confidence! The last one introduced is more interesting, because this method does not use boto3 but uses botocore to implement the signature, which is a kind of opportunistic approach.

# Key derivation functions. See:
# http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-python
defsign(key, msg):
returnnew(key, msg.encode('utf-8'), hashlib.sha256).digest()

defgetSignatureKey(key, dateStamp, regionName, serviceName):
kDate = sign(('AWS4' + key).encode('utf-8'), dateStamp)
kRegion = sign(kDate, regionName)
kService = sign(kRegion, serviceName)
kSigning = sign(kService, 'aws4_request')
returnkSigning

Why do requests to APIs need to be signed?

Almost every function of all Amazon cloud technology services provides an API, and these APIs are REST APIs. This means that we can complete the call to the Amazon API by means of HTTP request. It is very simple to implement such a call, but we also need to meet the following three requirements in this call process:

1. verify the identity of the requester

Make sure the request was sent by a user with a valid access key

2. protects data in

To prevent requests from being tampered with in transit, some request elements will be used to compute a hash (digest) of the request, and the resulting hash will be included in the request. When the Amazon service receives a request, it will use the same information to calculate the hash and match it against the hash value included in the request. If the values do not match, Amazon will deny the request.

3. prevents potential inversion attacks

In most cases, the request must reach Amazon within 5 minutes of the timestamp in the request. Otherwise, Amazon will deny the request.

This introduces a very important method - signed requests. When our application sends HTTP requests to Amazon, the requests need to be signed so that Amazon can identify the user who sent them. The request is signed using the Amazon access key, which contains the access key ID and secret access key. Some requests do not require signatures, such as anonymous requests to Amazon S3 and some API operations in Amazon STS, other API requests require signatures.

Signature Version 4

To sign a request, first compute the hash (digest) value of the request. Then, using this hash, some other information from the request, and the Amazon secret access key, another hash called a "signature" is calculated.

1. Create a canonical request for Signature Version 4 Organize the content of the request (host, action, headers, etc.) into a standard (canonical) format. A canonical request is one of the inputs used to create a string to sign. The request specification has the following format:

“ HTTP_Method” \ n“ Canonical_URI” \ n“ Canonical_Query” \ n“ Canonical_Headers” \ n“ Signed_Headers” \ n“ Request_payload”

2. Create Signature Version 4 String to Sign Create a String to Sign using the canonical request and additional information such as algorithm, request date, credential scope, and digest (hash) of the canonical request. Strings have the following format:

"AWS4-HMAC-SHA256"\n"UTC date"\n"date/region ID/s3/aws4_request"\n"Canonical_str"

3. Amazon Signature version 4 Using the Amazon secret access key as the key for the initial hash operation, perform a series of cryptographic hash operations (HMAC operations) on the request date, region, and service to derive the signature key . After deriving the signing key, the signature is computed by performing a cryptographic hash operation on the string to be signed. Use the derived signing key as the hash key for this operation. The format is as follows:

MAC_SHA256(HMAC_SHA256(HMAC_SHA256(HMAC_SHA256("Amazon4" secret key, date), region id), "s3"), "amazon4_request")

4. Add a signature to the HTTP request After computing the signature, add it to the HTTP header or query string of the request. Specifically, use the signature key in step 3 to convert the SHA256 HMAC calculation result of the signature string created in step 2 into hexadecimal characters. The format is as follows:

(signature key, signature string)

Next, the signature can be added to the request in one of two ways:

1. Use the HTTP Authorization header

2. Add the query string value to the request. Since the signature is part of the URL, such URLs are called pre-signed URLs

When the Amazon service receives the request, it will perform the same steps you did to calculate the signature sent in the request. After that, Amazon will compare the calculated signature with the signature you sent in the request. If the signatures match, process the request. If the signatures do not match, reject the request.

Regarding the details of the implementation, we can get a glimpse of it through two key functions (Python code)

#!/usr/bin/env python3
# -*- coding:utf-8 -*-

importsys
importos
importdatetime
importhashlib
importhmac
importrequests
fromexceptions importHTTPError
fromexceptions importTimeout

ALGORITHM = 'AWS4-HMAC-SHA256'
METHOD = 'GET'


def_sign(key, msg):
returnnew(key, msg.encode('utf-8'), hashlib.sha256).digest()


defget_SignatureKey(key, dateStamp, regionName, serviceName):
date = _sign(('AWS4'+ key).encode('utf-8'), dateStamp)
region = _sign(date, regionName)
service = _sign(region, serviceName)
signing = _sign(service, 'aws4_request')
returnsigning


defget_key():
returnenviron.get('AWS_ACCESS_KEY_ID'), \
environ.get('AWS_SECRET_ACCESS_KEY')


defget_datetime():
current = datetime.datetime.utcnow()
returnstrftime('%Y%m%dT%H%M%SZ'), current.strftime('%Y%m%d')


defget_endpoint(service, region):
return'https://{}.{}.amazonaws.com'.format(service, region)


defget_host(endpoint):
returnreplace('https://', '')


defget_reqUrl(endpoint, canonical_querystring):
return'{}?{}'.format(endpoint, canonical_querystring)


defget_header(region, service, request_parameters):
amzdate, datestamp = get_datetime()
endpoint = get_endpoint(service, region)
host = get_host(endpoint)

access_key, secret_key = get_key()
ifaccess_key is None or secret_key is None:
print('No access key is available.')
exit()

canonical_uri = '/'
canonical_querystring = request_parameters
canonical_headers = 'host:{}\nx-amz-date:{}\n'.format(host, amzdate)

signed_headers = 'host;x-amz-date'
payload_hash = hashlib.sha256(('').encode('utf-8')).hexdigest()

canonical_request = '{}\n{}\n{}\n{}\n{}\n{}'.format(
METHOD,
canonical_uri,
canonical_querystring,
canonical_headers,
signed_headers,
payload_hash
)

credential_scope = '{}/{}/{}/aws4_request'.format(
datestamp, region, service)

string_to_sign = '{}\n{}\n{}\n{}'.format(
ALGORITHM,
amzdate,
credential_scope,
sha256(
encode('utf-8')).hexdigest()
)

signing_key = get_SignatureKey(secret_key, datestamp, region, service)

signature = hmac.new(
signing_key,
(string_to_sign).encode('utf-8'),
sha256
).hexdigest()

authorization_header = \
'{} Credential={}/{},SignedHeaders={},Signature={}'.format(
ALGORITHM,
access_key,
credential_scope,
signed_headers,
signature
)

headers = {'x-amz-date': amzdate, 'Authorization': authorization_header}
request_url = get_reqUrl(endpoint, canonical_querystring)
return request_url, headers


defmain():
service = 'ec2'
region = 'us-west-1'

action = 'DescribeInstances'\
'&Filter.1.Name=instance-state-name&Filter.1.Value.1=running'
version = "2016-11-15"
request_parameters = 'Action={}&Version={}'.format(action, version)

request_url, headers = get_header(region, service, request_parameters)

print('\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++')
print('Request URL = {}'.format(request_url))
print('Request header = {}'.format(str(headers)))
try:
res = requests.get(request_url, headers=headers, timeout=(2, 5))
except Timeout:
print('The request timed out')
except HTTPError as http_err:
print(f'HTTP error occurred: {http_err}')
except Exception as err:
print(f'Other error occurred: {err}')
else:
print('\nRESPONSE++++++++++++++++++++++++++++++++++++')
print('Response code: %d\n' % res.status_code)
print(res.text)


if__name__ == "__main__":
main()

Using Python3, rrequests to implement the call to Amazon Translate, to achieve English-Chinese translation

#!/usr/bin/env python3
# -*- coding:utf-8 -*-
# Writen by Lianghong2020-03-12 11:42:56

importsys
importos
importdatetime
importhashlib
importhmac
importjson
importrequests
fromexceptions importHTTPError
fromexceptions importTimeout

ALGORITHM = 'AWS4-HMAC-SHA256'
METHOD = 'POST'


def_sign(key, msg):
returnnew(key, msg.encode("utf-8"), hashlib.sha256).digest()


defget_SignatureKey(key, datestamp, regionName, serviceName):
k_date = _sign(('AWS4'+ key).encode('utf-8'), datestamp)
k_region = _sign(k_date, regionName)
k_service = _sign(k_region, serviceName)
k_signing = _sign(k_service, 'aws4_request')
returnk_signing


defget_key():
returnenviron.get('AWS_ACCESS_KEY_ID'), \
environ.get('AWS_SECRET_ACCESS_KEY')


defget_datetime():
current = datetime.datetime.utcnow()
returnstrftime('%Y%m%dT%H%M%SZ'), current.strftime('%Y%m%d')


defget_host(service, region):
return'{}.{}.amazonaws.com'.format(service, region)


defget_header(service, region, request_parameters):
access_key, secret_key = get_key()
ifaccess_key is None or secret_key is None:
print('No access key is available.')
exit()

amz_date, date_stamp = get_datetime()
host = get_host(service, region)

canonical_uri = '/'
canonical_querystring = ''

content_type = 'application/x-amz-json-1.1'
amz_target = 'AWSShineFrontendService_20170701.TranslateText'

canonical_headers = \
'content-type:{}\nhost:{}\nx-amz-date:{}\nx-amz-target:{}\n'.format(
content_type,
host,
amz_date,
amz_target
)

signed_headers = 'content-type;host;x-amz-date;x-amz-target'
payload_hash = hashlib.sha256(
encode(
'utf-8'
)).hexdigest()

canonical_request = '{}\n{}\n{}\n{}\n{}\n{}'.format(
METHOD,
canonical_uri,
canonical_querystring,
canonical_headers,
signed_headers,
payload_hash
)

credential_scope = '{}/{}/{}/aws4_request'.format(
date_stamp, region, service)

string_to_sign = '{}\n{}\n{}\n{}'.format(
ALGORITHM, amz_date, credential_scope,
sha256(canonical_request.encode('utf-8')).hexdigest()
)

signing_key = get_SignatureKey(secret_key, date_stamp, region, service)
signature = hmac.new(
signing_key,
(string_to_sign).encode('utf-8'),
sha256).hexdigest()

authorization_header = \
'{} Credential={}/{},SignedHeaders={},Signature={}'.format(
ALGORITHM,
access_key,
credential_scope,
signed_headers,
signature
)

headers = {'Content-Type': content_type,
'X-Amz-Date': amz_date,
'X-Amz-Target': amz_target,
'Authorization': authorization_header}

return headers


defmain():
service = 'translate'
region = 'ap-northeast-1'

host = get_host(service, region)
endpoint = 'https://{}/'.format(host)

text = 'Amazon Translate is a text translation service that use '\
'advanced machine learning technologies to provide high-quality '\
'translation on demand. You can use Amazon Translate to translate '\
'unstructured text documents or to build applications that work in '\
'multiple languages.'\
'Amazon Translate provides translation between a source language '\
'(the input language) and a target language (the output language). ' \
'A source language-target language combination is known as a '\
'language pair.'

source_lang_code = 'en'
target_lang_code = 'zh'

request_parameters = '{{"{}": "{}","{}": "{}","{}": "{}"}}'.format(
"Text",
text,
"SourceLanguageCode",
source_lang_code,
"TargetLanguageCode",
target_lang_code
)

headers = get_header(service, region, request_parameters)
# print('endpoint is ==>\n{}\n'.format(endpoint))
# print('request_parameters is ==>\n{}\n'.format(request_parameters))
# print('headers is ==>\n{}\n'.format(headers))

try:
res = requests.post(
endpoint,
data=request_parameters,
headers=headers
)
except Timeout:
print('The request timed out')
except HTTPError as http_err:
print(f'HTTP error occurred: {http_err}')
except Exception as err:
print(f'Other error occurred: {err}')
else:
json_content = json.loads(res.text)
print('The original is -->\n{}\n'.format(text))
print('The translation is -->\n{}\n'.format(
json_content['TranslatedText']
))
# print('Response:\n\t{}'.format(res.text))


if__name__ == "__main__":
main()

如果不喜欢Python也没有关系。即使shell的脚本仅仅使用curl、openssl以及sed，就可以实现上传文件到Amazon S3的存储桶之中的操作

content-type:${contentType}
host:${bucket}${baseUrl}
x-amz-content-sha256:${payloadHash}
x-amz-date:${dateValueL}
x-amz-server-side-encryption:AES256
x-amz-storage-class:${storageClass}

${headerList}
${payloadHash}"

# Hash it
canonicalRequestHash=$(printf '%s'"${canonicalRequest}"| openssl dgst -sha256 -hex 2>/dev/null | sed 's/^.* //')

# 2. Create string to sign
stringToSign="\
${authType}
${dateValueL}
${dateValueS}/${region}/${service}/aws4_request
${canonicalRequestHash}"

# 3. Sign the string
signature=$(awsStringSign4 "${awsSecret}""${dateValueS}" "${region}" "${service}" "${stringToSign}")

# Upload
curl -s -L --proto-redir =https -X "${httpReq}"-T "${fileLocal}" \
-H "Content-Type: ${contentType}" \
-H "Host: ${bucket}${baseUrl}" \
-H "X-Amz-Content-SHA256: ${payloadHash}" \
-H "X-Amz-Date: ${dateValueL}" \
-H "X-Amz-Server-Side-Encryption: AES256" \
-H "X-Amz-Storage-Class: ${storageClass}" \
-H "Authorization: ${authType} Credential=${awsAccess}/${dateValueS}/${region}/${service}/aws4_request, SignedHeaders=${headerList}, Signature=${signature}" \
"https://${bucket}${baseUrl}/${fileRemote}"

On paper, I feel shallow at the end, and I absolutely know that this matter has to be done. Initially starting to read the documentation for Signature Version 4 felt cumbersome and almost impossible to stick to. After repeated setbacks, especially the example of the S3 upload implemented by the script tortured me for a day. But when I successfully completed a few examples, I immediately felt that I had mastered it, and I couldn't stop it. This little practice gave me a better understanding of the design and implementation of Amazon API.

References

Signatura Version：
https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/API/API_PutObject.html

Amazon-Cli：
https://aws.amazon.com/it/cli/

Author of this article

Fei Lianghong

Amazon Web Services Principal Developer Advocate

In the past 20 years, he has been engaged in the fields of software architecture, program development and technology promotion. He often speaks and shares at various technical conferences, and he is also an enthusiastic participant in several technical communities. He is good at the development of web applications, mobile applications and machine learning, and has also been engaged in the design, development and project management of many large-scale software projects. Currently, he focuses on cloud computing and the Internet and other technical fields, and is committed to helping Chinese developers build a new generation of Internet applications based on cloud computing.