Amazon Kinesis Data Streams enables cross-account application log collection

Overview

In our previous article, we introduced how to use Fluent Bit to combine Amazon Kinesis Data Streams, Amazon Lambda, and Amazon OpenSearch Search to build an application logging system under the same account. However, in some scenarios, the enterprise may have multiple accounts. Due to the needs of enterprise internal control, the administrator hopes to be able to transfer the application logs under each account to the specified account through Private Link for centralized log processing or auditing. The use of Amazon Lambda to write logs into Amazon OpenSearch Service in this article is based on the content introduced in the previous blog. If you have not read "Using Fluent Bit and Amazon OpenSearch Service to Build a Log System" before, it is recommended that you click here. Read https://aws.amazon.com/cn/blogs/china/build-a-logging-system-with-fluent-bit-and-amazon-opensearch-service/ . Through this article, you will learn how to use Fluent Bit to combine services such as Amazon STS AssumeRole and Amazon Kinesis Data Streams to perform centralized log processing in cross-account scenarios. After reading this article, you will learn:

• How to Configure Cross-Account Log Transfer Permissions via Amazon Role Trust Relationship and Amazon STS AssumeRole
• How to transfer logs across accounts with Amazon for Fluent Bit 2.21.1

brief introduction

Next, we will explain the entire implementation process in detail based on the overall architecture of the following figure, taking us-west-2 as an example.

Before we start, let's consider such a scenario. We deployed Nginx in EC2 of Amazon Web Services Account A to build our Web Server, and sent the Nginx logs of Amazon EC2 of Account A to Amazon Kinesis Data Stream of Account B through Private Link. For the creation of Amazon Lambda and Amazon Kinesis Data Stream, VPC Endpoint, and Amazon OpenSearch Service for account B, you can refer to our previous articles.

1. How to configure cross-account log transfer permissions through Amazon Role Trust Relationship and Amazon STS AssumeRole

We first log in to the B account, here we use the us-west-2 region, and create a role in the IAM service, as shown in the following figure: Select another account at the trusted entity type, and enter the ID of account A.

Click Next to create a policy. The policy document reference is as follows:

 { 
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kinesis:PutRecord",
                "kinesis:PutRecords"
            ],
            "Resource": "arn:aws:kinesis:us-west-2:<YOUR ACCOUNT ID>:stream/LogStream"
        }

        ]
}

Swipe left to see more

Note: Please replace it with the B account ID you use, and the created role name is demo-writing-kds.

• Log in to the A account and also select the us-west-2 region. Create Amazon EC2 and install Fluent Bit. The installation steps can also refer to the "Installing and Configuring Amazon for Fluent Bit 2.21.1" section of our previous article. During the installation process, we can set the content of the /etc/fluent-bit/conf/fluent-bit.conf file to be empty.
• Create a new IAM role for Amazon EC2, select Amazon EC2 for "use cases", click Next, create a policy, configure Amazon STS AssumeRole, and refer to the policy document as follows:

 { 
    "Version": "2012-10-17",
    "Statement": [{
        "Sid": "VisualEditor0",
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "<demo-writing-kds>"
    }]
}

Swipe left to see more

Note: Please replace it with the ARN of the demo-writing-kds role you created in the previous step, which is in the B account.

2. Configure Amazon for Fluent Bit 2.21.1 to transfer logs across accounts through Private Link

We still choose the us-west-2 region under the A account. Note that the following scenarios do not require an Amazon Kinesis Data Stream to be created under the A account.

• Let Amazon EC2 access the B account's Amazon Kinesis Data Streams through Private Link. Create a VPC Endpoint for Amazon Kinesis Data Streams. For simplicity, we can choose the VPC where Amazon EC2 is located when choosing a VPC.

If a new security group is created, the inbound rule of the security group needs to be added to allow the security group from Amazon EC2 to access port 443. For the inbound rule, refer to the following figure:

• Log in to Amazon EC2, edit the /etc/fluent-bit/conf/fluent-bit.conf file, and configure Arn for the output plugin "kinesis_streams" of Fluent Bit. The content of the complete configuration file is as follows:

 [ SERVICE]
    Flush           5
    Daemon          off
    Log_Level       info
    Log_File        /tmp/fluent-bit.log
    Storage.path    /tmp/chunks-buffer
    Storage.backlog.mem_limit 5M
[INPUT]
    Name tail
    # log file location
    Path /var/log/nginx/access.log
    Tag nginx-log
    Read_from_head true
    #use this as checkpoint
    DB /tmp/checkpoint-nginx.db
    DB.Sync Normal
[OUTPUT]
    Name kinesis_streams
    # Match Tag
    Match nginx-log
    # AWS region
    Region us-west-2
    # Kinesis stream name
    Stream LogStream
    Arn <demo-writing-kds>
    Retry_Limit False
    Auto_retry_requests True

Swipe left to see more

Again, we need to replace the ARN with the demo-writing-kds role

• Start Fluent Bit, check the log, and verify whether it is successful.

• Log in to the Amazon OpenSearch Service dashboard, and you can see the logs written.

Note: If you want to know how to consume log messages from Amazon Kinesis Data Stream through Amazon Lambda and write the logs to Amazon OpenSearch Service, you can click here. For how to use Nginx proxy to access OpenSearch dashboard in VPC through public network, you can refer to our official documentation https://docs.aws.amazon.com/opensearch-service/latest/developerguide/dashboards.html .

Summarize

In this article, we introduce how to use the open source log collection tool Fluent Bit to combine Amazon Role Trust Relationship and Amazon STS AssumeRole in a multi-account scenario, and by using VPC Endpoint, we can also use the intranet to access Kinesis Data Streams, thereby Meet the needs of enterprise internal control. In actual use, we also need to pay attention to that when using VPC Endpoint to connect Kinesis Data Streams across accounts, the corresponding services under the two accounts must be in the same Region. Enterprises can decide whether to enable VPC Endpoint based on their actual situation.

References

• VPC endpoint information:

• Fluent Bit data:

Author of this article

Ma Tao

Amazon Cloud Technology Solution R&D Architect

Mainly responsible for the design and development of cloud solutions.