Prevent, Detect, and Respond to Log4j Vulnerabilities with Amazon Cloud Security Services

Overview

In this article, we will provide some guidance for customers affected by the recently disclosed Log4j vulnerability. Topics include how to limit the risk of a vulnerability, how to try to identify if you are vulnerable to this issue, and how to update your infrastructure with appropriate patches.

The Log4j vulnerability (CVE-2021-44228, CVE-2021-45046) is a critical vulnerability (CVSS 3.1 base score of 10.0) in the ubiquitous logging platform Apache Log4j. This vulnerability allows an attacker to execute remote code on vulnerable platforms. Log4j version 2 between versions 2.0-beta-9 and 2.15.0 is affected.

The vulnerability uses the Java Naming and Directory Interface (JNDI), which is used by Java programs to look up data, typically through a directory, and in the case of this vulnerability, typically through an LDAP directory.

Figure 1 below highlights the Log4j JNDI attack flow.

Figure 1. Log4j attack process. Source: GovCERT.ch, Swiss Government Computer Emergency Response Team (GovCERT)

This post is intended as an immediate response to your interest in this tool designed to hot-patch a running JVM with any Log4j 2.0+. Steve Schmidt, chief information security officer at Amazon Cloud Technologies, also discussed the hotfix.

defense

Customers can use several services provided by Amazon Cloud Technologies to help limit the risk/exposure from Log4j vulnerabilities. Clients can establish tiered controls and/or select the controls identified below to help limit their exposure.

Amazon WAF

Customers can use Amazon WAF, following their appropriate hosting rules, to help protect their Amazon CloudFront distribution, Amazon API Gateway REST API, Amazon ALB, or Amazon AppSync GraphQL API resources.

• AWSManagedRulesKnownBadInputsRuleSet esp.: Log4JRCE rules that help check requests for Log4j vulnerabilities. Example patterns include ${jndi:ldap://example.com/}.
• AWSManagedRulesAnonymousIpList esp.: AnonymousIPList rule that helps to check the IP addresses of known sources of anonymized client information.
• AWSManagedRulesCommonRuleSet esp.: SizeRestrictions_BODY rule that verifies that the request body size is at most 8 KB (8192 bytes).

For customers using Amazon WAF Classic, you need to migrate to Amazon WAF or create custom regular expression match conditions.

Customers with multiple accounts can follow the instructions to deploy Amazon WAF rules centrally across Amazon Organizations using Amazon Firewall Manager.

Amazon Network Firewall

Customers can deploy network-based detection and protection using Suricata-compatible IDS/IPS rules in Amazon Network Firewall. Open source Suricata rules addressing Log4j are available from corelight, NCC Group, ET Labs, and CrowdStrike. These rules help identify scans and post-attack activities for Log4j vulnerabilities. Due to the large number of benign scans currently existing, we recommend that customers focus first on potential post-attack activity, such as sending outbound LDAP traffic from their VPC to untrusted internet destinations.

We also recommend that customers consider implementing outbound port/protocol enforcement rules to monitor or prevent instances such as LDAP from using non-standard LDAP ports such as 53, 80, 123, and 443. For monitoring or preventing outbound use of port 1389, it can be helpful to identify systems that are triggered by Internet scanners to make outbound command and control calls. We also recommend that systems with no legitimate business need to initiate network calls to the Internet should not be given this capability by default. Outbound network traffic filtering and monitoring is very useful not only for Log4j, but also for other types of vulnerabilities.

using IMDSv2

In the early days of the log4j vulnerability, researchers noticed that once a host was compromised by the initial JDNI vulnerability, attackers would sometimes attempt to obtain credentials from the host and send those credentials through mechanisms such as LDAP, HTTP, or DNS lookups. We recommend that customers use IAM roles instead of long-term access keys, and do not store sensitive information such as credentials in environment variables. Customers can also utilize Amazon Secrets Manager to store and automatically rotate database credentials instead of storing long-term database credentials in environment variables on the host.

To help Amazon Web Services protect against such attacks and to help keep all IMDS data private when Amazon EC2 roles may be used, customers should consider the need to use Instance Metadata Service Version 2 (IMDSv2). Since IMDSv2 is enabled by default, IMDSv2 can be required by disabling IMDSv1 (which is enabled by default). With IMDSv2, requests are protected by an initial interaction in which the calling process must first obtain a session token using HTTP PUT, and subsequent requests must include this token in the HTTP header. This makes it more difficult for an attacker to obtain credentials or any other data from IMDS. While all the latest Amazon SDKs and tools support IMDSv2, as with any potentially backwards-incompatible change, test this change on a representative system before widespread deployment.

detect

This article describes how to potentially limit the ability to exploit this vulnerability. Next, we'll highlight which Amazon Cloud Services services can help detect this vulnerability in your environment.

Amazon Inspector

As shown in Figure 2, the Amazon Inspector team created a coverage to identify the existence of this vulnerability in Amazon EC2 instances and Amazon Elastic Container Registry Images (Amazon ECR) (Amazon ECR). With the new Amazon Inspector, scans are automated and continuous. Continuous scanning is driven by events such as new packages released, new instances, and new Common Vulnerability Exposures (CVEs).

For example, as soon as the Inspector team added support for the Log4j vulnerability (CVE-2021-44228 and CVE-2021-45046), the Inspector immediately began to look for this vulnerability in all supported Amazon Systems Manager managed instances where Log4j via the operating system package manager installed, and this package exists in a Maven-compatible Amazon ECR container image. If this vulnerability exists, findings will start showing without any manual action. If you are using Inspector Classic, you need to make sure to run the assessment against all Amazon EC2 instances. You can follow the documentation to ensure that evaluation targets are created for all Amazon EC2 instances.

GuardDuty

In addition to discovering this vulnerability through the Inspector, the Amazon GuardDuty team has also started adding indicators of compromise related to exploiting the Log4j vulnerability and will continue to do so. GuardDuty will monitor attempts to access known bad IP addresses or DNS entries, and can also detect post-attack activity based on anomalous behavioral findings. For example, if an Amazon EC2 instance starts communicating on an unusual port, GuardDuty will detect this activity and create a lookup behavior: EC2/NetworkPortUnusual. However, this activity is not limited to NetworkPortUnusual results. GuardDuty has a number of different results related to post-attack activity that could be a response to compromised Amazon cloud technology resources.

Security Hub

Today, many customers also use Amazon Security Hub, which integrates with Inspector and GuardDuty, to aggregate alerts and enable automated remediation and response. In the short term, we recommend that you use Amazon Security Hub to set up alerts via Amazon Chatbot, Amazon Simple Notification Service, or the ticketing system to see when the Inspector finds this vulnerability in your environment. In the long run, we recommend that you use Security Center when appropriate to enable automatic remediation and response to security alerts. Here are some ideas on how to set up automatic remediation and response using Security Center.

VPC flow logs

Customers can query their VPC traffic logs using Athena or CloudWatch Logs Insights queries to help identify VPC resources related to outbound network activity following a log4j attack. Version 5 of the VPC traffic log is especially useful because it includes a "flow to" field. We recommend that customers first pay special attention to outbound network calls using destination port 1389, as outbound use of this port is less common in legitimate applications. Customers should also investigate outbound web calls to untrusted Internet destination IP addresses using destination port 389. Free tier IP reputation services, such as VirusTotal, GreyNoise, NOC.org, and pinfo.io, can provide useful insights about public IP addresses found in recorded activity.

Note : If you have a Microsoft Active Directory environment in the captured VPC flow logs of the query, you may get false positives due to the use of port 389.

responds to

The first two sections discussed ways to help prevent potential attack attempts, and how to detect the presence of vulnerabilities and potential attack attempts. In this section, we'll focus on steps you can take to mitigate this vulnerability. As we pointed out in the overview, it is recommended to respond to the blog immediately and use tools designed to hot-patch a running JVM with any Log4j 2.0+. Steve Schmidt, chief information security officer at Amazon Cloud Technologies, also discussed the hotfix.

Amazon Patch Manager

If you use Amazon Systems Manager Patch Manager and have set up a critical patch to install immediately in the patch baseline, your Amazon EC2 instance will already have that patch. It's worth noting that you haven't finished this step yet. Next, you need to update the classpath wherever the library is used in your application code to ensure you are using the latest version.

Container Mitigation

To install the hotfix mentioned in the overview onto Amazon EKS cluster worker nodes, Amazon Cloud Technologies has developed an RPM that performs a JVM-level hotfix that disables JNDI lookups from the Log4j2 library. ApacheLog4J2 node agent is an open source project built by the Kubernetes team of Amazon Cloud Technology.

Once identified, the Amazon ECR container image will need to be updated to use Log4j version 2.16.0. Downstream, you need to ensure that all containers built with the vulnerable Amazon ECR container image are updated to use the new image as soon as possible. This may vary depending on the service you use to deploy these images. For example, if you are using Amazon Elastic Container Service (Amazon ECS), you may want to update the service to force a new deployment, which will destroy the image with the new Log4j version. Check the documentation that supports the method you use to deploy the container.

We recommend that customers provide new application credentials and revoke existing credentials immediately after patching.

Mitigation strategy when upgrade cannot be

If you cannot upgrade to version 2.16.0, which disables access to JDNI by default, or if you are still determining a strategy for patching your environment, you can mitigate this vulnerability by changing your Log4j configuration. To implement this mitigation in >= 2.10 versions, you need to remove the JndiLookup class from the classpath: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

Conclusion

In this article, we outline key Amazon Cloud Security services that enable customers to take a layered approach to help prevent, detect, and respond to risks from Log4j vulnerabilities. We urge customers to continue to monitor our security advisories; we will continue to update our advisories with fixes for our shared responsibility model.

Given the severity of this vulnerability, we urge customers to pay close attention to this vulnerability and appropriately prioritize the implementation of the controls highlighted in this blog.

The author of this article

Marshall Jones

Global Security Expert Solutions Architect at Amazon Cloud Technologies

His background is in Amazon cloud technology consulting and security architecture, focusing on various security areas such as edge, threat detection and compliance. Today, he focuses on helping Amazon Cloud Enterprise customers adopt and operate Amazon Cloud Security Services to improve security efficiency and reduce risk.

Syed Shareef

Senior Security Solutions Architect at Amazon Cloud Technologies

He works with large financial institutions to help them achieve their business goals with Amazon Cloud Technology while complying with regulatory and security requirements.

Scan the QR code above to register now