How to reduce the explosion radius of security vulnerabilities and achieve zero-trust security between services?

Author: Wang Xining,

Recently, many security agencies at home and abroad have detected arbitrary code execution vulnerabilities in Apache Log4j (Vulnerability No.: CVE-2021-44228). Users who have not obtained identity authentication can send data requests remotely to enter data logs and easily trigger the vulnerability. Execute arbitrary code on the target.

As we all know, Log4j is widely used in multiple microservice application frameworks. These numerous distributed microservices will also increase security challenges. Each microservice is a target of attack. Kubernetes provides an excellent platform for hosting and orchestrating user microservices. However, all interactions between microservices are not secure by default. They communicate via plain text HTTP, but this is not enough to meet security requirements. Relying on the network boundary to ensure security is not enough, because once a certain internal service is compromised, the boundary security measures are like the Magino line of defense, and attackers can use the machine as a springboard to attack the internal network. Therefore, internal calls must also be safe. This is where zero trust comes in.

Zero trust is proposed by Forrester analyst John Kindervag, which means that there is no implicit trust whether inside or outside the network boundary. In other words, explicit authentication is required everywhere, and the principle of least privilege is used to restrict access to resources.

An important value proposition of service grid technology is how it effectively protects the application production environment without reducing the productivity of developers. Through the service grid technology, it provides the necessary foundation for the micro-service architecture to adopt the zero-trust network security method, so as to achieve the security goals of strong identity verification, context-based authorization, and record monitoring for all accesses. Using these grid functions, you can provide security control capabilities for all applications belonging to the grid, for example, all traffic is encrypted, all traffic to the application passes the verification of the policy enforcement point (PEP), etc.

The "Kubernetes Hardening Guidance" issued by the National Security Agency (NSA) in August 2021 (see related link 1 at the end of the article for details) also mentioned that administrators should consider using service mesh to strengthen the security of Kubernetes clusters.

Alibaba Cloud Service Grid ASM (see related link 2 at the end of the article for details) has become one of the important cloud-native zero-trust system landing carriers, offloading identity verification and authorization from the application code to the service grid, out of the box, dynamic It is easier to configure and update the strategy and take effect immediately. Based on the use of Kubernetes Network Policy to achieve three-layer network security control, the service mesh ASM provides a policy control based on OPA (Open Policy Agent) that includes peer-to-peer identity and request identity authentication capabilities, Istio authorization policies, and more refined management ability. The zero-trust security capabilities provided by Alibaba Cloud Service Grid ASM help users achieve these security goals.

The construction of a service grid-based zero-trust security capability system includes the following aspects:

1. The basis of zero trust-workload identity; how to provide a unified identity for cloud-native workloads; ASM products provide easy-to-use identity definitions for each workload under the service grid, and provide customized mechanisms for expansion according to specific scenarios Identity construction system, compatible with the community SPIFFE standard at the same time;
2. Zero-trust carrier-security certificate, ASM products provide mechanisms for how to issue certificates and manage the life cycle and rotation of certificates. The identity is established through X509 TLS certificates, and each agent uses this certificate. And provide certificate and private key rotation;
3. Zero-trust engine-policy execution. A policy-based trust engine is the key to building zero trust. In addition to supporting the Istio RBAC authorization strategy, ASM products also provide a more fine-grained authorization strategy based on OPA;
4. Zero-trust insight-visualization and analysis, ASM products provide an observable mechanism to monitor the logs and indicators of the execution of the strategy, to judge the execution of each strategy, etc.;

Why use service grid to achieve zero trust?

Compared with the traditional method of building these security mechanisms directly in the application code, the service grid architecture has the following security benefits:

• The life cycle of Sidecar agents remains independent of the application, so it is easier to manage these Sidecar agents.
• Allows for dynamic configuration, making it easier to update the strategy, and the update takes effect immediately without the need to redeploy the application.
• The centralized control architecture of the service grid enables enterprise security teams to construct, manage, and deploy security policies applicable to the entire enterprise, thereby ensuring the security of business applications built by application developers by default. They can use these security policies immediately without additional work.
• The service grid provides the ability to authenticate the end user credentials attached to the request, such as JWT.
• In addition, using the service grid architecture, the authentication and authorization system can be deployed as a service in the grid. In this way, just like other services in the grid, these security systems can also obtain security guarantees from the grid itself, including encryption, identity recognition, policy enforcement points, and end-user credential authentication and authorization during transmission.

With the help of Alibaba Cloud Service Grid ASM, a single control plane can be used to implement strong identity and access management, transparent TLS and encryption, authentication and authorization, and audit logging. Alibaba Cloud Service Grid ASM provides these features out of the box, and the simplicity of installing and managing it allows developers, system administrators, and security teams to properly protect their microservice applications.

The Zero Trust System in ASM Service Grid of Alibaba Cloud

The service grid can reduce the attacked area in the cloud native environment and provide the basic framework required by the zero-trust application network. Through the ASM management service-to-service security, you can ensure the end-to-end encryption, service-level identity authentication, and fine-grained authorization policies of the service grid.

Under the service grid system, it can support:

• Implement mutual TLS authentication or server-oriented TLS authentication between services, and support life cycle management such as automatic rotation of certificates. All communications within the grid are authenticated and encrypted.
• Enable fine-grained authorization based on identity and authorization based on other dimension parameters. Based on the basis of role access control (RBAC), it supports the "least privilege" position, that is, only authorized services can communicate with each other according to ALLOW/DENY rules.

Currently, the ASM service grid of Alibaba Cloud provides the following basic capabilities of zero trust security:

Workload identity

When the application runs in the service grid environment, the service grid provides a unique identifier for each service. This identifier will be used when connecting to other microservices running in the service grid. The service ID can be used for two-way verification of services to verify whether access between services is allowed, and the service ID can also be used in authorization policies.

When using the service grid ASM to manage workloads running on Kubernetes or define virtual machine workloads based on WorkloadEntry, ASM provides a service identity for each workload; the identity is implemented based on the service account token of the workload.

The service identity in ASM complies with the SPIFFE standard and has the following format: spiffe://<trust-domain>/ns/<namespace>/sa/<service-account>

On the service grid ASM console, open the corresponding ASM instance, and you can see the following workload identity under zero trust security in the left navigation bar.

The workload and its identity definition under the Kubernetes cluster in the data plane:

Define the virtual machine workload and its identity definition based on WorkloadEntry:

Peer authentication

Authentication refers to identity: who is this service? Who is this end user? Can I believe that they are what they say?

ASM products provide two types of authentication:

• Peer-to-peer authentication-When two microservices interact with each other, whether to enable or disable mutual TLS for peer-to-peer authentication.
• Request authentication-Allow end users and systems to interact with microservices using request authentication. Usually use JSON Web Token (JWT) to perform this operation

Install and deploy the bookinfo example according to the Getting Started Guide (see related link 3 at the end of the article for details).

First, when trying to access the details service using pure HTTP from the productpage pod in the same namespace (for example, default in this example), the request should be successfully returned with status 200 by default. This is because by default, both TLS and plain text traffic can be accepted.

kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl http://details:9080/details/1 -o /dev/null -s -w '%{http_code}\n'

Next, define peer authentication under the namespace default.

On the service grid ASM console, open the corresponding ASM instance, and you can see the following peer-to-peer authentication under zero trust security in the left navigation bar. On the right page, click the "New Mutual MTLS Mode" button to define the mTLS mode for workload details as STRICT.

Once again, when using the productpage pod to access the details service using pure HTTP:

kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl http://details:9080/details/1 -o /dev/null -s -w '%{http_code}\n'
000
command terminated with exit code 56

Exit code 56 means failure to receive network data. This is in line with expectations. The workload details define the mTLS mode as STRICT, so that TLS certificate authentication is required in each request.

In order to allow normal access, the peer-to-peer identity authentication defined above can be modified from STRICT to PERMISSIVE. The corresponding YAML is defined as follows:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: details-strict
  namespace: default
spec:
  mtls:
    mode: PERMISSIVE
  selector:
    matchLabels:
      app: details

\

Request authentication

First, we will create a request authentication policy to enforce JWT authentication for inbound requests of the details service. On the service grid ASM console, open the corresponding ASM instance, and in the left navigation bar, you can see the following request for identity authentication under zero trust security. On the right page, click the "New" button to define jwt rules for workload details.

Among them, the issuer value is set to "[e-mail]", and the value of jwks is taken from security/tools/jwt/samples/jwks.json in the Istio installation file, corresponding to the following

{ "keys":[ {"e":"AQAB","kid":"DHFbpoIUqrY8t2zpA2qXfCmr5VO5ZEr4RzHU_-envvQ","kty":"RSA","n":"xAE7eB6qugXyCAG3yhh7pkDkT65pHymX-P7KfIupjf59vsdo91bSP9C8H07pSAGQO1MV_xFj9VswgsCg4R6otmg5PV2He95lZdHtOcU5DXIg_pbhLdKXbi66GlVeK6ABZOUW3WYtnNHD-91gVuoeJT_DwtGGcp4ignkgXfkiEm4sw-4sfb4qdt5oLbyVpmW6x9cfa7vs2WTfURiCrBoUqgBo_-4WTiULmmHSGZHOjzwa8WtrtOQGsAFjIbno85jp6MnGGGZPYZbDAa_b3y5u-YpW7ypZrvD8BgtKVjgtQgZhLAGezMt0ua3DRrWnKqTZ0BJ_EyxOGuHJrLsn00fnMQ"}]}

Then, when trying to use the productpage pod to access the details service using pure HTTP, you can see that the returned result is 200.

The value of the variable TOKEN is set to:

export TOKEN=eyJhbGciOiJSUzI1NiIsImtpZCI6IkRIRmJwb0lVcXJZOHQyenBBMnFYZkNtcjVWTzVaRXI0UnpIVV8tZW52dlEiLCJ0eXAiOiJKV1QifQ.eyJleHAiOjQ2ODU5ODk3MDAsImZvbyI6ImJhciIsImlhdCI6MTUzMjM4OTcwMCwiaXNzIjoidGVzdGluZ0BzZWN1cmUuaXN0aW8uaW8iLCJzdWIiOiJ0ZXN0aW5nQHNlY3VyZS5pc3Rpby5pbyJ9.CfNnxWP2tcnR9q0vxyxweaF3ovQYHYZl82hAUsn21bwQd9zP7c-LS9qd_vpdLG4Tn1A15NxfCjp5f7QNBUo-KC9PJqYpgGbaXhaGx7bEdFWjcwv3nZzvc7M__ZpaCERdwU7igUmJqYGBYQ51vr2njU9ZimyKkfDe3axcyiBZde7G6dabliUosJvvKOPcKIWPccCgefSj_GNfwIip3-SsFdlR7BtbVUcqR-yv-XOxJ3Uc1MI0tz3uMiiZcyPV7sNCU4KRnemRIMHVOfuvHsU60_GhGbiSFzgPTAa9WTltbnarTbxudb_YEOx12JiwYToeX0DCPb43W1tzIBxgm8NxUg
kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl http://details:9080/details/1 -o /dev/null --header "Authorization: Bearer $TOKEN" -s -w '%{http_code}\n'
200

If an invalid token is passed, we should see a "401: Unauthorized" response:

kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl http://details:9080/details/1 -o /dev/null --header "Authorization: Bearer badtoken" -s -w '%{http_code}\n'
401

However, if we don't pass the token at all, RequestAuthentication will not call the policy. Requests that do not use JWT Token also return 200.

kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl http://details:9080/details/1 -o /dev/null  -s -w '%{http_code}\n'
200

Therefore, in addition to this authentication strategy, we also need an authorization strategy that requires JWT for all requests. The next part will describe how to define authorization policies in ASM products.

Authorization strategy

The ASM product provides an authorization policy. You can use the AuthorizationPolicy resource to activate the authorization mechanism between microservices, and use the following to establish an appropriate traffic authorization policy mechanism:

• Select the selector field of the workload label to specify the strategy target;
• The action field specifies whether it is an ALLOW or DENY request. If you do not specify an operation, the operation defaults to ALLOW. For clarity, we recommend that you always specify actions. (The authorization policy also supports AUDIT and CUSTOM operations);
• rules specify when to trigger the action:
• • The from field in rules specifies the source of the request;
  • The to field in rules specifies the requested operation;
  • The when field specifies other conditions that need to be met in order to apply the rule;

The corresponding YAML is defined as follows:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: require-jwt
  namespace: default
spec:
  action: ALLOW
  rules:
    - from:
        - source:
            requestPrincipals:
              - [e-mail]/[e-mail]
  selector:
    matchLabels:
      app: details

Send the request again without using the JWT Token, and you should now see 403-Forbidden. This is where the AuthorizationPolicy takes effect, and all front-end requests must have a JWT Token.

 kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl http://details:9080/details/1 -o /dev/null  -s -w '%{http_code}\n'
403

OPA strategy

As an incubation project hosted by CNCF (see related link 4 at the end of the article for details), Open Policy Agent (OPA) (see related link 5 at the end of the article for details) is a strategy engine that can be used to achieve fine-grained access to your applications control. As a general policy engine, OPA can be deployed as an independent service together with microservices. In order to protect the application, each request to the microservice must be authorized before it can be processed. In order to check authorization, the microservice makes an API call to OPA to determine whether the request is authorized.

The service grid ASM integrates the Open Policy Agent (OPA) plug-in, and defines access control policies through OPA, which enables your application to achieve fine-grained access control and supports dynamic update of OPA policies.

For details, please refer to "Achieving Dynamic Update OPA Strategy in ASM" (for details, please refer to the related link 6 at the end of the article)

Zero trust performance improvement

Intel® has introduced many encryption and decryption acceleration technologies and architecture innovations that we call Crypto Acceleration into the new third-generation Intel® Xeon® Scalable Processors (ICELAKE), which greatly improves the performance of some key encryption and decryption algorithms. The AVX512 instruction set used by the third-generation Intel Xeon processor and Multi-Buffer technology provides several different instance types in the seventh-generation Alibaba Cloud ECS server. Based on these instruction sets, coupled with Envoy upstream community capabilities that incorporate MultiBuffer technology, the ASM product of Alibaba Cloud Service Grid provides TLS encryption and decryption performance optimization capabilities based on MB technology.

Combine Intel's open source software libraries, such as IPP encryption library, IPsec multi-buffer encryption library (intel-ipsec-mb), Intel® QuickAssist Technology (Intel® QAT) and OpenSSL engine. These solutions significantly improve encryption operations, such as TLS connection handshake performance. These new components can handle multiple TLS private key operations in parallel. With the asynchronous private key processing mechanism available in both OpenSSL and BoringSSL, application software can submit handshake private key requests without waiting for one request to return before submitting another. In turn, once it is ready, a callback is called for each request. Below, the multi-buffer encryption process can operate 8 such asynchronously submitted private keys, and use AVX512 SIMD (single instruction multiple data) instructions to process in parallel, which greatly improves the overall application performance.

The performance optimization switch can be enabled with one click in the Alibaba Cloud ASM console. Currently, this function has been opened to the public in the latest version of the product.

At the KubeCon 2021 conference, Alibaba Cloud Service Grid ASM leader Wang Xining and Intel Software and Advanced Technology Cloud Computing Team Hu Wei will bring you a special technology sharing, then we will introduce more technologies Details and practices. (See the end of the article 7 for the technical sharing link)

Summary and reference cases

In summary, the service mesh ASM provides the following components to enhance security:

• Provide a managed certificate infrastructure with complete certificate life cycle management, which solves the complexity of certificate issuance and CA rotation;
• Managed control plane API, used to distribute authentication strategy, authorization strategy and security naming information to Envoy agent;
• The Sidecar agent helps ensure the security of the grid by providing the policy enforcement point PEP;
• Envoy proxy extension allows telemetry data collection and auditing;

Each workload establishes an identity through an X509 TLS certificate, which is used by each Sidecar agent. The service mesh ASM provides and periodically rotates certificates and private keys. If a particular private key is stolen, the service mesh will quickly replace it with a new private key, thus greatly reducing the attack surface.

References

1. Use authorization policies to implement IP-based access control on the ingress gateway (see related link 8 at the end of the article for details), or access control based on custom external authorization, as shown in the figure below. Cloud product cloud native application delivery platform ADP (see details for details) Related links at the end of the article 9) Implementation of gateway authorization strategy based on ASM Service Grid of Alibaba Cloud.

2. An interconnected financial customer used the authorization policy provided by Alibaba Cloud Service Grid ASM to isolate the external connection area and the application area in order to solve the access control of cross-cluster multilingual applications. At the same time, it can be combined with the egress gateway to audit the grid traffic, and in conjunction with the authorization strategy, it can also control the application's access to third-party services.

Related Links

1）《Kubernetes Hardening Guidance》：

​​https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF​​

2) Alibaba Cloud Service Grid ASM:

​​https://www.aliyun.com/product/servicemesh​​

3) Getting started:

​​https://help.aliyun.com/document_detail/149552.html​​

4）CNCF：

​​https://www.cncf.io/ ​​

5) Open Policy Agency (OPA):

​​https://www.openpolicyagent.org/ ​​

6) "dynamic updating of the OPA strategies in ASM":

​​https://help.aliyun.com/document_detail/277428.html​​

7) KubeCon technology sharing:

​​https://kccncosschn21.sched.com/event/rxVz/leng-ssxian-zhi-qiang-tang-jie-asmreintel-multi-bufferjiong-zha-zhi-hui-sponsored-session-using-alibaba-cloud-service-mesh-asm-and-intel-multi-buffer-technology-to-achieve-faster-encrypted-communication-between-application-services​​

8) Use authorization policies to achieve access control

​​https://help.aliyun.com/document_detail/214764.html?spm=a2c4g.11186623.6.613.6b213918q0rlZV#title-f15-wyq-fxg​​

9) Cloud native application delivery platform ADP:

​​https://adp.console.aliyun.com/ ​​