A gimmick caused by a certificate

- lead-

I didn't expect to encounter certificate-related problems several times in the past half year of Sensors Data.

- rise-

On September 3, 2021, a new customer was connected to our SaaS system. In a certain link, we will send an HTTPS request to the customer, but unexpectedly encountered a SSLHandshakeException :

Caused by: javax.net.ssl.SSLHandshakeException: ... unable to find valid certification path to requested target

Tried it with curl on the server, and also reported an error:

$ curl -v https://some.domain/
CAfile: /etc/pki/tls/certs/ca-bundle.crt
...
curl: (60) Peer's Certificate issuer is not recognized.

But there is no problem opening this URL with a browser, which means that the problem should be on our server side.

- Analysis-

We know that HTTPS relies on certificates to ensure communication security; but how does the client ensure that the certificate given by the server is credible?

Since certificates are always issued by a certificate authority (Certificate issuer, or Certificate Authority, abbreviated as CA), if we store a batch of trusted certificate authorities locally in advance, we can judge the certificate when initiating the request. Is it believable.

Sometimes the situation is a little more complicated: some authority is not in our list, but his certificate is issued by an authority we trust, and we also consider him to be trusted, so the certificate issued by him is also trusted.

So this constitutes a chain of trust. The end of the chain is the "Root Certificate Authority" (Root CA), which are usually large internationally recognized and reliable institutions, or institutions endorsed by national authorities.

Understanding this, it can be speculated that the list of authorities on our server is not updated in time; simply adding the issuer of the client certificate to the local list should solve the problem.

- untie-

Taking a closer look at the output of the curl command above, there is a line CAfile: /etc/pki/tls/certs/ca-bundle.crt , which is the list of certificate authorities used by curl.

Taking www.baidu.com as an example, we can obtain the trust chain of the client certificate with the following command:

$ openssl s_client -showcerts -servername server -connect www.baidu.com:443 > cacert.pem

In the resulting cacert.pem , we can see the following (slightly simplified):

Certificate chain
 0 s:/CN=baidu.com
   i:/CN=GlobalSign Organization Validation CA - SHA256 - G2

-----BEGIN CERTIFICATE-----
MIIKQDCCCSigAwIBAgIMEZhyT2Z0o9Yhv76iMA0GCSqGSIb3DQEBCwUAMGYxCzAJ
...(略)...
n3XcFtwQLBY9Iuqh8Mn7vtiv5k2azdGsYhZcFBCBAeUoRhDC
-----END CERTIFICATE-----

 1 s:/CN=GlobalSign Organization Validation CA - SHA256 - G2
   i:/OU=Root CA/CN=GlobalSign Root CA

-----BEGIN CERTIFICATE-----
MIIEaTCCA1GgAwIBAgILBAAAAAABRE7wQkcwDQYJKoZIhvcNAQELBQAwVzELMAkG
...(略)...
K1pp74P1S8SqtCr4fKGxhZSM9AyHDPSsQPhZSZg=
-----END CERTIFICATE-----

...(略)...

It can be seen that there are two base64 encoded strings wrapped with --BEGIN CERTIFICATE-- and --END CERTIFICATE-- , which is the certificate encoded in PEM format (Privacy Enhanced Mail) (sometimes with .crt as the extension).

There are some abstracts in front of BEGIN, which can help us understand the content of the certificate. For example, s:/CN=baidu.com indicates that the subject (s is the subject) of this certificate is baidu.com (CN is the common name), and i:/CN=GlobalSign indicates that its issuer (i is the issuer) is GlobalSign.

Therefore, it can be seen that this cacert.pem actually contains two certificates, one is the certificate used by Baidu, and the other is the certificate of the GlobalSign organization (CA) that issued the certificate.

Through curl --cacert cacert.pem https://www.baidu.com , we can confirm that this chain of trust can be used to verify the certificate of www.baidu.com (in fact, we only need the second certificate in it, delete the first certificate without affecting the execution of curl).

Back to the client's case, we use the same method to obtain the certificate of the client's certificate authority, put it in the /etc/pki/ca-trust/source/anchors/ directory, execute update-ca-trust to add it to the certificate list, and then use the curl command to request it normally.

FBI WARNING : Unless you fully understand what you are doing, do not directly update the CA list of production system (the operation steps in this article are for verification); please leave the actual operation to professional security personnel.

- Of course-

An article without a "but" is not a good article.

curl works fine, but our Java code still reports an error, which means that java and curl use different CA lists.

The problem is easy to solve. After a simple search, you will find that the jre certificate is stored in the file $JAVA_HOME/jre/lib/security/cacerts , and you need to use a special keytool tool to update it:

$ keytool -import -trustcacerts -file cacert.pem -alias 证书颁发机构的名称 -keystore $JAVA_HOME/jre/lib/security/cacerts

Enter keystore password:  changeit （这是jre自带的默认密码）

Certificate was added to keystore

Verify again, and the Java code should run normally.

Note: If you want to verify a certificate individually, you can do this

1. First create an empty keyStore (the password is storePassword):

   $ keytool -genkeypair -alias boguscert -storepass storePassword -keypass secretPassword -keystore keystore -dname "CN=Developer"
   $ keytool -delete -alias boguscert -storepass storePassword -keystore emptyStore.keystore

2. Add the certificate to the keyStore:

   $ keytool -import -trustcacerts -file cacert.pem -alias 机构名称 -keystore keystore

3. Specify keyStore to start the java program:

   $ java -Djavax.net.ssl.trustStore=keystore -Djavax.net.ssl.trustStorePassword=storePassword -cp $CLASS_PATH CLASS_NAME

- robbery-

Unfortunately, another certificate trust issue was encountered this week, this time the client's environment made a request to our server and reported the same error.

With the lessons learned, the above commands are easy to execute, but this time they don't work.

The investigation process is relatively trivial, and some detours have been taken because of being caught in a fixed mindset, but the reason is actually very simple, so I won't sell it here.

This customer is a pan-financial enterprise. The network security level of its production environment is very high. Not only does it have strict external network access restrictions, but it also hijacks all https requests by default and returns an error message with a self-signed certificate.

After communicating with the customer, after adding the domain name of Sensors Data to the whitelist, the certificate will no longer be hijacked, and the problem can be solved.

- story-

After the accident is over, tell the story.

A series of inventions such as asymmetric encryption, certificates, and trust chains constitute the cornerstone of current web communication security. It is difficult to imagine what the Internet can do without these infrastructures.

But there is a big bug hidden here: Why do we believe that these local certificate authorities are trusted?

There are at least three situations that would break this assumption:

1. Local CA list is polluted

Maybe your computer/mobile phone has been imported into the CA certificate by a virus; or you may have done this yourself, for example, the company's network management requires adding the company's self-signed certificate, or you can use Charles to capture https requests, import it from its own Signed Root CA certificate.

2. Institution's private key leaked

I haven't found any related incidents in public channels (there is an agent who leaked the private key of the client's certificate); if an institution's private key is leaked, the institution should not be far from bankruptcy.

3. Institutions that look decent can also be insane

The CA agencies controlled by the governments of various countries have probably done some "unclean" things (at least this impulse), some have been discovered, some have not. For the safety of this article, the details are not expanded here. In addition, those institutions that are "not controlled by the government" must be clean? After all, institutions are always under the jurisdiction of the country where they are located, and when they encounter government executive orders, they may not have the ability to resist.

To sum up, in theory, there is no 100% reliable communication security solution.

If your application has strict requirements on communication security, and you don't even trust the local CA list, you can consider adding more means to improve the security level of communication.

In simpler scenarios (for example, the app does not want to be captured to crack the protocol), you can verify the server's certificate (certificate fingerprint, or specify the list of certificate authorities yourself); in more demanding scenarios (for example, you need to access the internal control system), you can Issue a certificate to the client, and the browser will provide a certificate for verification when requesting. If you are interested, you can refer to the imperfect project .

- receive-

At the end, make a summary as usual:

1. HTTPS is based on certificate chain to ensure communication security;
2. The cornerstone of trust is a local list of Certificate Authorities (CAs);
3. The certificate that needs to be trusted can be solved by adding the CA certificate to the local list;
4. Local CAs are not necessarily trusted;
5. The security level of communication can be enhanced by stricter verification, or client certificate.

Finally, Shence is recruiting development, product, QA and other positions in Beijing, Shanghai, Chengdu, Wuhan, Shenzhen and other places. Interested friends welcome private messages to hook up; you can also click on my push link or scan the code Check out JD and submit your resume:

Welcome to follow