End-to-end encryption technology of Rongyun Internet communication security series

The development and application of Internet communication technology allows information to travel far and wide, and connections occur at any time. Follow [Rongyun Global Internet Communication Cloud] to learn more

But at the same time, Internet communication also exposed a series of security problems. We need to apply relevant security technologies in a targeted manner, enhance the security and reliability of communication systems, and build a protective wall to protect the security of users' personal information.

In the first article "Link Security" in the Rongyun Internet Communication Security series, we introduced the related technologies and practices of link encryption. TLS link encryption is enabled when transmitting instant messaging messages to ensure that messages cannot be eavesdropped and tampered with before reaching the server; CA authentication mechanism is used to prevent man-in-the-middle attacks.

These are all effective ways to improve the security of data transmission between clients and servers, but they fail to solve the risks of privacy and security of communication between users. Because after the data is transmitted to the server, all those who have access to the server, including employees, related suppliers and other relevant personnel and even hackers, may read the user's data.

Therefore, end-to-end encryption technology was vigorously promoted, and it was quickly recognized and accepted by the public after its appearance. It is used in instant messaging software such as WhatsApp, Signal, and Telegram.

Today, we share the third article in a series of articles on Internet communication security, leading you to understand end-to-end message encryption in one article.

Design Ideas of End-to-End Encryption Scheme

When it comes to end-to-end encryption, the first solution that comes to our mind is to encrypt the entire message before the sender sends it, and decrypt it after the receiver receives it. In this way, the message relay server cannot obtain the content of our message.

In fact, this is indeed a simplified solution for sending and receiving messages in end-to-end encryption, but we are more complex and more secure in practical applications.

The pre-problem we need to solve first is how to securely pass the key for message encryption and decryption.

The answer is to transmit keys using asymmetric encryption, similar to how keys are exchanged securely in SSL/TLS.

The algorithm of asymmetric encryption and transmission of symmetric encryption keys generally boils down to two methods: one is mainly based on RSA, ECC, etc., and the public key encryption and private key decryption method is essentially an encryption and decryption algorithm; the other is based on encryption and decryption. DH and ECDH-based methods for generating shared keys are essentially negotiating a common key through calculation rather than an encryption and decryption algorithm.

Most of the end-to-end encryption in instant messaging software uses the method of generating a shared key to transmit the session key. Why is this?

---

For the convenience of understanding, here is an introduction to the DH algorithm:

Based on the definition and properties of the original root, the Diffie-Hellman key exchange algorithm can be defined. The algorithm is described as follows:

1. There are two globally exposed parameters, a prime q and an integer a, a being a primitive root of q.

2. Suppose users A and B want to exchange a key, user A chooses a random number XA(XA<q)< span=""> as the private key, and calculates the public key YA=a^XA mod q. A keeps the value of XA private so that YA is publicly available to B. Similarly, user B chooses a private random number XB<q< span="">, and calculates the public key YB=a^XB mod q. B keeps the value of XB secretly so that YB is publicly available to A.

3. User A generates the shared secret key by K = (YB)^XA mod q. Likewise, the computation for user B to generate the shared secret key is K = (YA)^XB mod q. Both calculations yield the same result.

The derivation process is as follows:

K = (YB)^XA mod q

= (a^XB mod q)^XA mod q

= (a^XB)^XA mod q (obtained according to the modulo operation rule)

= a^(XBXA) mod q

= (a^XA)^XB mod q

= (a^XA mod q)^XB mod q

= (YA)^XB mod q

So it is equivalent to that both parties have exchanged an identical secret key.

4. Since XA and XB are kept secret, the only parameters an adversary can use are q, a, YA and YB. Thus the adversary is forced to take discrete logarithms to determine the key. For example, to obtain User B's secret key, the adversary must first compute XB and then compute its secret key K using the same method that User B does.

The security of the Diffie-Hellman key exchange algorithm relies on the fact that while computing the exponent modulo a prime number is relatively easy, computing the discrete logarithm is difficult. For large prime numbers, it is nearly impossible to compute the discrete logarithm.

Briefly describe the process of DH shared key as follows:

The "key S" is the final shared key.

---

In summary, there are several reasons why end-to-end encryption uses the method of generating a shared key to transmit the session key:

1. If the public key encryption and private key decryption methods such as RSA and ECC are used to transmit the key, a temporary key needs to be generated when the session is created, and encrypted with the other party's public key and transmitted to the receiver. This requires full assurance of the reliability of the message, and if the message is lost or corrupted in any of the links, subsequent communication cannot proceed. Alternatively, a more reliable transmission scheme needs to be adopted. The usual practice is that the receiving end needs to be online, and various confirmations are used to ensure this reliability. With the shared key method, you only need to know the public key of the other party to complete the generation of the shared key, and the other party does not necessarily need to be online.

2. If an already generated ephemeral symmetric key is lost, the key needs to be renegotiated. In the shared key method, only the public key of the other party needs to be known to complete the generation of the shared key without renegotiation.

3. Using the public key encryption and private key decryption method requires at least one more communication process of exchanging the symmetric key than the method of generating the shared key.

4. The key negotiation method can not only complete the key negotiation between two points, but also can be extended to the common negotiation of the same key among multiple people, which can meet the needs of multi-person group communication.

Preliminary Scheme of End-to-End Encryption

Combined with the knowledge of the shared key method of DH algorithm (the public key can be freely disclosed), we first design a simple end-to-end message encryption process.

1. When the client APP is installed for the first time, it generates its own DH public key and private key based on the two global parameters exposed by the server.

2. Upload your own public key to the certificate server, and the certificate server saves the relationship between the user ID and its public key. The private key is stored on the client.

3. When sending a message to the other party for the first time or receiving the other party's message for the first time, the certificate server is queried for the other party's public key.

4. The shared key is calculated based on the other party's public key and its own private key.

5. All subsequent messages with the other party are encrypted and decrypted based on this key and the same symmetric encryption and decryption algorithm.

(End-to-end message encryption process)

So far, we have completed a simple end-to-end message encryption scheme. In this scheme, we have introduced a third-party role for storing the user's public key. The existence of this role allows either party to not care about the online status of the other party. Send an encrypted message to the other party at any time, and the message forwarding server cannot decrypt the message.

Next, we analyze and optimize the various problems of this simple solution.

End-to-end encryption scheme optimization

HMAC

In the process of message transmission, both parties need to confirm the integrity of each other's messages. The simple method is to hash the message, and the obtained hash value is attached to the message and sent with the message; after the peer receives it, the same hash is performed to verify the message. has been tampered with.

The key point is that the Hash value obtained by different data must be different, and the Hash value with the key is the MAC.

In addition, in order to avoid using the same Hash function to operate on the same data and always obtain the same value, an additional key is added, so that different MACs can be obtained by using different keys. Of course, this key is known to both peers.

In this way, we get the algorithm of message integrity authentication based on encrypted Hash - Hash-based MAC.

ECDH

The DH algorithm is based on the mathematical problem of discrete logarithms. With the gradual enhancement of computer computing power, we have to keep using larger numbers to increase the difficulty of cracking. At present, it is generally believed in the industry that at least 2048-bit DH algorithm is required. better security.

Here we introduce the ECDH algorithm to replace the DH algorithm. The ECDH key agreement algorithm is a combination of the ECC algorithm and the DH key exchange principle. ECC is a cryptosystem based on the discrete logarithm problem based on elliptic curves. Under the same cracking difficulty, ECC has the advantages of smaller key length and faster forward calculation speed.

ECDH on our system can directly use the currently public sepc256kl and Curve25519 curves without the need for the service to provide public large number parameters.

forward security

In the process of message transmission, if the negotiated key is leaked, it means that all information will be exposed to risk. In order to prevent this from happening, we need to use a different key for each encryption from the previous one, and the previous key cannot be deduced backwards.

A Hash algorithm is introduced here. This Hash algorithm can derive another key with greater discreteness by entering a key. Every time a message is sent, the Hash operation is performed with the last message key to obtain this key. , because the Hash algorithm is one-way and irreversible, it is impossible to derive the previous key from this key. Visually, it's like a ratchet, which is a special kind of gear that can only turn in one direction, but not back.

Double ratchet

For extreme safety requirements, we will consider forward safety and backward safety. How to ensure that in a certain communication, the decrypted key cannot decrypt the previous message, and within a certain period, the decrypted key will no longer work.

Between this we introduce another ratchet to ensure its backward security. This is the double ratchet algorithm in the famous Signal protocol.

The double ratchet algorithm consists of a KDF ratchet and a DH ratchet.

---

KDF (Key derivation function) The key derivation function is used to derive one or more keys from an original key. Essentially a Hash function, usually used to turn short passwords into long ones. In addition, KDF needs to add a "salt" to prevent rainbow tables. Due to the characteristics of Hash, the length of this "salt" must be at least greater than the length of the Hash result.

KDF (original key, salt) = derived key

---

KDF ratchet is to use the KDF algorithm to design a key changing effect. The process is as follows:

(KDF ratchet process)

The first step is to use the KDF algorithm to derive a new key from the initial key. The new key is cut into two parts. The first half is used as the input for the next KDF calculation, and the second half is used as the message key.

Every iteration (or ratchet step, so to speak), a new message key is generated.

Due to the one-way nature of the KDF algorithm, the key of the previous message cannot be deduced from the key of this message. This guarantees the forward security of the key. But if the salt in the KDF is mastered, then it can calculate all future message keys according to this algorithm.

In order to ensure backward security, a method should be designed so that the salt introduced in each iteration is random, so as to ensure that each message key cannot be backward calculated.

From the DH algorithm introduced above, two pairs of key pairs can generate a secure negotiated key through the DH protocol. If one of the key pairs is replaced, the new negotiated key will also change.

According to this method, we can devise a way to update the salt safely. We add a temporary public key certificate to the certificate server. This temporary certificate is a temporary public key pair constructed according to the identifiers of the receiving parties, that is, each individual session of each person has a temporary public key. Every time a message round is performed, the temporary public key of one party is updated, and the negotiation is carried out according to the temporary public key of the other party and the private key of one party, and the negotiated key is used as the salt, so that the message key generated by the KDF ratchet algorithm is used. Has backward security.

At the beginning, we cannot predict all new two-person sessions of each person, then we can stipulate that when creating a new two-person session, the initiator first generates a new temporary DH public-private key pair, and uploads its own temporary DH to the server. Second, the sender uses the long-term public key announced by the receiver and its own temporary private key to negotiate a key as the key for message encryption to encrypt the message; finally, the receiver uses its own long-term public key after receiving the message for the first time. Calculate the message key with the sender's temporary private key, and generate a temporary public and private key when replying to the message for the first time, and upload the temporary public key at the same time.

The problem is that if the receiver is not online, and the sender updates its own temporary public key certificate for each message, it will cause the messages sent out to be unable to be decrypted normally after the receiver is online and received.

In order to solve this problem, we need to stipulate that the temporary certificate will only be updated after sending a message and getting a reply from the other party. If the other party does not reply to the message, the temporary certificate will not be updated. If the receiving end can reply to the message, it means that it has been online and has received the message, so as to ensure that the offline message or the out-of-order message can also be parsed by the other party normally. This method is another DH ratchet in the double ratchet algorithm.

X3DH

Compared with the original scheme, in order to satisfy the forward security and backward security of the message, we added a double ratchet algorithm, and added a set of session-level temporary DH keys for each person on the original basic scheme, and each person has a long-term key. and a set of ephemeral keys.

However, since the long-term key cannot be replaced, the scheme still has security risks. Therefore, Signal protocol designed a more complex and secure DH key exchange process, called X3DH, which is a 3-fold extended version of the DH protocol.

In the X3DH protocol, everyone has to create 3 key pairs, which are as follows:

1. Identity Key Pair - a long-term key pair that conforms to the DH protocol, created during user registration and bound to the user's identity;

2. Signed pre-shared key (Signed Pre Key) - a mid-term key pair that conforms to the DH protocol, created when the user registers, signed by the identity key, and periodically rotated. This key may be used to protect the identity key. The key is not leaked;

3. One-Time Pre Keys - A queue of one-time use Curve25519 key pairs, generated during installation and supplemented when insufficient.

Everyone has to upload the public keys of these 3 key pairs to the server for others to use when initiating a session.

If Alice wants to send a message to Bob, she must first determine the message key with Bob. The process is roughly as follows:

1. Alice wants to create a temporary key pair (ephemeral key), we set it to EPK-A, this key pair is prepared for the later ratchet algorithm, and it has little effect here;

2. Alice obtains the public keys of Bob's three key pairs from the server: identity key pair IPK-B; signed pre-shared key SPK-B; one-time pre-shared key OPK-B;

3. Alice starts to use the DH protocol to calculate the negotiated key. The parameters to be introduced include: the private keys of the two key pairs created by herself, and the three public keys of Bob. Then, in a similar way of permutation and combination, bring your own private key and the other party's public key into the DH algorithm for calculation.

DH1 = DH(IPK-A, SPK-B)

DH2 = DH(EPK-A, IPK-B)

DH3 = DH(EPK-A, SPK-B)

DH4 = DH(IPK-A, OPK-B)

as the picture shows:

Then connect the four calculated values before and after to get the initial key, as follows:

DH = DH1 || DH2 || DH3 || DH4

Note: "||" represents a connector, such as 456 || 123 = 456123

However, the DH key is too long to be used as a message key, so a KDF calculation is performed on this initial key to derive a fixed-length message key S

S = KDF (DH1 || DH2 || DH3 || DH4)

In this step, Alice finally calculates the message key S.

1. Alice uses the message key S to encrypt the message, and sends it to Bob together with her own identity public key IPK-A and ephemeral public key EPK-A.

2. After receiving Alice's message, Bob takes out Alice's 2 public keys, together with his own key, and uses the same algorithm as Alice to calculate the message key S.

3. Bob and Alice communicate encrypted using the message key.

As can be seen from the above, X3DH is actually a complex version of the DH protocol.
So far, we have briefly introduced the most core X3DH protocol and double ratchet algorithm in Signal Protocol, which can basically satisfy forward security and backward security. Of course, the real process would be more complex and secure.

End-to-end encryption scheme for group chats

In the instant messaging scene, in addition to the chat between two people, another important scene is the group chat, so how to encrypt the messages between the groups end-to-end?

We go back to the derivation process on the DH key agreement algorithm again. Obviously, the DH key agreement algorithm can continue to be used in the case of multiple parties, which is the basis of end-to-end encryption in group chats.

However, the design of Signal Protocol in group chat is different from that of two-person chat. Since the confidentiality requirements of group chat are relatively low, only KDF chain ratchet + public key signature is used for encrypted communication to ensure the pre-encryption. to safety.

The communication process is as follows:

1. Each group member must first generate a random 32-byte KDF Chain Key, which is used to generate the message key to ensure the forward security of the message key, and also generate a random Curve25519 signature key. Key pair, used for message signing.

2. Each group member sends the chain key (Chain Key) and signature public key to other members individually. At this point, each member has the chain key and signature public key of all members in the group.

3. When a member sends a message, it first encrypts the message with the message key generated by the KDF chain ratchet algorithm, then signs it with the private key, and then sends the message to the server, which sends it to other members.

4. After receiving the encrypted message, other members first use the sender's signature public key to verify, and after successful verification, use the corresponding chain key to generate the message key, and decrypt it with the message key.

5. When a group member leaves, all group members clear their own chain key and signature public key, regenerate it, and send it to each member individually. By doing so, the members who leave will not be able to view the messages in the group.

It can be seen from the above that a person in different groups will generate different chain keys and signature key pairs to ensure the isolation between groups. In each group, each member also stores the KDF chain and signature public key of other members. If there are too many group members, the amount of encryption and decryption operations is very large, which will affect the speed of sending and receiving. At the same time, the key management database is also will be very large, and the read efficiency will be reduced.

Therefore, the group chat uses the Signal Protocol protocol, and the group size should not be too large.

Supplementary Description of End-to-End Encryption Scheme

Above we have introduced the entire process of end-to-end encryption for two-person chat and group chat in instant messaging. However, under normal circumstances, the end-to-end message encryption is only the actual payload part of the encrypted message, and the control layer of the message is not encrypted, because the message forwarding server needs to forward or route the message according to the control information.

In order to prevent messages from being targeted for analysis (analyzing when a user sent a message to whom, or who received a message), we still need to encrypt and protect the long connection link of the overall instant messaging to prevent information from being intercepted and analyzed by intermediate network devices ; In order to prevent the key server from being attacked by man-in-the-middle, it is also necessary to enable link encryption protection.