How to understand the 101 exchange protocol used by mqtt?

Students who have used mqtt know that when mqtt is connected, the status in the Network panel is 101.

Name	Status	Time
mqtt	101（Switching Protocols）	Pending

So what exactly does 101 (Switching Protocols) mean?
This article will take you to understand what the 101 exchange protocol is and the protocol upgrade mechanism used by the 101 exchange protocol.

• 101 Exchange Agreement
• Protocol Upgrade Mechanism

101 Exchange Agreement

The HTTP 101 exchange protocol means that the message sent by the client to the server contains the Upgrade request header, and the server will switch the protocol according to the request header sent by the client.

The server will add an Upgrade response header to the response to indicate the protocol the server is switching to.

In one sentence: client tells the server to switch the protocol by adding Upgrade to the request header, and the server adds the upgrade to the response header to indicate the protocol after the switch.

Another simple point is: the client tells the server to switch the protocol.

General

Request URL: wss://foo.bar
Request Method: GET
Status Code: 101 Switching Protocols

Request Headers

Connection: Upgrade 
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Sec-WebSocket-Key: xxx
Sec-WebSocket-Protocol: mqtt
Sec-WebSocket-Version: 13
Upgrade: websocket // client告诉server使用websocket协议
...

Response Headers

connection: Upgrade
sec-websocket-accept: fNs9ByuvC+rD75+tj2GMQAzbJms= // server基于client发出的Sec-WebSocket-Key:xxx计算得出，计算过程文章末尾有介绍
sec-websocket-protocol: mqtt
Upgrade: websocket // server告诉client，我们（client，server）使用的是websocket协议
...

What do these request headers mean? Connection, Sec-WebSocket-Extensions, Sec-WebSocket-Key, Sec-WebSocket-Protocol, Sec-WebSocket-Version, etc.
What about the response headers? sec-websocket-accept, sec-websocket-protocol.

Read the following protocol upgrade mechanism to understand.

Protocol Upgrade Mechanism

The HTTP 1.1 version of the protocol has a special mechanism: upgrades an established connection to another protocol, usually through the Upgrade header.

This mechanism is optional, it cannot force a protocol change . While the implementation supports the new protocol, it may also choose not to upgrade. In practical applications, this mechanism is usually used for to guide WebSocket to connect to .

Note that HTTP 2.0 explicitly prohibits the use of this mechanism. Can only be used for HTTP1.1.

Upgrade HTTP/1.1 connections

The client can use the Upgrade header to invite the server to switch to an item in the protocol list, in descending order.

Because Upgrade is a hop-by-hop header, it requires a Connection header.
This means that a typical request with an Upgrade header is:

GET /index.html HTTP/1.1
Host: www.example.com
Connection: upgrade
Upgrade: example/1, foo/2

Other headers are generally request protocol dependent; for example, the WebSocket upgrade allows additional headers to configure a WebSocket connection with some security when opened.

If the server decides to upgrade the connection, there are two cases: upgrade success and upgrade failure:

• The upgrade is successful: it will return a 101 Switching Protocols response status, and the Upgrade header indicates the protocol it switched.
• Upgrade failed: If the server cannot upgrade the connection, it ignores the Upgrade header and returns a regular response (eg 200 OK)

After the server has sent the 101 status code, it can immediately start using the new protocol and perform handshakes with other protocols. Once the connection is established, the connection becomes a two-way pipe, and requests to initiate upgrades can be initiated on top of the protocol.

General usage of the protocol upgrade mechanism

In which scenarios will the Upgrade header be used? And what are the request headers related to WebSocket connections?

• Upgrade to WebSocket connection

• Request headers related to WebSocket connections

  • Sec-WebSocket-Extensions
  • Sec-WebSocket-Key
  • Sec-WebSocket-Protocol
  • Sec-WebSocket-Version
  • Sec-WebSocket-Accept (read only)

Upgrade to WebSocket connection

The most common scenario for upgrading HTTP connections is to use WebSockets, usually by upgrading HTTP or HTTPS connections. If you use the WebSocket API to open a connection, or any WebSockets library, most or all of this is already done for you.

For example: establishing a WebSocket connection is as simple as this:

webSocket = new WebSocket("ws://destination.server.ext", "optionalProtocol")

WebSockek() constructor does all the things that create an HTTP/1.1 connection, handshake and upgrade internally for the developer.

You can use "wss://" to establish a secure WebSocket connection.

If you want to manually establish a WebSocket connection, you need to handle the handshake process yourself. After creating the HTTP/1.1 session, you need to add the two request headers Upgrade and Connection to the request.

Connection: Upgrade
Upgrade: websocket

Request headers related to WebSocket connections

The following request headers are the request headers included in the WebSocket upgrade process. Unlike the Upgrade and Connection headers, the following request headers

Sec-WebSocket-Extensions

Declare one or more protocol-level WebSocket extensions to tell the server to use. It is OK to use one or more Sec-WebSocket-Extension headers in a request; it is also OK to separate them with a semicolon.

Sec-WebSocket-Extensions: extensions

Extensions need to be separated by semicolons. Need to choose from plugin list .

For example: Sec-WebSocket-Extensions: superspeed, colormode; depth=16

The permessage-deflate from our example above is also in it.

permessage-deflate | WebSocket Per-Message Deflate | [RFC7692] | None | [RFC7692]

Sec-WebSocket-Key

Provide the server with the information that the client has the right to upgrade to WebSocket. This header can be used when insecure HTTP wants to upgrade, in order to provide some level of protection against misuse. The value of the key is generated using the algorithm defined in the WebSocket specification, so this does not guarantee security.

This key is intended to prevent non-WebSocket clients from inadvertently making websocket connections or misusing them.
Essentially, this key means: "Yes, I do want to open a WebSocket connection."

This header is automatically added by clients using it and cannot be added by XMLHttpRequest.setRequestHeader()

Sec-WebSocket-Key: key

Based on this key, the server will add a calculation data based on this key in the Sec-WebSocket-Accept header of the response.

Sec-WebSocket-Protocol

This header declares one or more WebSocket protocols you want to use.
The request header sends Sec-WebSocket-Protocol, and the response header also returns Sec-WebSocket-Protocol.

Sec-WebSocket-Protocol: subprotocols

subprotocols include these protocols: https://www.iana.org/assignments/websocket/websocket.xml#subprotocol-name

The mqtt in the example above is also in it: mqtt | mqtt | [MQTT Version 5.0]

Sec-WebSocket-Version

As request header:
Declare the WebSocket protocol version used by the client.

Sec-WebSocket-Version: version

WebSocket protocol version for server-client communication: https://www.iana.org/assignments/websocket/websocket.xml#version-number

The most commonly used is 13.

As a response header:
If the server does not support the WebSocket protocol, it will return something like 426 (Upgrade Required) and return a list of supported WebSocket versions in the Sec-WebSocket-Version header. If not supported, the Sec-WebSocket-Version header is not returned.

Sec-WebSocket-Version: supportedVersions

Sec-WebSocket-Accept

Response headers in the handshake process between the server and the client. Occurs at most once:

Sec-WebSocket-Accept: hash

If there is a Sec-WebSocket-Key, concatenate the string "258EAFA5-E914-47DA-95CA-C5AB0DC85B11" to the string, and take the 20-bit hash value of SHA-1. Finally do base64 encoding.

In our case:

Sec-WebSocket-Key: xxx
Sec-WebSocket-Accept: fNs9ByuvC+rD75+tj2GMQAzbJms=

The encoding process of generating Sec-WebSocket-Accept through Sec-WebSocket-Key is as follows:

const SecWebSocketKey = "xxx"
const helper = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
const result = SecWebSocketKey + helper

const crypto = require('crypto')
const shasum = crypto.createHash('sha1')
shasum.update(result)
const SecWebSocketAccept = shasum.digest('base64');
console.log(SecWebSocketAccept) // fNs9ByuvC+rD75+tj2GMQAzbJms=

Online demo: https://www.jdoodle.com/ia/e3G

Now that the algorithm from key to accept is clear, can the key be solved in reverse through accept?
the answer is negative. This is because sha1 encryption is used.

Cryptographically strong hash functions have two characteristics: one of the most important characteristics is that they are irreversible. Irreversibility means that the original data cannot be reconstructed from its hash, so the key cannot be reversely solved by accept.

References:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/101
https://developer.mozilla.org/en-US/docs/Web/HTTP/Protocol_upgrade_mechanism
https://stackoverflow.com/questions/6984139/how-can-i-get-the-sha1-hash-of-a-string-in-node-js
https://nodejs.org/api/buffer.html#buffer_buffers_and_character_encodings
https://stackoverflow.com/questions/10918983/how-can-decode-sha1-code

Looking forward to communicating with you and making progress together:

• WeChat public account: Dada big front end / excellent_developers
• Front-end Q&A mutual aid planet: t.zsxq.com/yBA2Biq

Strive to become an excellent front-end developer!