Front-end basics-TCP/IP is not fully explained

TCP/IP is not fully explained

TCP/IP is a collective term for communication protocols. First, let’s get to know the OSI reference model

OSI reference model

The roles of each layer in the OSI reference model are as follows:

nutshell:

application layer: provide services for the application and specify the relevant details of the communication in the application. Including file transfer, email, remote login and other protocols
represents the layer: Convert the information processed by the application into a format suitable for network transmission, or convert the data of the next layer into a format that can be processed by the upper layer. Therefore, it is mainly responsible for the conversion of the data format. Specifically, it is to convert the inherent data format of the device into a network standard transmission format. Comparing the same bitstream with different devices, the interpretation results may be different. Therefore, keeping them consistent is the main role of this layer.
Session layer: responsible for the establishment and disconnection of communication links, as well as data segmentation and other data transmission related management.
transport layer: plays the role of reliable transmission. The processing is only performed on the nodes of the communicating parties, without the need for processing on the router.
network layer: transfer data to the destination address. The destination address can be a certain address formed by connecting multiple networks through a router. Therefore, this layer is mainly responsible for addressing and routing.
data link layer: responsible for the communication and transmission between nodes interconnected on the physical layer. For example, communication between two nodes connected to an Ethernet. Divide the sequence of 0 and 1 into meaningful data frames and send them to the opposite end (generation and reception of data frames).
Physical layer: Responsible for the interchange between 0 and 1 bit streams (0, 1 sequence), voltage levels, and light flashes.

`TCP/IP basics`

TCP/IP and OSI reference model

`Hardware (physical layer)`

The bottom layer of TCP/IP is the hardware responsible for data transmission. This kind of hardware is equivalent to physical layer equipment such as Ethernet or telephone lines. The content of it has not been able to be defined uniformly. Because as long as people use different transmission media at the physical level (such as using network cables or wireless), the bandwidth, reliability, security, delay, etc. of the network will be different, and there is no established index in these aspects. In short, TCP/IP is a protocol that was proposed under the premise that the devices connected to the network can communicate.

`Network interface layer (data link layer)`

Use the data link layer in Ethernet to communicate, so it belongs to the interface layer. In other words, it doesn't hurt to think of it as the "driver" that makes the NIC work. The driver is the software that acts as a bridge between the operating system and the hardware. The computer's peripheral additional equipment or expansion card can't be used immediately by directly plugging it into the computer or the expansion slot of the computer, and it needs the support of the corresponding driver.

`Internet layer (network layer)`

The Internet layer uses the IP protocol, which is equivalent to the third network layer in the OSI model. The IP protocol forwards packet data based on the IP address.

The functions of the Internet layer and the transport layer in the TCP/IP layer are usually provided by the operating system. Especially a router, it must realize the function of forwarding packet data packets through the Internet layer.

In addition, all hosts and routers connected to the Internet must implement IP functions. Other network devices connected to the Internet (such as bridges, repeaters, or hubs) do not necessarily implement IP or TCP functions.

IP

IP is a protocol that transmits data packets across the network so that the entire Internet can receive data. The ip protocol enables data to be sent to the other side of the earth, during which time it uses the IP address as the identification of the host.
IP also implies the function of the data link layer. Through IP, communication between hosts communicating with each other can be achieved regardless of the underlying data link.
Although IP is also a protocol for packet switching, it does not have a retransmission mechanism. Even if the packet data fails to reach the opposite host, it will not be retransmitted. Therefore, it is an unreliable transmission protocol.

ICMP

Once an abnormality occurs during the sending of an IP data packet and the destination address of the opposite end cannot be reached, it is necessary to send an exception notification to the sending end. ICMP is made for this function. It is also sometimes used to diagnose the health of the network.

ARP

A protocol that resolves the physical address (MAC address) from the IP address of the packet data packet.

`Transport layer`

The transport layer of TCP/IP has two representative protocols. The function of this layer itself is similar to the transport layer in the OSI reference model.

The main function of the transport layer is to enable communication between applications. Inside a computer, multiple programs are usually running at the same time. To this end, it is necessary to distinguish which programs are communicating with which programs. It is the port number that identifies these applications.

TCP

TCP is a connection-oriented transport layer protocol. It can ensure that the communication between the communication hosts at both ends is reachable. TCP can correctly handle abnormal situations such as packet loss and disordered transmission sequence during transmission. In addition, TCP can effectively use bandwidth and relieve network congestion.
However, in order to establish and disconnect the connection, sometimes it needs to send and receive packets at least 7 times, resulting in a waste of network traffic. In addition, in order to improve the utilization of the network, various complex specifications are defined in the TCP protocol, which is not conducive to the use of video conferences (the amount of audio and video data is fixed).

UDP

UDP is different from TCP in that it is a connectionless-oriented transport layer protocol. UDP does not pay attention to whether the peer end has actually received the transmitted data. If you need to check whether the peer end receives the packet data packet or whether the peer end is connected to the network, you need to implement it in the application.
UDP is often used in multimedia fields such as packet data less or multicast, broadcast communication, and video communication.

`Application layer (layering above the session layer)`

In the layering of TCP/IP, the functions of the session layer, the presentation layer and the application layer in the OSI reference model are all integrated into the application program. These functions are sometimes implemented by a single program, and sometimes may be implemented by multiple programs. Therefore, if you look closely at the application functions of TCP/IP, you will find that it not only implements the content of the application layer in the OSI model, but also implements the functions of the session layer and the presentation layer.

Most of the architecture of TCP/IP applications belongs to the client/server model. The program that provides the service is called the server, and the program that receives the service is called the client. In this communication mode, the program that provides the service will be deployed on the host in advance, waiting to receive the request that the client may send at any time.

The client can send a request to the server at any time. Sometimes the server may handle exceptions, overload, etc. At this time, the client can re-send the request after waiting for a while.

WWW

WWW can be said to be an important driving force behind the popularity of the Internet. Users can easily surf the Internet freely with the help of a mouse and keyboard on a software called a Web browser. That is to say, various information set up on the remote server with a single click of the mouse will be presented to the browser. The browser can not only display text, pictures, animation and other information, but also play sounds and run programs.
The protocol used for communication between the browser and the server is HTTP (HyperText Transfer Protocol). The main format of the transmitted data is HTML (HyperText Markup Language). HTTP in the WWW belongs to the OSI application layer protocol, and HTML belongs to the presentation layer protocol.

E-Mail

E-mail actually refers to sending letters on the Internet. With e-mail, no matter how far away people are, they can send mails to each other as long as they are connected to the Internet. The protocol used when sending emails is called SMTP.

File Transfer (FTP)

File transfer refers to transferring files stored on the hard disk of other computers to the local hard disk, or transferring files from the local hard disk to the hard disk of other machines.
The protocol used in this process is called FTP (File Transfer Protocol). FTP has been in use for a long time, and you can choose to use binary or text mode during transmission.
During file transfer in FTP, two TCP connections are established, which are the control connection used when sending a transmission request and the data connection used when actually transmitting data.

Telnet (TELNET and SSH)

Remote login refers to a function of logging on to a remote computer and enabling programs on that computer to run. Two protocols, TELNET and SSH, are commonly used for remote login in TCP/IP networks.

Network Management (SNMP)

When network management is performed in TCP/IP, SNMP (Simple Network Management Protocol) protocol is adopted. The host, bridge, router, etc. that use SNMP management are called SNMP agent (Agent), and the management section is called Manager (Manager). SNMP is the protocol used by this Manager and Agent.
On the SNMP agent side, information such as network interface information, communication data volume, abnormal data volume, and device temperature are stored. This information can be accessed through MIB (Management Information Base). Therefore, in TCP/IP network management, SNMP is an application protocol, and MIB is a presentation layer protocol.
The larger the scope of a network and the more complex the structure, the more effective it is needed to manage it. SNMP allows administrators to check network congestion in time, find faults early, and collect necessary information for future network expansion.

`TCP/IP layered model and communication example`

Data packet header

In each layer, a header is added to the data sent, and the header contains the necessary information for the layer, such as the destination address and protocol-related information sent. Usually, the information provided for the protocol is the packet header, and the content to be sent is data.

The data packet transmitted in the network consists of two parts: one part is the header used by the protocol, and the other part is the data transmitted from the upper layer. The structure of the header is defined in detail by the specific specifications of the protocol. For example, to identify how many bits should be taken from which bit of the packet in the domain of the upper layer protocol, how to calculate the checksum and which bit of the packet should be inserted. If the computers at both ends of the communication are not the same in identifying the serial number of the protocol and the calculation method of the checksum, the communication will not be realized at all.

Therefore, in the header of the data packet, it is clearly indicated how the protocol should read the data. Conversely, when you see the header, you will be able to understand the necessary information of the agreement and the content to be processed. Therefore, seeing the packet header is like seeing the specification of the protocol. The first part is like the face of agreement.

Example: Send data packet

Suppose A sends an email to B with the content: "Good morning". From the perspective of TCP/IP communication, an e-mail is sent from one computer A to another computer B. We will use this example to explain the process of TCP/IP communication.

application processing Start the application to create a new email, fill in the recipient's mailbox, and then enter the email content "Good morning" from the keyboard, and click the "Send" button to start TCP/IP communication. First, the coding process will be carried out in the application. (UTF-8, etc.) These codes are equivalent to the OSI presentation layer function. After the encoding is converted, the actual email may not be sent out immediately, because some email software has the function of sending multiple emails at the same time, and there may also be the function of receiving new emails after the user clicks the "receive" button. The management function of when to establish a communication connection and when to send data, in a broad sense, belongs to the function of the session layer in the OSI reference model. The application establishes a TCP connection at the moment of sending a mail, thereby using this TCP connection to send data. Its process is to first send the application data to the next layer of TCP, and then do the actual forwarding processing.
TCP module processing TCP is responsible for establishing a connection, sending data, and disconnecting according to the instructions of the application. TCP provides reliable transmission for smoothly sending data from the application layer to the opposite end. In order to realize this function of TCP, a TCP header needs to be attached to the front end of the application layer data. The TCP header includes the source port number and destination port number (used to identify the application on the sending host and the receiving host), sequence number (used to which part of the packet sent is the data), and checksum (used to determine whether the data is damaged ). Then send the packet with the TCP header attached to the IP.
IP module processing IP combines the TCP header and TCP data passed by TCP as its own data, and adds its own IP header to the front end of the TCP header. Therefore, the IP header in the IP data packet is followed by the TCP header, and then the data header of the application and the data itself. The IP header contains the receiver IP address and the sender IP address. Following the IP header is information used to determine whether the following data is TCP or UDP. After the IP packet is generated, refer to the routing control table to determine the route or host that accepts the IP packet. Subsequently, the IP packet will be sent to the driver connected to the network interface of these routers or hosts in order to actually send the data. If you do not know the MAC address of the receiving end, you can use ARP (Address Resolution Protocol) to find it. As long as the MAC address of the opposite end is known, the MAC address and IP address can be handed over to the Ethernet driver to realize data transmission.
network interface (Ethernet driver) processing The IP packet transmitted from the IP is nothing but data for the Ethernet driver. Attach the Ethernet header to this data and send it. The Ethernet header contains the MAC address of the receiving end, the MAC address of the sending end, and the protocol of the Ethernet data indicating the Ethernet type. The Ethernet data packet generated based on the above information will be transmitted to the receiving end through the physical layer. The FCS in the sending process is calculated by the hardware and added to the end of the packet. The purpose of setting FCS is to determine whether the data packet is destroyed due to noise.

Summarized as follows:

passing through the data link

When the packet flows, the Ethernet packet header, the IP packet header, the TCP packet header (or UDP packet header), and the application's own packet header and data are appended accordingly. At the end of the packet, the Ethernet packet tail is appended

Each packet header will contain at least two pieces of information: one is the sender and receiver addresses, and the other is the protocol type of the upper layer.

After each protocol layering, there must be information identifying the sender and receiver of the packet. Ethernet will use the MAC address, IP will use the IP address, and TCP/UDP will use the port number as the address to identify the hosts at both ends. Even in an application, information such as an email address is an address identifier. These address information are appended to the packet header corresponding to the protocol when each packet passes through each layer. As shown in the figure:

In addition, each layered packet header also contains an identification bit, which is used to identify the type information of the upper layer protocol. For example, the Ethernet type in the packet header of the Ethernet, the protocol type in IP, and the port numbers of the two ports in TCP/UDP, etc., all play a role in identifying the protocol type. Even in the header information of the application, sometimes it also contains a label to identify its data type.

Data packet reception processing

Receiving and sending are in reverse order:

network interface (Ethernet driver) processing After receiving the Ethernet packet, the host first finds the MAC address from the Ethernet packet header to determine whether it is a packet sent to itself. If it is not a packet sent to itself, the data is discarded. And if you receive a packet that happens to be sent to yourself, look up the type field in the Ethernet packet header to determine the type of data transmitted by the Ethernet protocol. In this example, the data type is obviously an IP packet, so the data is passed to the subroutine that processes IP. If it is not IP but other protocols such as ARP, then the data is passed to ARP for processing. In short, if the type field of the Ethernet packet header contains an unrecognized protocol type, the data is discarded.
IP module processing After the IP module receives the IP packet header and the following data part, it also performs similar processing. If it is determined that the IP address in the packet header matches your own IP address, you can receive the data and search for the upper-level protocol from it. If the upper layer is TCP, the part after the IP packet header is passed to TCP for processing; if it is UDP, the part after the IP packet header is passed to UDP for processing. In the case of routers, the receiving end address is often not its own address. In this case, you need to use the routing control table to forward the data after investigating the host or router that should be delivered.
TCP module processing In the TCP module, the checksum is first calculated to determine whether the data is corrupted. Then check whether the data is being received according to the sequence number. Finally, check the port number to determine the specific application. After the data is received, the receiving end sends a "confirmation receipt" to the sending end. If this receipt message fails to reach the sender, the sender will think that the receiver has not received the data and keep sending it repeatedly. After the data is completely received, it will be passed to the application identified by the port number.
application processing The receiving end application will directly receive the data sent by the sending end. By analyzing the data, it can be known that the recipient's address of the mail is B's address. If there is no mail box of B on host B, host B returns an error message of "No such recipient address" to the sender. But in this example, host B happens to have B's inbox, so host B and recipient B can receive the body of the email. The mail will be saved to the hard disk of this machine. If the saving can be performed normally, the receiving end will return a "processing normal" receipt to the sending end. On the contrary, once the disk is full, the mail cannot be saved successfully, etc., a "handling exception" receipt will be sent to the sender. Thus, user B can use the mail client on host B to receive and read the e-mail sent by user A on host A-"Good morning".

`Data link`

Data link refers to the data link layer in the OSI reference model, and sometimes also refers to communication means such as Ethernet and wireless LAN.

The protocol of the data link layer defines the specifications for transmission between devices interconnected by communication media. Communication media include twisted-pair cables, coaxial cables, optical fibers, radio waves, and infrared rays. In addition, data is sometimes transferred between devices through switches, bridges, and repeaters.

The data link can also be regarded as the smallest unit in network transmission. In fact, if you carefully observe the Internet that connects the whole world, you can find that it is nothing more than composed of many such data links, so the Internet can also be called the "collection of data links."

MAC address

The MAC address is used to identify the nodes interconnected in the data link. In Ethernet or FDDI, the MAC address is used according to the IEEE802.3V specification. Other devices such as wireless LAN (IEEE802.lla/b/g/n, etc.) and Bluetooth also use the same MAC address.

The MAC address is 48 bits long. The 3~24 bits (bits) in the MAC address represent the manufacturer identification code, and each NICT vendor has a specific and unique identification number. 25-48 bits are used internally by the manufacturer to identify each network card. Therefore, it can be guaranteed that there will be no network cards with the same MAC address in the world.

IEEE802.3 does not limit the type of data link when formulating the MAC address specification, that is, no matter what kind of data link network (Ethernet, FDDI, ATM, wireless LAN, Bluetooth, etc.), the same MAC address will not appear.

Exception: People can freely set their own MAC address on the microcomputer board. For another example, if multiple virtual machines are started on a host, because the network card without hardware can only be set by the virtual software to set the MAC address to multiple virtual network cards, it is difficult to ensure that the generated MAC address is unique. . However, no matter which protocol member communication device, the design premise is the uniqueness of the MAC address. This can also be said to be the basic principle of the online world.

Shared media network

From the perspective of the use of communication media (communication, media), networks can be divided into shared media and non-shared media.

A shared medium network refers to a network in which multiple devices share a communication medium. The earliest Ethernet and FDDI are media sharing networks. In this way, devices use the same carrier channel for transmission and reception. For this reason, half-duplex communication is basically adopted, and it is necessary to control access to the medium.

There are two media access control methods in shared media networks: one is contention, and the other is token passing.

contention mode Contention refers to competing for the right to obtain data transmission, also called CSMA (Carrier Sense Multiple Access). This method usually makes each station in the network occupy the channel to send data on a first-come, first-served basis. If multiple stations send frames at the same time, conflicts will occur. It will also cause network congestion and performance degradation.

In another part of Ethernet, another method of improving CSMA is adopted-CSMA/CD method. CSMA/CD requires each station to check for conflicts in advance, and once a conflict occurs, release the channel as soon as possible. Its specific working principle is as follows:

If there is no data flowing on the carrier channel, any station can send data.
Check if there will be a conflict. Once a conflict occurs, give up sending data and immediately release the carrier channel.
After giving up sending, a random delay is given for a period of time, and then re-contend for the medium and send the frame again.

token passing method The token transfer method is to send a special message called "token" along the token ring, which is a way to control the transmission. Only the station that has obtained the token can send data. This method has two characteristics: one is that there will be no conflict, and the other is that each station has the opportunity to obtain tokens through equal circulation. Therefore, even if the network is congested, it will not cause performance degradation. Of course, in this way, a station cannot send data frames before receiving the token, so the utilization rate of the data link will not reach 100% when the network is not too congested. For this reason, a variety of token passing technologies have been derived. For example, early token release, token addition, etc., and multiple tokens circulating at the same time. The purpose of these methods is to improve network performance as much as possible.

Non-shared media network

Non-shared media network refers to the non-shared media, which is a special transmission control method for the media. In this way, each station in the network is directly connected to the switch, and the switch is responsible for forwarding data frames. In this mode, the sending end and the receiving end do not share the communication medium, so in many cases, full-duplex communication is used.

Not only does ATM use this transmission control method, it has recently become the mainstream method of Ethernet. The network is constructed through the Ethernet switch, so that a one-to-one connection is formed between the computer and the switch port, and full-duplex communication can be realized. There is no conflict in this one-to-one connection full-duplex communication mode, so more efficient communication can be achieved without the CSMA/CD mechanism.

Half-duplex and full-duplex communication

Half-duplex refers to a communication method that only sends or receives only. It is similar to a radio transceiver. If both ends are talking at the same time, they cannot hear what the other party is saying. Unlike full-duplex, it allows data to be sent and received at the same time. Similar to a phone call, both parties can talk at the same time.

forward according to the MAC address

In a shared network using coaxial cables such as Ethernet (10BASE5, 10BASE2) and other media, only one host can send data at the same time. When the number of connected hosts increases, the communication performance will decrease significantly. If devices such as hubs or concentrators are connected in a star shape, a new network device- switching hub appears, which is a technology that uses switches used in non-media sharing networks in Ethernet. , The switching hub is also called Ethernet switch.

An Ethernet switch is a bridge that holds multiple ports. They decide which network interface to send data from according to the destination MAC address of each frame in the data link layer. The table used to record the sending interface referred to at this time is called the forwarding table.

switch forwarding mode There are two switch forwarding methods, one is called store and forward, and the other is called direct forwarding. The store-and-forward method checks the FCS bit at the end of the Ethernet data frame before forwarding. Therefore, it is possible to avoid sending frames that are corrupted due to collisions or erroneous frames caused by noise. In the direct forwarding mode, it is not necessary to receive the entire frame before forwarding it. You only need to know the destination address to start forwarding. Therefore, it has the advantage of a shorter delay. But at the same time, there is inevitably the possibility of sending wrong frames.

`IP protocol`

IP is equivalent to layer 3 of the OSI reference model.

IP (IPv4, IPv6) is equivalent to the third layer in the OSI reference model-the network layer. The main role of the network layer is to "realize communication between terminal nodes." This kind of communication between terminal nodes is also called "end-to-end communication".

The main function of the next layer of the network layer, the data link layer, is to transfer packets between nodes interconnecting the same data link. Once across multiple data links, the network layer is needed. The network layer can span different data links, and data packet transmission between nodes at both ends can be realized even on different data links.

IP is to send data packets to the final destination address in a complex network environment.

In the Internet world, those devices with IP addresses are called "hosts". It can be a super sky computer or a small computer. However, to be precise, the definition of a host should mean "a device that is configured with an IP address but does not perform routing control." A device equipped with both an IP address and routing control capability is called a "router", which is different from a host. The node is the collective name for the host and the router.

The relationship between the network layer and the data link layer

The data link layer provides the communication function between directly connected two devices. In contrast, ip as the network layer is responsible for communication and transmission between two networks that are not directly connected.

`IP basics`

IP is roughly divided into three functional modules, which are IP addressing, routing (forwarding to the final node), and IP sub-packing and grouping.

MAC address is an identification code used to identify different computers in the same link. IP address is used to "identify the destination address for communication among all hosts connected to the network".

routing control

Routing control (Routing) refers to the function of sending packet data to the final destination address. Even if the network is very complex, routing control can be used to determine the path to the destination address. Once the operation of this routing control is abnormal, the packet data is very likely to be "lost" and unable to reach the target address. Therefore, the reason why a data packet can successfully reach the final destination address depends entirely on routing control.

IP belongs to the connectionless type

ip faces connectionless. That is, before sending the packet, there is no need to establish a connection with the target address of the opposite end. If the upper layer encounters data that needs to be sent to the ip, the data will be immediately compressed into an ip packet and sent out.

In the case of a connection, the connection needs to be established in advance. If the peer host is shut down or does not exist, it is impossible to establish a connection. Conversely, a host that has not established a connection cannot send data.

The situation for connectionless is different. Even if the remote host is shut down or does not exist, the data packet will still be sent out. Conversely, for a host, when and where it will receive data is also unknown. Generally, network monitoring should be performed so that the host only receives data packets sent to itself. If you are not prepared, you may miss some packages that should be received. Therefore, there may be a lot of redundant communication in a connection-oriented way.

ip faces connectionless reasons: one is to simplify, and the other is to speed up. Connection-oriented processing is relatively more complicated than connection-oriented processing. Even managing each connection itself is a rather tedious task. In addition, a connection must be established before each communication, which will reduce the processing speed. When a connection is needed, the upper layer can be entrusted to provide this service. Therefore, ip adopts a connectionless approach in order to achieve simplification and speed.

and the corresponding upper layer (transport layer) TCP adopts a link-oriented type

The role of ip is to try its best to send the data packet to the destination, it does not verify whether it is finally received or not. The ip data packet may have problems such as packet loss, misalignment, and doubling of the data volume during transmission. At this time, tcp is required to provide this kind of guarantee, and tcp is responsible for ensuring that the opposite host does receive the receipt.

`DNS`

that represents the correspondence between host names and IP addresses within the organization. In the application, when the user enters the host name (domain name), the DNS will automatically search the database where the host name and IP address are registered, and quickly locate the corresponding IP address. Moreover, if the host name and IP address need to be changed, it only needs to be processed within the organization, and there is no need to apply or report to other organizations.

The domain name and domain name server need to be set up according to the hierarchy. If the domain name server is down, then DNS queries for that domain will not work properly. Therefore, in order to improve disaster tolerance, at least two domain name servers are generally set up. Once the first domain name server fails to provide a query, it will automatically go to the second or even the third domain name server to perform disaster recovery processing in order.

The host and software that perform DNS queries are called DNS resolvers. The workstation or personal computer used by the user belongs to the resolver. A resolver must register at least one IP address of a domain name server. Usually, it includes at least the IP address of the domain name server within the organization.

`DNS query`

dns query mechanism :

Computer A wants to visit www.baidu.com.
First check the corresponding IP address in the domain name server.
If the DNS server knows the IP, it will return it directly, and if it doesn't know it, it will request a query from the root domain name server.
The return address of the root domain name.
The DNS server queries the domain name server of www.baidu.com for the IP address.
Return the found IP address to the client.
Computer A establishes communication with www.baidu.com.

`TCP and UDP`

TCP is a connection-oriented and reliable streaming protocol. Stream refers to the uninterrupted data structure. In order to provide reliable transmission, TCP implements a "sequence control" or "retransmission control" mechanism. In addition, it has many functions such as "flow control (flow control)", "congestion control", and improving network utilization.

UDP is an unreliable datagram protocol. The subtle processing will be handed over to the upper application layer to complete. In the case of UDP, although the size of the sent message can be guaranteed, there is no guarantee that the message will arrive. Therefore, sometimes the application needs to be retransmitted.

`Features of UDP`

UDP does not provide a complex control mechanism, and uses IP provide connectionless communication services. And it is a mechanism for sending the data from the application to the network as it is the moment it is received. Even in the case of network congestion, UDP cannot perform flow control and other actions to avoid network congestion. In addition, even if packets are lost during transmission, UDP is not responsible for retransmission. There is no correction function even when the arrival order of the packets is out of order. If these detailed controls are needed, then they have to be handled by an application that uses UDP. Because UDP is oriented to connectionless and its processing is simple and efficient, it is often used in the following aspects:

Communication with a small amount of packets (DNS, SNMP, etc.)
Multimedia communication such as video and audio
Limited to application communication in specific networks such as LAN
Broadcast communication

`Features of TCP`

TCP and UDP is quite big. It fully implements various control functions during data transmission, can perform retransmission control when packets are lost, and can also perform sequence control on out-of-sequence sub-packets. These are not available in UDP. In addition, as a connection-oriented protocol, TCP will only send data when it is confirmed that the communication peer exists, so that the waste of communication traffic can be controlled. TCP realizes reliable transmission through mechanisms such as checksum, sequence number, confirmation response, retransmission control, connection management, and window control.

TCP handshake waved schematic diagram:

TCP window control and retransmission control

In the use of window control, if there is a segment loss, first consider the situation that the confirmation response fails to return. In this case, the data has arrived at the opposite end, and there is no need to retransmit. However, when window control is not used, data that has not received a confirmation response will be retransmitted.

Secondly, consider the loss during sending. If the receiving host receives data other than the sequence number that it should receive, it will return an acknowledgement response to the data received so far. As shown below:

When a message segment is lost, the sender will always receive a confirmation response with serial number 1001. This confirmation response seems to remind the sender that "I want to receive data starting from 1001". Therefore, in the case that the window is relatively large and the message segment is lost, the confirmation response of the same sequence number will be repeatedly returned. If the sending host receives the same confirmation response three times in a row, it will retransmit the corresponding data. This mechanism is more efficient than the aforementioned timeout management, so it is also called high-speed retransmission control.

Flow control

The sender sends data according to its actual situation. However, the receiving end may receive an irrelevant data packet and may spend some time dealing with other issues. Therefore, it will take some time to do other processing for this data packet, and it will not even be able to receive any data under high load. In this way, if the receiving end discards the data that should have been received, it will trigger the retransmission mechanism, resulting in unwarranted waste of network traffic.

In order to prevent this phenomenon from happening, TCP provides a mechanism that allows the sender to control the amount of data sent according to the actual receiving capability of the receiver. This is the so-called flow control. Its specific operation is that the receiving host informs the sending host of the size of the data that it can receive, so the sending end will send data that does not exceed this limit. This size limit is called the window size.

In the TCP header, there is a special field used to notify the window size. The receiving host puts the size of the buffer that it can receive into this field to notify the sender. The larger the value of this field, the higher the throughput of the network.

However, once this buffer on the receiving end faces a data overflow, the value of the window size will also be set to a smaller value and notified to the sending end, thereby controlling the amount of data sent. In other words, the sending host will control the amount of data sent according to the instructions of the receiving host. This also forms a complete TCP flow control.

Congestion control

Generally speaking, computer networks are in a shared environment. Therefore, the communication between other hosts may cause network congestion. When the network is congested, if a large amount of data is suddenly sent, it is very likely to cause the paralysis of the entire network.

In order to prevent the occurrence of this problem, TCP uses a value derived from an algorithm called slow start at the beginning of communication to control the amount of data sent.

`Application protocol`

Application layer protocol definition

There are many applications that use the Internet, including Web browsers, e-mail, remote login, file transfer, network management, etc. It is the application protocol that enables these applications to perform specific communication processing.

Lower-layer protocols such as TCP and IP are protocols that do not depend on the types of upper-layer applications and have very wide applicability. The application protocol is a protocol designed and created to implement a certain application.

The TCP/IP application layer covers all the functions of the 5th, 6th and 7th layers in the OSI reference model. It not only includes the session layer function for managing communication connections, the presentation layer function for converting data formats, but also the communication with the peer host All functions including interactive application layer functions.

`Remote login`

function of logging in to the computing function of the other end of the network from one's own local computer is called 16082626b127a1 remote login. After logging in to general-purpose computers or UNIX workstations remotely, not only can you directly use the applications on these hosts, you can also set parameters for these computers. Remote login mainly uses two protocols, TELNET and SSH.

TELNET

TELNET uses a TCP connection to send text commands to the host through this connection and execute them on the host. Local users seem to be directly connected to the Shell inside the remote host, and operate directly locally.

TELNET can be divided into two basic services. One is the emulation terminal function, and the other is the negotiation option mechanism.

SSH

SSH is an encrypted remote login system. It can be sent without entering a password when logging in in TELNET, which is easy to cause the danger of communication eavesdropping and illegal intrusion. The communication content can be encrypted after using SSH. Even if the information is eavesdropped, it is impossible to crack the password sent, the specific command, and the result of the command.

`file transfer`

FTP is a protocol used when transferring files between two connected computers.

FTP working mechanism

FTP uses two TCP connections to realize file transfer: one for control, and the other for data (file) transfer.

The TCP connection used for control is mainly used in the control part of FTP. For example, the authentication of login user name and password, the name of the file to be sent, and the setting of the sending method. Using this connection, you can send requests and receive responses via ASCII code strings. Data cannot be sent on this connection, and the data requires a dedicated TCP connection.

The connection for control will remain connected until the user requests to disconnect. However, most FTP servers will forcibly disconnect users who have not entered any new commands for a long time.

Usually, the TCP connection used for data transmission is established in the opposite direction to the connection used for control.

`e-mail`

The protocol that provides e-mail services is called SMTP (Simple Mail Transfer Protocol) o SMTP uses the TCP protocol in its transport layer in order to achieve efficient delivery of e-mail content.

`WWW`

WWW defines three important concepts, which are the means and location of accessing information (URI, Uniform Resource Identifier), the form of information (HTML, HyperText Markup Language), and information forwarding (HTTP, HyperText Transfer Protocol) operations.

URI

URI is the abbreviation of Uniform Resource Identifier, used to identify resources. URI is an efficient identification code that can be used outside the WWW. It is used in various combinations of homepage addresses, e-mails, and phone numbers.

URL is often used by people to indicate the specific location of a resource (file) on the Internet. But URI is not limited to identifying Internet resources, it can be used as an identification code for all resources. Now, in valid RFC documents, URLs are no longer used, and URIs are used instead. Compared with the narrow concept of URL, URI is a broad concept. Therefore, URI can be used in other application protocols besides WWW.

URI format

http: //Host name: port number/path? Access content#partial information

TLS/SSL and HTTPS

Encrypting HTTP communication via TLS/SSL is called HTTPS communication. As shown in the figure:

`The last question`

This article has been written almost here. Next, we will consolidate the knowledge system in depth with a question

from the URL input to the page loading process?

First sort out the main part:

From the browser receiving the url to opening the network request thread (this part can expand the mechanism of the browser and the relationship between the process and the thread)
Start the network thread to issue a complete http request (this part involves dns query, tcp/ip request, five-layer Internet protocol stack, etc.)
From the server receiving the request to the corresponding background receiving the request (this part may involve load balancing, security interception, internal processing in the background, etc.)
HTTP interaction between the background and the foreground (this part includes knowledge of http headers, response codes, message structures, cookies, etc., which can mention cookie optimization of static resources, as well as encoding and decoding, such as gzip compression, etc.)
The caching problem that comes out separately, http caching (this part includes http caching header, etag, catch-control, etc.)
The parsing process after the browser receives the http data packet (parses html-lexical analysis and then parses it into a dom tree, parses css to generate a css rule tree, merges it into a render tree, and then layout, painting rendering, composite layer synthesis, GPU drawing, Processing of external chain resources, loaded and domcontentloaded, etc.)
CSS visual format model (elements rendering rules, such as the concept of including blocks, control boxes, BFC, IFC, etc.)
JS engine parsing process (JS interpretation stage, preprocessing stage, execution stage to generate execution context, VO, scope chain, recycling mechanism, etc.)
Others (different knowledge modules can be expanded, such as cross-domain, web security, hybrid mode, etc.)

`From the browser receiving the url to opening the network request thread`

The browser is multi-process, there is a master process, and each tab page will open a new process.

Processes may include master process, plug-in process, GPU, tab page (browser kernel), etc.

Browser process: the main process of the browser (responsible for coordination and main control), there is only one
Third-party plug-in process: each type of plug-in corresponds to a process, which is created only when the plug-in is used
GPU process: at most one, used for 3D rendering
Browser rendering process (kernel): By default, there is one process for each Tab page, which does not affect each other, controls page rendering, script execution, event handling, etc. (sometimes it will be optimized, such as multiple blank tabs will be merged into one process)

Each tab page can be seen as a browser kernel process, and then this process is multi-threaded, it has several types of sub-threads

GUI thread
JS engine thread
Event trigger thread
Timer thread
Network request thread

After entering the URL, it will be parsed (the essence of the URL is the Uniform Resource Locator). URL generally includes several parts:

protocol , protocol header, such as http, ftp, etc.
host , host domain name or IP address
port , port number
path , directory path
query , which is the query parameter
fragment , the hash value after #, generally used to locate a certain position

A separate thread needs to be opened for each network request. For example, if the URL is parsed to the http protocol, a new network thread will be created to process resource downloads.

Therefore, the browser will open up a network thread according to the parsed protocol to request resources (here, it is temporarily understood as being developed by the browser kernel, and if there is an error, it will be repaired later)

`Start the network thread to issue a complete http request`

The main content of this part includes: dns query, tcp/ip request construction, five-layer Internet protocol stack, etc.

DNS resolution

If the input is a domain name, it needs to be resolved into IP by dns. The general process is:

If the browser has a cache, use the browser cache directly, otherwise use the local cache, if not, use the host
If there is no local, query the DNS domain name server (of course, there may be routing and caching in the middle) to find the corresponding IP

The domain name query may have passed through the CDN scheduler (if there is a CDN storage function), and you need to know that DNS resolution is time-consuming, so if you resolve too many domain names, the first screen loading will become too slow. Consider dns-prefetch optimization

TCP/IP request

TCP divides the long http message into short messages, and establishes a connection with the server through a three-way handshake for reliable transmission.

Then, when the connection is disconnected, you need to wave your hands four times (because it is full-duplex, you need to wave your hands four times)

tcp/ip concurrency limit

Browsers have restrictions on concurrent tcp connections under the same domain name (ranging from 2-10). And in http1.0, a resource download often needs to correspond to a tcp/ip request

The difference between get and post

Although get and post are both tcp/ip in essence, they are also different at the tcp/ip level in addition to the http level.

get will generate one tcp packet and two post

When a get request is made, the browser will send the headers and data together, and the server will respond with 200 (return data),
When requesting a post, the browser first sends headers, the server responds with 100 continue, the browser sends data again, and the server responds with 200 (return data).

Differences at the http level:

GET is harmless when the browser rolls back, while POST will submit the request again.
GET requests will be actively cached by the browser, while POST will not, unless manually set.
GET requests can only be url-encoded, while POST supports multiple encoding methods.
The parameters transmitted in the URL for GET requests are limited in length, while POST does not.
GET is less secure than POST, because parameters are directly exposed on the URL, so it cannot be used to transmit sensitive information.
GET parameters are passed through the URL, and POST is placed in the Request body.

Five-layer Internet Protocol Stack

The five layers here are: application layer, transport layer, network layer, data link layer, and hardware layer. The corresponding OSI model:

`From the server receiving the request to the corresponding background receiving the request`

Load balancing

All requests initiated by users point to the scheduling server (reverse proxy server, such as installing nginx to control load balancing), and then the scheduling server allocates different requests to the servers in the corresponding cluster according to the actual scheduling algorithm, and then the scheduler waits for the actual server HTTP response and feed it back to the user

http message structure

Universal head

Request Url: 请求的web服务器地址
Request Method: 请求方式
（Get、POST、OPTIONS、PUT、HEAD、DELETE、CONNECT、TRACE）
Status Code: 请求的返回状态码，如200代表成功
Remote Address: 请求的远程服务器地址（会转为IP）

status code

1xx——指示信息，表示请求已接收，继续处理
2xx——成功，表示请求已被成功接收、理解、接受
3xx——重定向，要完成请求必须进行更进一步的操作
4xx——客户端错误，请求有语法错误或请求无法实现
5xx——服务器端错误，服务器未能实现合法的请求

// 常见状态码
200——表明该请求被成功地完成，所请求的资源发送回客户端
304——自从上次请求后，请求的网页未修改过，请客户端使用本地缓存
400——客户端请求有错（譬如可以是安全模块拦截）
401——请求未经授权
403——禁止访问（譬如可以是未登录时禁止）
404——资源未找到
500——服务器内部错误
503——服务不可用

Request/response header

Accept: 接收类型，表示浏览器支持的MIME类型
（对标服务端返回的Content-Type）
Accept-Encoding：浏览器支持的压缩类型,如gzip等,超出类型不能接收
Content-Type：客户端发送出去实体内容的类型
Cache-Control: 指定请求和响应遵循的缓存机制，如no-cache
If-Modified-Since：对应服务端的Last-Modified，用来匹配看文件是否变动，只能精确到1s之内，http1.0中
Expires：缓存控制，在这个时间内不会请求，直接使用缓存，http1.0，而且是服务端时间
Max-age：代表资源在本地缓存多少秒，有效时间内不会请求，而是使用缓存，http1.1中
If-None-Match：对应服务端的ETag，用来匹配文件内容是否改变（非常精确），http1.1中
Cookie: 有cookie并且同域访问时会自动带上
Connection: 当浏览器与服务器通信时对于长连接如何进行处理,如keep-alive
Host：请求的服务器URL
Origin：最初的请求是从哪里发起的（只会精确到端口）,Origin比Referer更尊重隐私
Referer：该页面的来源URL(适用于所有类型的请求，会精确到详细页面地址，csrf拦截常用到这个字段)
User-Agent：用户客户端的一些必要信息，如UA头部等

Common response headers (parts)

Access-Control-Allow-Headers: 服务器端允许的请求Headers
Access-Control-Allow-Methods: 服务器端允许的请求方法
Access-Control-Allow-Origin: 服务器端允许的请求Origin头部（譬如为*）
Content-Type：服务端返回的实体内容的类型
Date：数据从服务器发送的时间
Cache-Control：告诉浏览器或其他客户，什么环境可以安全的缓存文档
Last-Modified：请求资源的最后修改时间
Expires：应该在什么时候认为文档已经过期,从而不再缓存它
Max-age：客户端的本地资源应该缓存多少秒，开启了Cache-Control后有效
ETag：请求变量的实体标签的当前值
Set-Cookie：设置和页面关联的cookie，服务器通过这个头部把cookie传给客户端
Keep-Alive：如果客户端有keep-alive，服务端也会有响应（如timeout=38）
Server：服务器的一些相关信息

cookie

Long and short connections

TCP/IP level:

Long connection: A tcp/ip connection can send multiple data packets continuously. During the tcp connection maintenance period, if no data packets are sent, both parties need to send detection packets to maintain the connection. Generally, you need to do online maintenance by yourself (similar to a heartbeat packet) )
Short connection: When the two communication parties have data exchange, a tcp connection is established, and after the data is sent, the tcp connection is disconnected

HTTP level

In http1.0, a short connection is used by default, that is to say, a connection is established once the browser does not perform an http operation, and the connection is terminated when the task ends. For example, each static resource request is a separate connection
Starting from http1.1, long connections are used by default. When long connections are used, there will be this line Connection: keep-alive. In the case of long connections, when a web page is opened, the tcp used to transmit http between the client and the server The connection will not be closed. If the client accesses the server's page again, it will continue to use the established connection

keep-alive will not keep forever, it has a duration, generally configured in the server (such as apache), and long connections need to be supported by both the client and the server to be effective

http 2.0

Significant differences between http2.0 and http1.1:

In http1.1, every time a resource is requested, a tcp/ip connection needs to be opened, so the corresponding result is that each resource corresponds to a tcp/ip request. Because tcp/ip itself has a limit on the number of concurrent, so when the resource It slows down significantly as soon as it gets too much
In http2.0, a tcp/ip request can request multiple resources, that is to say, as long as a tcp/ip request, you can request several resources, divided into smaller frame requests, and the speed is significantly improved.

http2.0 features:

Multiplexing (that is, a tcp/ip connection can request multiple resources)
Header compression (http header compression, reducing volume)
Binary framing (a binary framing layer is added between the application layer and the transport layer to improve transmission performance and achieve low latency and high throughput)
Server-side push (the server can send multiple responses to a request from the client, and can actively notify the client)
Request priority (If the stream is given a priority, it will be processed based on this priority, and the server determines how many resources are needed to process the request.)

https

SSL/TLS handshake:

The browser requests the establishment of an SSL connection and sends a random number to the server-Client random and the encryption method supported by the client, such as RSA encryption, which is transmitted in plain text at this time.
The server selects a set of encryption algorithms and Hash algorithms from it, replies with a random number-Server random, and sends its identity information back to the browser in the form of a certificate (the certificate contains the website address and the public key for asymmetric encryption) , And information such as the certification authority)
After the browser receives the server certificate

Verify the legitimacy of the certificate (whether the issuing authority is legit, whether the URL contained in the certificate is the same as the one you are visiting), if the certificate is trusted, the browser will display a small lock, otherwise there will be a prompt
After the user receives the certificate (whether it is trusted or not), the browser will generate a new random number-Premaster secret, and then the public key in the certificate and the specified encryption method will be encrypted with Premaster secret and sent to the server.
Use Client random, Server random and Premaster secret to generate a symmetric encryption key for HTTP link data transmission through a certain algorithm- session key
Use the agreed HASH algorithm to calculate the handshake message, and use the generated session key<