Do you know all about message queue MQ/JMS/Kafka?

Do you usually hear a bunch of terms such as message queue, JMS, MQ, and kafka, but don’t understand? Confusion? Let us take a look at what they are today.

Introduction to Message Queue

First of all, let’s take the chestnuts that receive the express. The traditional express delivery, the courier brother sends our express to our hands. What conditions does he need?

• The courier has time to deliver,
• We have time to take it,
• The courier brother agreed with us a time and place.

But um. The courier brother has so many couriers to deliver. Maybe when I deliver the courier, I’m not at home, or when I’m at home, the courier brother will send the courier to other places. So um, at this time, you are either sitting at home and waiting for the delivery, or you can only deliver it from the new appointment time. So what should I do to avoid this situation, huh?

So the express cabinet appeared. The courier brother does not need to care about when I am home, because the courier brother has time, he puts the courier in the courier cabinet, and when I have time, I go to the courier cabinet to pick up my courier.

Then the function of the express cabinet is the message queue we want to receive today. We can compare the message queue to a courier cabinet that stores couriers. When we need to obtain our courier, we can get our courier from the courier cabinet.

What is a message queue

We can compare the message queue to a container for storing messages. When we need to use the message, we can take out the message for our own use. Let’s take a look at the description on Wikipedia: In computer science, a message queue is a way of inter-process communication or communication between different threads of the same process. Software storage is used to process a series of inputs, usually It is from the user.

Is it difficult to understand, let’s put it another way to understand

We can compare the message queue to a container for storing messages. When we need to use the message, we can take out the message for our own use.

What is the use of message queue?

Message queuing is an important component in a distributed system. The main purpose of using message queuing is to improve system performance and peak clipping and reduce system coupling through asynchronous processing. Commonly used message middleware 17 dimensions comprehensive comparison

Improve system performance through asynchronous processing (peak clipping, reduce response time).

For example: when we register an account on a certain website, we need to do the following four things:

• Fill in our registration information;
• Submit our registration information;
• Our mailbox receives registration information;
• We received registration information via SMS.

If synchronous serial is used, the time required is: a+b+c+d

If synchronous parallel is used, the time required is: a+b+max(c,d)

If a message queue is used, the time required is: a+b+message queue

Reduce system coupling

For example, company A has built a certain system, and company B thinks that a certain function of company A is good, so company B and company A’s systems are integrated. At this time, company C also felt that this function of company A was very good, so company C also integrated with company A's system. There will be D company in the future...

In this situation, the degree of coupling between A company's system and other companies is very high, and every time a company's system is integrated, A company needs to modify its own system. If a message queue is used, it becomes as follows:

No matter how many company applications want to use company A's programs in the future, they don't need to be integrated with company A. Whoever needs this function will get it in the message queue.

Two modes of message queue

• Point-to-point mode

The application consists of: message queue, sender, and receiver.

Each message is sent to a specific queue, and the receiver gets the message from the queue. The queue holds messages until they are consumed or time out.

• Publish and subscribe model

The user program is composed of: the role topic (Topic), the publisher (Publisher), and the subscriber (Subscriber).

The publisher publishes a message, which is delivered to all clients through the topic. In this mode, the publisher and the subscriber are anonymous, that is, the publisher and the subscriber do not know who the other party is. And you can dynamically publish and subscribe Topic. Topic is mainly used to save and deliver messages, and will keep saving the message until the message is delivered to the client.

After introducing the message queue, then we introduce JMS

Introduction to JMS

JMS is the Java Message Service (Java Message Service) application program interface. It is an API for message-oriented middleware (MOM) in the Java platform, similar to JDBC. Used to send messages between two applications or in a distributed system for asynchronous communication. It provides services for creating, sending, receiving, and reading messages. The application program interface and corresponding syntax designed by Sun and its partners enable Java programs to communicate with other message components.

JMS is a message service standard or specification, allowing application components to create, send, receive, and read messages based on the JavaEE platform. It makes distributed communication less coupled, message services more reliable and asynchronous.

• So far, you should understand the difference between message queue and JMS, right?
• Message queue: In computer science, a way for A and B to communicate.
• JMS: A standard or specification for distributed communication between java platforms.
• In other words, JMS is Java's implementation of message queues.

JSM message model

Point-to-point, publish and subscribe, the message queue has already said very clearly, so I won't repeat it here.

JMS consumption

• Synchronous

The subscriber/receiver receives the message by calling receive(). In the receive() method, the thread will block until the message arrives or the message has not arrived after the specified time.

• Asynchronous

Message subscribers need to register a message listener, similar to an event listener, as long as the message arrives, the JMS service provider will deliver the message by calling the listener's onMessage().

JMS programming model

The JMS programming model is very similar to JDBC. Recall the MyBatis we talked about earlier.

• SqlSessionFactoryBuilder to construct the SqlSessionFactory session factory;
• The SqlSessionFactory session factory opens the SqlSession session for us;
• SqlSession helps us to connect to the database, and then we can perform addition, deletion, and modification.

sqlSessionFactory = new SqlSessionFactoryBuilder().build(inputStream);
SqlSession openSession = sqlSessionFactory.openSession(true);
ProjectMapper mapper = openSession.getMapper(ProjectMapper.class);
mapper.queryAllTaskByInit("init");

The JMS model is as follows

• Connection Factory created a Connection connection for me;
• The Connection connection creates a Session session for us;
• Session session creates consumers and producers for us;
• Producer generates messages;
• Consumer consumption news;

MQ introduction

Above, we mentioned that JMS is not a real technology, but an interface and a specification. Just like JDBC, whether it is mybatis, hibernate, or springJPA, no matter what kind of technology you implement, you have to comply with the JDBC specification anyway.

In Java, the current common technologies for message queues based on JMS are ActiveMQ, RabbitMQ, and RocketMQ. It is worth noting that RocketMQ does not fully comply with the JMS specification, and Kafka is not an implementation of JMS.

AMQP protocol

Here we take RabbitMQ as an example to introduce MQ, first introduce AMQP

AMQP protocol (Advanced Message Queuing Protocol, Advanced Message Queuing Protocol) is a network protocol for transferring asynchronous messages between processes.

• AMQP model

• AMQP working process
• The publisher (Publisher) publishes the message (Message) via the exchange (Exchange).
• The switch distributes the received message to the queue (Queue) bound to the switch according to the routing rules.
• Finally, the AMQP agent will deliver the message to consumers who have subscribed to this queue, or consumers can obtain them on their own needs.

RabbitMQ is a typical representative of MQ products, and is a reusable enterprise messaging system based on the AMQP protocol.

RabbitMQ model

RabbitMQ consists of two parts, namely the server side and the application side;

• The server includes: queues and switches.
• Clients include: producers and consumers.

Multiple virtual message brokers can be created on rabbitmq server. Each broker is essentially a mini-rabbitmq server, which manages its own exchange and bindings.

The broker is equivalent to a physical server, which can provide boundary isolation for different apps, so that applications can run on different broker instances safely without interfering with each other. The producer and consumer need to specify a broker to connect to the rabbit server. Message Queue – Introduction to the RabbitMQ Message Queue

There are 4 types of Exchange: direct (default), fanout, topic, and headers

• Direct: Direct exchange, which works like unicast. Exchange will send messages to a Queue that exactly matches ROUTING_KEY.
• Fanout: Broadcasting is a type of exchange. No matter what the ROUTING_KEY setting of the message is, Exchange will forward the message to all bound Queues (the so-called binding is to bind a specific Exchange to a specific Queue. Exchange and Queue are bound. It can be a many-to-many relationship).
• Topic: Theme exchanger works like multicast, Exchange sends the message and forwards the same for all queues ROUTING_KEY match mode, for example, ROUTING_KEY to user.stock the Message will be forwarded to the binding mode to match .stock, the User. stock, . and #.user.stock.# queue. (The table is to match an arbitrary phrase, # means to match 0 or more phrases).
• As for how to use RabbitMQ in the code, let's not write the code here. This article only introduces the theoretical knowledge points at present.

Kafka

At the end of the previous section, we mentioned that Kafka is not an implementation of JMS, so we did not mention it in the MQ chapter. Let's start learning kafka now. The vernacular takes you to know Kafka

First, let's put the diagram of the principle of kafka. I believe that when you see this picture, your heart is broken. My grass, what the hell. Next, let's digest it bit by bit.

Kafka schematic

First introduce the terminology in the figure above.

• Producer: The message producer is the client that sends messages to the Kafka broker.
• Consumer: Message consumer, a client that fetches messages from Kafka broker.
• Topic: The classification method provided by kafka for messages. The broker is used to store message data for different topics. A topic can be considered as a type of message. Each topic will be divided into multiple partitions. Each partition is an append log file at the storage level. Any message posted to this partition will be directly appended to the end of the log file. The position of each message in the file is called the offset (offset), which is a long number, which uniquely marks a message. It uniquely marks a message. Kafka does not provide other additional indexing mechanisms to store offsets, because "random reads and writes" of messages are almost not allowed in Kafka.
• Broker: Kafka cluster of middleware, storing messages, is a cluster composed of multiple servers.
• Partition: In order to achieve scalability, a very large topic can be distributed to multiple brokers (ie servers), a topic can be divided into multiple partitions, and each partition is an ordered queue. Each message in the partition will be assigned an ordered id (offset). Kafka only guarantees that messages will be sent to consumers in the order in a partition, and does not guarantee the order of a topic as a whole (among multiple partitions).
• Offset: Kafka's storage files are named after offset.kafka. For example, if you want to find the location at 2049, just find the file 2048.kafka. Of course, the first offset is 00000000000.kafka.

Similar to the characteristics of JMS, but not the implementation of the JMS specification. Kafka categorizes messages according to Topic when they are stored. The sender of the message becomes the Producer, and the recipient of the message becomes the Consumer. In addition, the kafka cluster is composed of multiple Kafka instances, and each instance (server) becomes a broker. Both the Kafka cluster, the producer and the consumer rely on zookeeper to ensure system availability and the cluster saves information.

Kafka is based on file storage. Through partitioning, the log content can be distributed to multiple servers to prevent the file size from reaching the upper limit of the single-machine disk. Each partition will be saved by the current server (kafka instance); a topic can be divided into any number of partitions. The efficiency of message storage/consumption. In addition, more partitions means more consumers can be accommodated, effectively improving the ability of concurrent consumption.

The difference between Kafka and JMS is that even if the message is consumed, the message will not be deleted immediately. The log file will be deleted after a certain period of time according to the configuration requirements in the broker.

Kafka high availability mechanism

• Multiple brokers, each broker is a node;
• You create a topic, this topic can be divided into multiple partitions, each partition can exist on different brokers, each partition puts a part of the data.
• Using the replica mechanism, the data of each partition will be synchronized to other machines to form multiple replica copies.
• All replicas will elect a leader, then production and consumption will deal with this leader, and then other replicas will be followers.
• When reading data, read from the leader, and when writing data, the leader synchronizes the data to all followers. If a broker goes down, this broker keeps a copy of the other brokers. Assuming that there is a leader on this broker, then a new leader is selected.

There is a lot of content, and it needs to be digested little by little with pictures. What is Kafka and in what scenarios?

Producer structure chart

So far, although you can see it in the clouds, but I believe you can still distinguish it?

Tidy up

Message queue: refers to a way of communication between A and B in the computer field;

JMS: Java interface specification for message queues;

ActiveMQ/RabbitMQ: A technology that implements the JMS interface specification;

RocketMQ: Not exactly a technology that is specifically implemented by the JMS interface specification;

Kafka: A technology for the specific implementation of non-JMS interface specifications;

Source: blog.csdn.net/m0_37892044/article/details/106603925