Cloud Native Message Queuing RocketMQ: Why We Choose RocketMQ

Author: Li Wei

Speaking of message queues, ActiveMQ, RabbitMQ, RocketMQ, Kafka, Pulsar, etc. have flooded into our minds. Among so many open source message queue products, how can a qualified architect give a cost-effective solution? Commercial products are not included in the options for now.

Next, I will explain why I choose RocketMQ from the two aspects of selection elements and RocketMQ's advantages .
​

Selection elements

​
First, a brief summary and analysis is made from the perspectives of the company, the message queue service provider (usually the middleware team), and the end user.
​

1. From the perspective of the company, pay attention to the following points:

​
1. Technology cost
Technical costs generally include server costs, secondary development costs, post-maintenance costs, etc. In short: it is money.

At present, servers basically use cloud servers, and the performance of servers with the same configuration of different cloud vendors is also different. Generally, the server cost needs to be understood: cloud vendor machine performance, cloud vendor discounts, required server configuration, number of servers, and the current cost of a single server. Price, discounted price for a single server, etc.
​
2. Labor cost
The labor cost generally includes the cost of existing technical personnel and the cost of recruiting new personnel.
​
How the new technology selection is accepted by the current technicians and how easy it is to learn all need to be considered. If it is too difficult, the launch cycle will become longer, the realization of business requirements will be slow, and some people will even leave the company directly.

The cost of recruiting new recruits is generally as follows: resume screening, appointment interviews, several rounds of interviews, making offers, accepting offers, formal entry, probationary period, and regularization. This involves headhunting costs, human resources communication costs, interview costs, and newcomers’ post-employment environment adaptation costs.
​
3. Other
At present, Internet companies at different stages have different requirements for technology costs and labor costs, but many companies of a certain scale actually treat them with a "buy, buy, buy" mentality: as long as the business develops rapidly, buying servers, recruiting People are not a problem. If the cost is high, technology will be used to reduce costs and lay off employees. This is not only the pain of employees, but also the pain of the business, and the pain of the company.
​

2. From the perspective of the middleware group, pay attention to the following points:

​
1. Stable
The first and foremost point of company-level service is stability. With stable components and stable services, business can proceed in an orderly manner. So, no matter when, stability is king.
​
2. Function support

Different business scenarios require different functions. Usually, we will consider retry, dead letter mechanism, site reset, timing delay messages, transaction messages, master-slave switching, and permission control.
​
3. Performance
Currently includes write latency and throughput.
​
4. Management Platform
First of all, it needs to meet the needs of end-user access, viewing, and troubleshooting, administrators to control topics, and consumer convenience. The management platform has the best existing ones, which is convenient for secondary development.
​
5. Monitoring and alarming
Monitor whether the alarm is complete, whether it is convenient to access the company's internal self-developed system, or the industry's de facto standard Prometheus.
​
6. Operation & Support & Open Source Community
If the product is launched, most of the time, we are doing operation and maintenance & support. Operation and maintenance includes service deployment, migration, service upgrade, system bug resolution, user Q&A, management platform and monitoring and alarm platform upgrade, etc.
​
7. Other
In addition to relying on ourselves, we can also use the power of the community. The same problem may have been encountered by others and submitted PR, and it has been solved, and we can use it as a reference. Therefore, the activity of the community is also a very important consideration.
​

3. From the perspective of end users (generally including business back-end research and development and their leaders)

​
1. Stability
For business development and their leader, their core task is to implement business logic. If a service always has problems for three days and two ends, it is fatal to them, so stability is a core part of the comparison.
​
2. Difficulty in retrofitting existing projects
The transformation of old projects is actually the most practical part of business R&D access to new middleware.
​
3. Is it convenient to access new projects
Ease of access is directly related to their workload.
​
4. How compatible with the current App microservice framework
The integration of new projects and the compatibility of the company's microservice framework are relatively easy. Generally, middleware will consider the convenience of business R&D access when providing services.
​

Advantages of RocketMQ

​
The following will analyze the advantages of RocketMQ in this regard according to the requirements of the option elements.
​

1. How RocketMQ solves and amicably faces company-level demands

​
1. Technology cost

In terms of technological maturity, after experiencing the multi-trillion peak of Alibaba's Double Eleven, WeBank, Minsheng Bank, Ant Financial, Ping An, ByteDance, Kuaishou, Meituan, JD.com, NetEase and other big companies in the industry. After the test, it goes without saying.
​
RocketMQ does not have high requirements on the configuration of the server, and ordinary cloud hosts can be used. We once verified that the 8C 16G 500G SSD cluster with 2 masters and 2 slaves can send tps up to 4~5w, and consume tps peak at 20w+, which is stable at 8w~9w. Moreover, it can also scale horizontally without feeling according to the actual needs of the business.
​
In general, technology costs are relatively controllable and there are many talents.
​
2. Labor cost
The labor cost is mainly the cost of learning the existing technical personnel and the cost of recruiting new personnel.
​
RocketMQ is developed in java, and the code is also very stable and organized. Apart from functional differences between versions, there are almost no changes in APIs and transmission protocols, which is more convenient for upgrades.
​
Java is also a relatively mainstream language currently used by middleware, and it is widely used by technical personnel. RocketMQ is used in the financial industry such as: WeBank, Minsheng Bank, Ant Financial, and Ping An; other industry companies, such as Ali, ByteDance, Kuaishou, Meituan, JD.com, NetEase, etc., are used by a large number of small and medium-sized enterprises. The range of candidates relatively bigger.
​
The RocketMQ community is also relatively active. There are many DingTalk groups, WeChat groups, and QQ groups. The community documents are very rich and complete. There are also many video and documents for principle analysis, which is very easy to learn and get started.
​
The following is the DingTalk group. You are welcome to join the group to leave a message and answer questions.

Compared with C/C++, C#, Python, Go, etc., there are more talents in message queue in java: the mainstream Kafka is scala + java, and pulsar is java, which also has great advantages for recruitment.
​
In general, RocketMQ technicians are friendly to labor costs.
​

2. From the perspective of the middleware group, how does RocketMQ provide excellent capabilities to escort the business?

​
1. Stability
Financial-grade reliability, Alibaba Double Eleven stably supports trillion-level news peaks, and the company where I worked before had 2 years + zero accidents .
​
2. Rich functions and many supported scenarios

• Retry, dead letter mechanism, friendly and insensitive business retry mechanism.
• Sequence message, transaction message
• 10,000-level topic quantity support
• message filtering
• message tracking
• Master-slave automatic switching
• Native support for Prometheus monitoring
• Native support for easy-to-use management platform: RocketMQ Console
• Access Rights Control (ACL)

3. Performance

• RocketMQ can support 99.9% of the write latency of 2 ms, and other open source message queue middleware is basically greater than 5 ms; at present, most of the message queue middleware supports horizontal expansion, and the horizontal expansion in throughput can almost be satisfied. RocketMQ's performance test on Didi: _ https://developer.aliyun.com/article/664608 _, for your reference.
• Sending and consuming tps and kafka is an order of magnitude, and the sharp increase in the number of topics has little impact on performance.

4. Management Platform
RocketMQ Console natively supports:
https://github.com/apache/rocketmq-externals/tree/master/rocketmq-console
​
5. Monitoring and alarming
RocketMQ Exporter natively supports Prometheus:
https://github.com/apache/rocketmq-exporter
​
6. Operation & Support & Open Source Community

• No third-party dependencies such as zk, out of the box
• The Dingding group, WeChat group, and QQ group in the community are very active, and the Dingding group and WeChat group have all questions and answers.
• The community recently came to a new young lady, Commiter, and the team is also growing.

In general, RocketMQ is stable, reliable, and has good performance. It works out of the box and does not depend on Zookeeper. The system is more stable and less complex. The monitoring and alarming and other surrounding facilities are complete, the scene support is complete, the community is active, and the documentation is rich. It is the best choice for the middleware team.
​

3. For end users: business research and development, business research and development leaders, their core concerns are whether the provided technology is stable and reliable, and whether it can be accessed quickly and easily

​
When looking at this problem from the middleware group level, RocketMQ is stable and reliable. Is it friendly to access?
​
RocketMQ provides java native client, Spring client, C++ client, Python client, Go client and other multi-type and multi-language clients, which can be accessed uniformly for various projects.
​
Spring Cloud has basically become the de facto standard in the microservice framework. RocketMQ supports Spring boot Starter and Spring Cloud Function to integrate into the microservice framework, making it more convenient and efficient to support the Spring system.
​

Kafka vs RocketMQ

​
In practice, many people should have faced RocketMQ vs Kafka. Kafka is suitable for scenarios that are insensitive to delay, batch type, controllable number of topics, and insensitive to message loss. For example, MySQL-2Hive, MySQL-2-Flink data flow channel, log data flow channel, etc. in big data scenarios.

RocketMQ is suitable for business scenarios such as financial transfer messages, order status change messages, and mobile phone message Push. In these scenarios, the number of topics is usually over 10,000, which is extremely sensitive to message delay and loss, and the data is usually processed in articles. For the problem of massive data, generally horizontal expansion can completely solve the problem.

Choose the right product for the right scenario. There is no such thing as a universal product. It is a compromise and a trade-off.

about the author

​
Li Wei, Apache RocketMQ community Commiter, Python client project leader, co-founder of Apache RocketMQ Beijing community, Apache Doris Contributor. Currently working at Tencent, mainly responsible for OLAP database development, has rich experience in distributed storage system design and research and development, and is also keen on knowledge sharing and community activities.

RocketMQ learning materials

​
Alibaba Cloud Zhixing Lab provides a series of RocketMQ online practical environment, including operation documents and ubuntu experimental environment. You can try it at any time:
​

• Best practices for getting started with Apache RocketMQ open source:

https://start.aliyun.com/course?spm=a2ck6.17690074.0.0.53c52e7dSi19ML&id=eAz6VTK5

• "RocketMQ Distributed Messaging Middleware: Core Principles and Best Practices" is accompanied by the book: _ https://start.aliyun.com/course?spm=a2ck6.17690074.0.0.7aec2e7dCPMDFG&id=ASeJlmpX_
  
• Play RocketMQ in the Spring ecosystem:

https://start.aliyun.com/course?spm=a2ck6.17690074.0.0.241e2e7d0aEIxJ&id=hzidp9W1

The preview of the experiment is as follows:
​

Other resources

​

• RocketMQ vs. ActiveMQ vs. Kafka:

http://rocketmq.apache.org/docs/motivation/

• RocketMQ source code:

https://github.com/apache/rocketmq

• RocketMQ Exporter source code:

https://github.com/apache/rocketmq-exporter

• RocketMQ Spring source code:

https://github.com/apache/rocketmq-spring

• RocketMQ C++ client source code:

https://github.com/apache/rocketmq-client-cpp

• RocketMQ Python client source code:

https://github.com/apache/rocketmq-client-python

• RocketMQ Go client source code:

https://github.com/apache/rocketmq-client-go

• RocketMQ Console source code:

https://github.com/apache/rocketmq-externals/tree/master/rocketmq-console

• RocketMQ Flink Connector source code:

https://github.com/apache/rocketmq-externals/tree/master/rocketmq-flink

• How RocketMQ guarantees message reliability:

   [https://mp.weixin.qq.com/s/imLTVwgm8MOiY1_5s3rdFQ](https://mp.weixin.qq.com/s?__biz=MzIzOTU0NTQ0MA==&mid=2247502152&idx=1&sn=3c356a4b65d50e964f0350a13ba08df3&scene=21#wechat_redirect)

• Big reveal! How RocketMQ manages consumption progress:

   [https://mp.weixin.qq.com/s/rHs9L1gTuFs05Cs2F4JXOw](https://mp.weixin.qq.com/s?__biz=MjM5NTk0NjMwOQ==&mid=2651114644&idx=1&sn=fa93f0264989b536153dc683a246601a&scene=21#wechat_redirect)