Dewu technology builds a complete message system from 0 to 1

background

With the rapid development of the company, the volume of messages pushed every day is increasing. The previous old message system has become increasingly unable to meet the functional requirements of the push scenario at this stage, and we started to build a complete message system from 0 to 1.

The past of the messaging platform

There are various pain points and challenges in the original architecture:

• Slow access, slow sending

Service access is relatively slow, and different types of messages need to be connected to different apis. Message consumption processing is very slow, which affects the event operation experience.

• No traffic analysis, each business directly calls each other

The business call volume statistics are not clear, and it is impossible to close/limit the traffic for different businesses.

• Lack of special news priority push

Lack of priority messages, when marketing messages are pushed in large quantities, normal order-related messages will accumulate, and blocking will be delayed.

Optimize through the following 4 aspects:

• Unified access interface, business identity recognition
• Speed up sending, convert a single consumption to multiple simultaneous consumption
• Support message priority processing

• Switch data source storage, choose mongodb that writes a bit higher than es and reads a bit less than es
  

  The present of the messaging platform

  The current overall structure of the messaging platform*
  

  How to implement the priority queue in the above process?

  At present, the priority of the message platform is realized by two types of sending. First, the traditional message queue kafka is used for priority sending and consumption, and then the priority thread pool task is used for priority message sending.

  Queue priority

  Kafka itself does not support priority. We use the following two solutions to artificially send messages of different priorities to different queues in Kafka.

By creating different topics, different priority messages are sent to different topics, and at the same time, when the messages are consumed, the data of different topics is obtained according to different proportions for consumption.

The current order is that the highest priority is twice as high as the second highest, and proceed in order. Finally, the remaining pull message value is added to the highest priority. For example, if you pull 50 items and 3 topics at a time, then we will pull them according to (25 +5/ 13 / 7).

If high priority messages are gone, how can I pull low priority messages at full load, that is, pull 50 messages at a time according to the above-mentioned lowest priority messages?

Introduce the message pull state machine, when the priority message is relatively low, increase the consumption of the low priority. The current message service state machine has several states such as initialization, low load, high load, etc. The current state of the message consumer is determined by judging the number of messages processed last time and the pull parameter is modified. Currently, it is modified by reflection. The number of kafka pulls.

In order to speed up the sending, we also use a local thread pool, local thread tasks, we use task priority queues. The following is the process of submitting a thread task.

In the above figure, we compare a thread task with an auto-incremented sequence number and the previously defined priority value to uniquely determine the execution priority of a task.

How do we implement the delay queue in the overall flowchart?

First of all, we define the delay/timing strategy and have the following strategies:

• Push messages more than 30 minutes
• Push messages in less than 30 minutes
• Push messages less than 15s

After we distinguish the above three types of delayed timed messages, there are different implementation methods respectively. When it is less than 15s, we directly use the delayTask that comes with java for message judgment & push. For those higher than 15s and lower than 30 minutes, we have created a single time wheel with a basic level of one second to push messages. The following is the execution of the time wheel.

But we have performed some optimizations on this basis, referring to the delay queue of kafka. When there is no task to be executed in the time round, we directly wait for the execution thread until the next task is submitted to notify and wake up. For delayed tasks greater than 30 minutes, we generally store the message task first. We add the task data to the second-level time wheel 30 minutes before the task is about to be executed, and refer to the second method for message sending. (Why don't you use the day/hour level time wheel, just don't want to waste memory)

In our message push process, users’ anti-fatigue is necessary. At present, the anti-fatigue scenarios of the message center mainly include the following types:

• User cannot receive M messages for N days
• Cannot receive M messages within N days in a specific scenario
• A specific business can only receive 1 message in a day

In the above scenarios, we mainly use mongo for data storage and aggregation query, because if you use the redis scenario in multiple users, redis will be frequently operated, which is not very good. And we only keep most of the anti-fatigue data for 1 week at most. The collection of mongo can easily satisfy our functions. Of course, in a scenario where only one message can be received for a specific business within one day, we use redis' helperLogLog to prevent fatigue and reduce query and memory consumption. Although there is a little error, the impact on our usage scenarios is not very high.

Finally, we summarize the points that need to be considered to build a messaging platform from our own actual combat:

• Simple and easy to access
• Respond quickly without affecting business
• Urgent messages are delivered to users as soon as possible, and the messages can be classified
• The message can be traced back and withdrawn
• Effect visualization
• Content security

Pay attention to Dewu Technology, and work hand in hand to the cloud of technology