Visualization of asynchronous request backlog | How to quickly locate function calculation backlog in 1 minute

Author | Qianfeng

This article is divided into three parts: the backlog problem is introduced in the overview, and the basic link of asynchronous calls of function computing is introduced; and the indicator view method is introduced in detail in the indicator introduction section, and the meaning of different indicators is classified and interpreted; finally, a common indicator is used. Take the asynchronous request backlog scenario as an example to introduce how to quickly locate the backlog problem within 1 minute.

Escort for asynchronous calls

The main concern of developers using function computing asynchronous calls is whether the call request can be processed in the expected time. If the processing is not completed, the client sees a backlog of asynchronous call requests. However, based on the previous function to calculate the asynchronous call indicator system, whether it is locating the backlog or checking the backlog, the process is very cumbersome.

In response to the above problems, Function Compute has launched a series of indicators related to the backlog of asynchronous call requests, which can help users quickly locate the request backlog and show users the quantified value of the backlog. This article will introduce in detail how to quickly locate the backlog of function asynchronous calls through these monitoring indicators, and explain the upgraded asynchronous call indicator system for developers.

Before we start, let's briefly introduce function computation asynchronous calls.

Asynchronous call is a way for function computing to call a function. Through asynchronous call, you can not only ensure that the function will be executed at least once, but also save the state transition information and execution result during the execution of the call. The call chain is as follows:

After the user/event source initiates an asynchronous call request, the request ID will be returned immediately, and then the function computing system will convert the relevant information of this call into message format, and put it in the MNS message queue for consumption by downstream modules in the system. The function call will be made based on the parsed call message.

After the call is completed, if the function is configured with a Destination, the system will further process based on the call result and the content of the Destination. For the introduction of the related content of Destination, please refer to the asynchronous call documentation:

https://help.aliyun.com/document_detail/181866.html

Indicator upgrade

The upgraded Function Compute asynchronous call link monitoring indicators mainly add the following categories:

Below we will explain the above indicators in detail.

Indicator view

At present, you can view the monitoring indicators of the function through the function computing console or the Serverless Devs tool. Below we will take the console as an example to guide you how to view the monitoring indicators related to the asynchronous call link, based on the serverless Devs viewing method You can refer to:

https://github.com/devsapp/fc/blob/main/docs/zh/command/metrics.md

The steps described below are based on the premise that the Function Compute service has been activated, and the service and function have been successfully created. If these operations have not been performed, please refer to Creating a Function Using the Console:

https://help.aliyun.com/document_detail/51783.html

First open the Function Compute console, click the monitoring panel tab on the left, and scroll to the bottom to view the asynchronous call processing of all services in the region and an overview table of the average delay of asynchronous message processing:

At this point, we click on any service name, and after entering, we can see the asynchronous call processing of all functions under the service; and an overview table of the average delay of asynchronous message processing:

Next, we click on any function name, and after entering, we can see the monitoring indicators of all function latitudes, and display them in the form of a graph:

So far, we have learned how to view these indicators. The following continues to introduce and interpret the above-mentioned asynchronous link-related indicators for developers.

Interpretation of indicators

We will classify and interpret monitoring indicators according to different indicator types.

Asynchronous call processing

Asynchronous request enqueue

In an asynchronous call, the number of requests that arrive at the function calculation, when the number of enqueued requests is greater than the number of completed request processing, it means that there is a backlog of requests, and the speed of the function processing asynchronous requests is lower than the speed at which asynchronous requests are initiated. Please adjust the upper limit of function elastic scaling (including reserved resources), refer to:

https://help.aliyun.com/document_detail/185038.html#task-2538034

Or you can search for DingTalk and join the Ali Function Computing official website customer group (11721331) to contact us to process pictures.

Asynchronous request processing completed

In an asynchronous call, the function calculates the number of completed requests, and the number of completed asynchronous requests should always be no greater than the number of asynchronous requests queued.

Asynchronous request backlog

Among the asynchronous requests that have arrived in Function Compute, the requests waiting to be processed and those being processed are regarded as backlog requests. The number of these requests is the number of asynchronous message backlogs. When this value is not 0, it means that there is a backlog of asynchronous call requests.

This indicator quantifies the backlog of asynchronous call requests, solves the problem of invisible backlogs, and greatly improves the observability of asynchronous calls. It is also one of the important contents of this upgrade.

Asynchronous request processing delay

Average processing delay

The delay from entering the processing queue to the start of processing the asynchronous function call request is averaged according to the specified time granularity. When this value is higher than expected, it indicates that there may be a backlog of requests for asynchronous calls to the function.

The three indicators of "Asynchronous Request Enqueue", "Asynchronous Request Processing Completion" and "Average Processing Delay" are placed in the overview chart of the monitoring market to help users quickly locate the backlogged functions and solve the difficulty in locating the backlog. The problem.

1 minute to locate backlog issues

In the previous asynchronous call indicator system, if you want to locate the backlog problem, you first need to find the backlog function. At this time, you need to view the details of its function monitoring indicators one by one. After the positioning is successful, you cannot directly see the specific backlog quantification value.

The upgraded asynchronous call indicator system can well solve the problem of difficult positioning of the backlog and quantification of the backlog. The following will focus on the scenario of the backlog problem and describe how to use the above indicators to quickly locate the backlog problem.

Business scene

Problem Description:

Xiao Zhang's business involves three functions, all of which are called asynchronously. One day, the user's business has a problem, and the asynchronous processing delay of each link increases. In order to quickly locate the problem, the user thought of the asynchronous link monitoring indicators, and performed the following positioning actions.

Positioning process:

First, open the monitoring dashboard at the regional level, select the target time period, and view the monitoring indicators of each service in the region;

It is found that the average processing delay of asynchronous calls of multiple services is higher than expected, and the number of asynchronous requests queued is greater than the number of completed request processing, indicating that these services have a certain degree of asynchronous call message backlog, and the asynchronous requests of A-Service are queued The difference between the number and the number of completed asynchronous call requests is the largest, and the backlog is the most serious. Click A-Service to view the monitoring indicators:

You can see that the function A-Function under this service is the backlog source. Click A-Function to view the monitoring indicators of the function latitude:

From the request backlog figure, we can see that the backlog started at 15:07. The current maximum number of unfinished asynchronous call requests under the account is about 7,000. At the same time, the average processing delay of asynchronous call requests is gradually increasing. It is currently around 300,000 milliseconds. The number of asynchronous call requests processed per minute is between 800 - 900.

Note: Since Xiao Zhang currently uses an account-level shared queue, the asynchronous request backlog displayed by the backlog of asynchronous calls under the entire account. If a function needs an exclusive queue due to business needs, you can contact the function computing team to activate it.

It was further found that the number of instances in the map of regional instances by quantity has been filled, so the reason is located because the number of A-Function requests surged, and the limit on the number of instances by quantity at the account level was full, which made the asynchronous calls of other functions also affected. The impact has caused every aspect of the business to be affected.

problem solved:

After locating the problem, Xiao Zhang immediately contacted the function computing team and adjusted the regional instance limit by volume based on the business volume.

At the same time, A-Function has the largest amount of calls, which may have a certain impact on the scheduling of asynchronous call requests in geographical latitude and the number of instances by volume, and affect the asynchronous call requests of other functions. Therefore, the function computing team recommends enabling exclusive access for A-Function. Queue function, and set the upper limit of elastic instance at the same time, so as to isolate the asynchronous call request of A-Function and avoid the impact on other functions.

Summarize

The upgraded Function Compute asynchronous call monitoring indicator system can help users solve problems such as difficulty in locating backlog problems and backlog quantification. Combined with the settings of cloud monitoring and alarms, it greatly improves the stability of Function Compute asynchronous call applications.

At the same time, in order to avoid the occurrence of request backlog as much as possible, we are currently optimizing the function computing asynchronous processing system, including the queue recycling mechanism, exclusive queue capability and backlog message redirection strategy, so as to improve the function computing system to handle asynchronous call requests Ability. In this way, through a powerful asynchronous call request processing system and a comprehensive monitoring indicator system, the asynchronous call of function computing is escorted.

Welfare

How can developers improve themselves? How can I expand my skills and understand the optimal learning path to get started quickly? Alibaba Cloud Serverless provides free and complete developer learning materials, and brings the most cutting-edge technical knowledge to you. Contains: technical e-books, technical conference materials collection, knowledge map, 18 introductory video courses, etc., to help all developers learn and progress together!

Follow the serverless official account to reply to the study in the background to get the download link of developer study materials!