Alibaba&#39;s practice and optimization on the open source stress testing tool JMeter

Authors: Ling Ran, Jianquan

Apache JMeter [1] is an open-source stress testing tool under Apache. It was founded in early 1999 and has a history of more than 20 years. JMeter has rich functions and a large community (user group), and is one of the mainstream open source stress testing tools.

Performance testing usually focuses on the launch of new systems or before large-scale events (such as e-commerce promotions, Spring Festival events, etc.) to verify system capabilities and help troubleshoot performance bottlenecks and other issues.

A stress test activity can be roughly divided into several steps:

Scene configuration. Configure a stress test scenario to simulate the interaction between users (services) and the system.
Pressure test execution. Start the pressure measurement at the specified pressure level.
Stress monitoring and analysis. Pressure testing usually focuses on key indicators such as pressure RPS, success rate, service response time (RT), and network bandwidth.
Summary of the report. Disclose whether the system capability meets the requirements, and record the evolution and optimization process of system performance at the same time.

Native JMeter implements stress testing

Edit the stress test script on the JMeter GUI page and click the Start button to debug the JMeter script. For details, please refer to the JMeter official website [1] .

For simple scenarios and low test concurrency requirements, JMeter local testing can meet the requirements. However, with the increase of Internet users, the demand for the system to carry more concurrency is increasing, and the pressure capacity of a single JMeter press has a certain upper limit, so it is necessary to use multiple presses to improve the pressure capacity of JMeter. , which requires the use of JMeter's distributed pressure function.

JMeter's distributed pressure measurement requires users to manage and maintain multiple machines by themselves. Pay attention to the following points during use:

The presser's firewall is closed or the correct port is open. SSL is set up or disabled for RMI.
All presses are on the same subnet. If you use a 192.xxx or 10.xxx IP address, the server is on the same subnet.
Use the same version of JMeter and Java on all presses.
All presses have already copied the sliced CSV data files, dependency jars, etc.
During the pressure measurement process, it is necessary to monitor whether the pressure machine is sending flow normally, and keep the pressure consistent with the configuration.
The collection of monitoring data is configured before pressure is applied to facilitate the generation of reports after the pressure test is completed.

It can be seen that the distributed pressure measurement of JMeter needs to coordinate various resources, and the pre-preparation and the maintenance of the pressure engine during the pressure application process are troublesome, and the pressure measurement efficiency is low for the personnel who implement the pressure measurement.

JMeter in Practice on the Cloud

Alibaba has a very rich business form. Behind each business form, a series of distributed technology systems provide services. With the rapid development of the business, especially in the event scenarios such as Double 11 and other big promotions, accurate evaluation The service capability of the entire business site has become a major technical problem.

During this process, we built our own full-link stress testing system to meet more complex and diverse stress testing requirements, and exported this technology to the performance testing PTS, while supporting native JMeter stress testing.

Practice JMeter with the console

upload script

Open the home page of [2] of the PTS console, select Pressure Measurement Center > Create Scenario > JMeter Pressure Measurement in the left navigation bar, and create a new JMeter pressure measurement scenario. Fill in the scene name, such as jmeter-test. Click the upload file button on the scene configuration page to upload the test.jmx script that has passed the local test.

pressure configuration

On the pressure configuration page, the number of concurrency is set to 50, and the pressure measurement duration is set to 2 minutes.

Save pressure test

Click Save to remove the stress test, a prompt box will pop up and click OK, and PTS will start to execute the JMeter script on the cloud engine to initiate stress.

The page under pressure test is as follows:

Note: Due to differences in machine configuration and network environment (PTS pressure machine defaults to 4-core 8G, BGP multi-line public network), the pressure test results on the PTS may be different from the local pressure test results. In addition, the pressure configuration on the PTS will overwrite the configuration in the original script. It does not matter whether the original script is a fixed configuration or a JMeter property configuration.

Practice JMeter with OpenAPI

Cloud computing will develop like water, electricity and coal, and become the infrastructure of society. OpenAPI is like a fast pipeline, connecting enterprises and Alibaba Cloud, and continuously delivering resources to enterprises. The use of cloud computing to build IT infrastructure is the future development trend, which has become a social consensus. OpenAPI is an important window for the opening of cloud services. Cloud services without OpenAPI will be difficult to be integrated by customers' systems, which not only affects user experience, but also restricts the development of cloud vendors themselves. Similarly, in the field of pressure measurement, with the increasingly diversified pressure measurement requirements, more users want to inherit the pressure measurement capabilities on the cloud to their own systems, or to arrange a custom pressure measurement platform according to their own business systems, so as to Realize automated and customized pressure measurement requirements.

The following code implements the one-click JMeter stress test scenario using PTS's OpenAPI, and the stress test report is viewed after the stress test is completed.

Introduce pom dependencies

<!--创建PTS场景需要的实体类，如果只使用JMeter压测则不需要引入-->
<dependency>
  <groupId>com.aliyun</groupId>
  <artifactId>pts-api-entity</artifactId>
  <version>1.0.1</version>
</dependency>
<!--PTS Java SDK依赖。-->
<dependency>
  <groupId>com.aliyun</groupId>
  <artifactId>pts20201020</artifactId>
  <version>1.8.10</version>
</dependency>
<!--阿里云核心库。-->
<dependency>
  <groupId>com.aliyun</groupId>
  <artifactId>aliyun-java-sdk-core</artifactId>
  <version>4.5.2</version>
</dependency>

Copy the following code

import com.aliyun.pts20201020.Client;
import com.aliyun.pts20201020.models.*;
import com.aliyun.teaopenapi.models.Config;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;


public class StartingDemo {

    public static void main(String[] args) throws Exception {
        Client client = getClient();
        // 创建场景
        String sceneId = createScene(client);
        // 启动场景
        String reportId = startTesting(client, sceneId);
        // 最多等待次数
        int count = 0;
        // 查询是否已生成报告
        while (!hasReport(client, reportId) && count++ < 20) {
            // 若报告还未生成，则等待(30s)一段时间再查询
            // 根据压测时间酌情等待
            Thread.sleep(30 * 1000);
        }
        // 查看报告
        getJMeterReport(client, reportId);
    }

    private static boolean hasReport(Client client, String reportId) throws Exception {
        ListJMeterReportsRequest request = new ListJMeterReportsRequest();
        // 分页设置
        request.setPageNumber(1);
        request.setPageSize(1);
        // 查询条件设置
        request.setReportId(reportId);
        ListJMeterReportsResponse response = client.listJMeterReports(request);
        return response.getBody().getReports().size() > 0;
    }

    private static void getJMeterReport(Client client, String reportId) throws Exception {
        // 查看机器日志
        GetJMeterLogsResponse getJMeterLogsResponse = getJMeterLogs(client, reportId);
        List<Map<String, ?>> logs = getJMeterLogsResponse.getBody().getLogs();
        // 查看采样器聚合数据
        GetJMeterSampleMetricsResponse getJMeterSampleMetrics = getJMeterSampleMetrics(client, reportId);
        List<String> sampleMetricList = getJMeterSampleMetrics.getBody().getSampleMetricList();
        // 查看采样日志
        GetJMeterSamplingLogsResponse getJMeterSamplingLogs = getJMeterSamplingLogs(client, reportId);
        List<String> sampleResults = getJMeterSamplingLogs.getBody().getSampleResults();
    }

    private static GetJMeterSamplingLogsResponse getJMeterSamplingLogs(Client client, String reportId) throws Exception {
        GetJMeterSamplingLogsRequest request = new GetJMeterSamplingLogsRequest();
        // 分页设置
        request.setPageNumber(1);
        request.setPageSize(10);
        // 条件设置
        request.setReportId(reportId);
        GetJMeterSamplingLogsResponse response = client.getJMeterSamplingLogs(request);
        return response;
    }

    private static GetJMeterSampleMetricsResponse getJMeterSampleMetrics(Client client, String reportId) throws Exception {
        GetJMeterSampleMetricsRequest request = new GetJMeterSampleMetricsRequest();
        // 设置报告id
        request.setReportId(reportId);
        GetJMeterSampleMetricsResponse response = client.getJMeterSampleMetrics(request);
        return response;
    }

    private static GetJMeterLogsResponse getJMeterLogs(Client client, String reportId) throws Exception {
        GetJMeterLogsRequest request = new GetJMeterLogsRequest();
        // 分页设置
        request.setPageNumber(1);
        request.setPageSize(10);
        // 查询的压测引擎索引
        request.setReportId(reportId);
        GetJMeterLogsResponse response = client.getJMeterLogs(request);
        return response;
    }

    private static String startTesting(Client client, String sceneId) throws Exception {
        StartTestingJMeterSceneResponse startTestingSceneResponse = startTestingScene(client, sceneId);
        String reportId = startTestingSceneResponse.getBody().getReportId();
        return reportId;
    }

    private static StartTestingJMeterSceneResponse startTestingScene(Client client, String sceneId) throws Exception {
        StartTestingJMeterSceneRequest request = new StartTestingJMeterSceneRequest();
        request.setSceneId(sceneId);
        StartTestingJMeterSceneResponse response = client.startTestingJMeterScene(request);
        return response;
    }

    private static String createScene(Client client) throws Exception {
        SaveOpenJMeterSceneRequest request = new SaveOpenJMeterSceneRequest();
        // 定义场景
        SaveOpenJMeterSceneRequest.SaveOpenJMeterSceneRequestOpenJMeterScene scene = new SaveOpenJMeterSceneRequest.SaveOpenJMeterSceneRequestOpenJMeterScene();
        // 设置场景名
        scene.setSceneName("test");
        // 设置文件列表，包括JMeter脚本、JMeter压测依赖jar包、配置额度数据文件等
        List<SaveOpenJMeterSceneRequest.SaveOpenJMeterSceneRequestOpenJMeterSceneFileList> fileList = new ArrayList<SaveOpenJMeterSceneRequest.SaveOpenJMeterSceneRequestOpenJMeterSceneFileList>();
        // 设置文件的属性 需要设置文件的名称和文件公网可访问的oss地址
        SaveOpenJMeterSceneRequest.SaveOpenJMeterSceneRequestOpenJMeterSceneFileList testFile = new SaveOpenJMeterSceneRequest.SaveOpenJMeterSceneRequestOpenJMeterSceneFileList();
        testFile.setFileName("baidu.jmx");
        testFile.setFileOssAddress("https://pts-openapi-test.oss-cn-shanghai.aliyuncs.com/baidu.jmx");
        fileList.add(testFile);
        scene.setFileList(fileList);
        // 设置场景并发，可设置为100万
        scene.setConcurrency(1000000);
        // 设置引擎数量 说明：一台引擎最多能发500并发，最少1并发所以此处能设置的引擎数为[2,1000]，另外引擎数量越多消耗vum越快
        scene.setAgentCount(2000);
        // 设置压测持续时间 60s
        scene.setDuration(60);
        // 设置测试文件的名称，这个文件需包括在文件列表中
        scene.setTestFile("baidu.jmx");
        request.setOpenJMeterScene(scene);
        SaveOpenJMeterSceneResponse response = client.saveOpenJMeterScene(request);
        return response.getBody().getSceneId();
    }

    private static Client getClient() throws Exception {
        // 填写自己的AK/SK
        String accessKeyId = "ak";
        String accessKeySecret = "sk";
        Config config = new Config();
        config.setAccessKeyId(accessKeyId);
        config.setAccessKeySecret(accessKeySecret);
        Client client = new Client(config);
        return client;
    }
}

Fill in your own ak/sk

Fill in the correct ak/sk in the getClient of the above code

Click to start

Click the main method to start

Practice JMeter with plugins

For users who have been using JMeter for a long time, it still takes a certain amount of time to learn a new stress measurement tool. Therefore, PTS has developed a PTS-JMeter plug-in, which can help JMeter users to directly use PTS's pressure measurement resources without changing the original pressure measurement behavior. Users are hardly aware of the existence of the PTS-JMeter plug-in, which is the same as the native JMeter usage. Just save/open the JMeter script and click to start the stress test.

Download and install

Click the link to download the latest version jar package [3]

Copy the jar package to the lib/ext extension directory in the JMeter home directory

Click pressure test

Create a new JMeter script, or open an existing JMeter script, and click the PTS-JMeter start button to start the stress test

View report

During the stress testing process, the JMeter graphical interface will display some stress testing indicators, and users can go to the console to view the stress testing process at any time. After the stress test is over, PTS will generate a more detailed stress test report, which is retained for 30 days by default, and users can view it at any time in the console.

other

More detailed usage of PTS-JMeter plugin can be found in the PTS help document [4] .

Pressure measurement monitoring analysis

Performance testing is not just simply initiating stress, but monitoring and analyzing stress loads (RPS, network bandwidth, etc.) and business performance (RT, success rate, etc.) is also an important part of stress testing activities. Each request node (Sampler) in the JMeter script can set a name with business meaning (such as home and download page), we can call it business API. JMeter monitoring statistics are aggregated by business API name. For example, two request nodes with the same name will aggregate statistics into one business API. When configuring the script, it should be noted that different business API nodes should be configured with different names.

Business API stress load and performance

In actual work, there may be huge differences in the statistical data of different business APIs (for example, browsing commodity RT is usually much faster than submitting an order), so PTS displays each business API independently by default (such as the home and download pages displayed on the page in the above stress test). ).

The data PTS at each time point in the stress test is recorded in the background, and a complete and intuitive stress test report will eventually be formed. Click the business API real-time monitoring trend graph button to view the corresponding RPS, success rate, response time, network bandwidth and other monitoring data trends.

Business API Sampling Log

Many times we also want to see the details of a specific request execution. If 1% of requests fail, you need to view the complete request and response content to troubleshoot the cause of the failure. When testing scripts under the JMeter graphical interface, you can add View Results Tree to view the detailed information of a single request, but when performing a stress test, recording detailed information for each request is not only unnecessary, but also very resource-intensive, affecting the performance of stressing.

Alibaba Cloud PTS adopts a compromise method. The pressure engine samples each business API (pressure test sampler) at intervals to record a request details of success and failure (if any). In the stress test or stress test report page, click the View Sampling Log button to query the recorded request sampling information, and support searching and filtering by business API (stress test Sampler), response status (whether successful or not), request time, etc.

Click View Details to see individual request details. Currently, two display templates, general and HTTP, are provided for detailed information. The HTTP display template can be used for more friendly typesetting and display of HTTP requests. The display content includes request URL, request method, return code, complete request header, request body, and response header. , response body, etc.

Because only text content is displayed on the page, when the request body or response body contains images and other content that cannot be recognized as text, it may be displayed as garbled characters. In addition, when the request body or response body is large, the corresponding content may be truncated.

JMeter log

When executing a JMeter script locally, the default is to log to the jmeter.log file. When executing JMeter scripts on PTS, you can view JMeter logs in real time through the JMeter log page, and support query filtering based on log level, time or thread name.

JMeter logs are mainly used to troubleshoot error causes when script execution errors are reported. Some plugins may output some important information through JMeter logs, and users can also print logs directly in codes such as groovy scripts.

report summary

After the stress test is over, the PTS will aggregate the monitoring data to form a stress test report. The user analyzes and evaluates whether the system performance meets the requirements according to the stress test report, such as whether the RPS, success rate and RT (response time) meet the expectations. It can assist users in troubleshooting and analyzing business system performance bottlenecks.

On the PTS stress test report page, you can query the list of historical stress test reports.

Click View Report to open to view the report details. The stress test report is saved on the PTS for 30 days by default. You can click the report export button to export and save the PDF version of the stress test report to the local. The summary information of the stress test report includes summary data such as the stress test execution time, RPS, RT, and success rate. Scenario details include monitoring statistics of the full scenario dimension and the business API dimension.

Compared with manual command line execution of JMeter scripts, PTS is easier to use, provides simple and intuitive monitoring, and provides massive pressure capabilities.

Alibaba's practice and optimization on the open source stress testing tool JMeter

Native JMeter implements stress testing