This article explains FISCO BCOS performance stress test (with example tutorial)

"How much TPS can my blockchain run?" "Can it reach the peak that the 'official' said?" "Why can't it always go up? Is my machine not good enough?"

Nowadays, blockchain technology is widely used in all walks of life, and it is also challenged by massive users and massive data. More and more students are beginning to pay attention to the performance of chains and contracts.
This article provides you with detailed FISCO BCOS performance stress testing guidelines from the three aspects of principle, practical operation and skills, combined with examples to demonstrate, and summarizes the practical skills and common problems of stress testing, so that you can better improve performance.

Pressure measurement principle

The principle of stress testing is actually not complicated. Start one or a bunch of blockchain clients, first deploy a contract for stress testing or a smart contract that needs to evaluate performance on the chain, and then go all out on the chain.” Concurrently send the transaction, after receiving the transaction execution result (transaction receipt) returned by the blockchain, TPS is counted.

Since the blockchain distributed network broadcasting, transaction queuing and packaging, consensus confirmation and other processes are still relatively long, the middle is full of technical details, which often leads to the results not as expected. This requires the environment, parameters, stress testing procedures and contract logic. It takes a lot of effort to get the desired result (using the computing resources to the extreme), and the exact result (my environment peaks so high).

In-depth stress test details

"Adjust parameters" before stress test

The stress test is generally a long-term "strength work", and you must be well prepared before starting it. The following key parameters should be adjusted to "optimal" first.
Total number of transactions : The total number of transactions sent by the stress test program in one stress test. It should be emphasized that if there are no transactions of a certain order of magnitude, the stress test results have no statistical meaning. For example, the actual TPS should be able to reach 5000, then if only a few thousand to ten thousand to twenty thousand transactions are sent, the result is not significant due to the influence of boundary conditions such as program start and stop, at least more than two orders of magnitude of transactions (such as 100,000 transactions, the entire number of transactions) The process lasts for more than 1 minute), and the TPS performance in the smooth processing stage can be seen.

QPS (Queries Per Second) : The number of requests per second, that is, the rate at which the client sends transactions under pressure. Figuratively speaking, it is: whether the pressure test client can "feed" the chain nodes. For example, if the TPS performance of the chain is in the 10,000-level, then whether the sending capacity of a pressure test client is sufficient, we need to evaluate it specifically. If one is not enough, we need to start multiple Stress test the client, and the mode sent by each client should be asynchronous. If one transaction is sent every time and then synchronously waits for the result to send the next one, it is difficult to "feed" the blockchain nodes (how to ensure that Asynchronous, see SDK interface definition).

However, the QPS cannot be too large. When the QPS is too large, transactions will accumulate in the transaction pool. These stacked transactions cannot be executed immediately, so that the measured TPS will be less than the actual TPS.

Contract complexity : The complexity of the contract will have a great impact on the TPS result. The simple understanding of contract complexity is the amount of code (the number of instructions), the amount of data (specifically, the amount of state data involved in the contract), and a very important point is whether the contract supports parallel transaction logic. Only in parallel can the server resources be exhausted. To achieve the effect of parallel processing, in addition to the fact that the pressure test client needs to be sent asynchronously as mentioned in the previous article, the contract of FISCO BCOS supports DAG parallel computing. An ordinary contract, if it can support the transaction In parallel, open the DAG parallel setting to use up all the CPU computing power on the server, otherwise it is actually "serial" processing, and only one CPU is used, and the effect will definitely not be very good.
For DAG parallel contract development, see:
https://fisco-bcos-documentation.readthedocs.io/zh_CN/latest/docs/manual/transaction_parallel.html）

chain parameter configuration : There are several key chain parameters related to performance, including how long it takes to generate a block and how many transactions can be packaged in a block. In the stress test scenario, these numbers should be configured appropriately. By default, we try to pack the transactions in the transaction pool (size: [tx_pool].limit) into a block (packaging interval: [consensus].min_block_generation_time) at 500ms by default. , each with a default of up to 1000 transactions (maximum number of transactions in a block: tx_count_limit).

At the same time, the node will automatically guarantee to produce a block within 1 second. If the time to generate the current block is greater than 1 second, the next block will package less transactions. If it is less than 1 second, the transaction will be packaged as much as possible according to the set upper limit of the maximum number of transactions in the block. If it is purely for stress testing, it is recommended to reduce the packing interval as small as possible, and repeatedly adjust the maximum number of transactions in the block, so as to make the block generation time even and stable, and try to approach the maximum value as much as possible. Here, it is recommended to set the packing interval to 1ms, and the maximum number of transactions in a block is at least 10,000, and adjust upwards according to the actual results. You can refer to the "FISCO BCOS Configuration Document" at the end of the article for configuration.

hardware configuration : If you want to get the highest possible TPS, you should use hardware with better performance. Specifically, the more cores the server has, the better, and the more ample memory the better. Be sure to use high-speed mechanical hard disks or SSD hard disks, and use relatively low-speed network storage devices with caution. The IO speed of the hard disk writes the blocks and states of the chain. Speed has a huge impact; the larger the network bandwidth, the lower the latency, the better, to ensure that the network between the stress test program and the chain nodes, as well as the network between nodes is smooth and efficient.

Other adjustments : It is better to set the log level of the stress test client and chain to the Error level to minimize the amount of log output. If there are other programs in the environment that take up resources (CPU, memory, hard disk network, etc.), you may wish to temporarily quit. A developer's stress test has filled the hard disk before, so it is necessary to free up the hard disk in advance.

pressure test result meaning

• Number of correct/incorrect transactions: If the stress test result contains many incorrect or timed out transactions, then this stress test is of little significance. Generally speaking, a correct rate of 99% and above is meaningful for business.
• TPS (Transactions Per Second): The number of transactions processed per second, that is, the performance of the blockchain in processing this type of transaction.
  

pressure measurement operation

Use the existing pressure test program Java-SDK-Demo

FISCO BCOS provides a stress test program based on Java-SDK: Java-SDK-Demo.
*The Caliper-based stress test program is provided in the old version, and developers are welcome to continue to optimize the Caliper stress test program. In this article, Java-SDK-Demo is more recommended.
Java-SDK-Demo Github address

• Stress test FISCO BCOS 2+ version:

https://github.com/FISCO-BCOS/java-sdk-demo/tree/release-2.8.0

• Stress test FISCO BCOS 3+ version:

https://github.com/FISCO-BCOS/java-sdk-demo

stress test scene

• transfer scene
• KVTable scene
• DMC scene
• For more information, please refer to the stress test program catalog

pressure measurement method

README under the reference project

Operation example
This operation takes the transfer scenario in the Java-SDK-demo as an example.

• Transfer Contract: ParallelOk.sol
• Stress test program: ParallelOkPerf.java

Step 1: View the pressure test command
*The actual meaning of [tps] in the command is QPS, that is, the transaction sending rate of the stress test program.

$ cd java-sdk-demo/dist/
$ java -cp 'conf/:lib/*:apps/*' org.fisco.bcos.sdk.demo.perf.ParallelOkPerf
Usage:
===== ParallelOk test===========
java -cp 'conf/:lib/*:apps/*' org.fisco.bcos.sdk.demo.perf.ParallelOkPerf [parallelok] [groupId] [add] [count] [tps] [file] [enableDAG].
java -cp 'conf/:lib/*:apps/*' org.fisco.bcos.sdk.demo.perf.ParallelOkPerf [parallelok] [groupId] [transfer] [count] [tps] [file] [enableDAG].

Step 2: Execute the stress test program

The operation of the stress test in the transfer scenario is divided into two steps:

1. Preparation: Generate transfer users in batches and initialize the user amount (add)
   *After execution, the user list file user1000.txt is generated, which will be used in the next step.

# java -cp 'conf/:lib/*:apps/*' org.fisco.bcos.sdk.demo.perf.ParallelOkPerf parallelok] [groupId] [add] [count] [tps] [file] [enableDAG]
# 参数：压测parallelok合约，群组group0，操作是添加用户add，用户数1000，qps=1000，要生成的用户列表文件名，启动交易并行执行
java -cp 'conf/:lib/*:apps/*' org.fisco.bcos.sdk.demo.perf.ParallelOkPerf parallelok group0 add 1000 1000 user1000.txt true

2. Stress test: transfer between users
   *This step can be repeated continuously to adjust the parameters for pressure measurement.

# java -cp 'conf/:lib/*:apps/*' org.fisco.bcos.sdk.demo.perf.ParallelOkPerf [parallelok] [groupId] [transfer] [count] [tps] [file] [enableDAG]
# 参数：压测parallelok合约，群组group0，操作时用户转账transfer，总交易数10000，qps=1000，使用的用户列表文件，启动交易并行执行
java -cp 'conf/:lib/*:apps/*' org.fisco.bcos.sdk.demo.perf.ParallelOkPerf parallelok group0 transfer 10000 1000 user1000.txt true

3: View

TPS

• Include error transactions: TPS (include error requests)
• Exclude error transactions: TPS (exclude error requests)

of correct/incorrect transactions

validation:
user count is 1000
verify_success count is 1000
verify_failed count is 0

full result

Total transactions: 10000
Total time: 12676ms
TPS(include error requests): 788.8923950773114
TPS(exclude error requests): 788.8923950773114
Avg time cost: 1221ms
Error rate: 0.0%
Time area:
0   < time < 50ms   : 21 : 0.21%
50   < time < 100ms : 1559 : 15.590000000000002%
100 < time < 200ms : 2460 : 24.6%
200 < time < 400ms : 318 : 3.18%
400 < time < 1000ms : 1562 : 15.620000000000001%
1000 < time < 2000ms : 1194 : 11.940000000000001%
2000 < time           : 2886 : 28.860000000000003%
===================================================================
validation:
user count is 1000
verify_success count is 1000
verify_failed count is 0

test custom contract

To develop a stress testing program and stress testing a custom contract, just like developing a blockchain application, you can refer to the blockchain application development documentation, and you can refer to the imitation.
Taking Ok.sol as an example, the contract contains a basic on-chain asset transfer logic, which is a relatively streamlined contract and is suitable for testing the basic processing capabilities of the chain. If your business logic is more complex, please understand this clearly first: the more complex the contract logic, the lower the TPS. (So we recommend that transactions on the chain be as concise as possible, including only core transactions that require multi-party consensus.)

1. Find out the contract under stress test (Ok.sol);
2. Compile the contract to generate java code (Ok.java) and integrate it into Java-SDK-Demo;
3. Write a stress test program (PerformanceOk.java);
4. Compile the code and deploy it to your stress testing environment;
5. Stress testing, and ensure that all transaction results are normal, wrong transaction results make stress testing meaningless.

All development is carried out in Java-SDK-Demo. The three files and directory structures to be developed are as follows:

• Contract: Ok.sol
• The Java code compiled by the contract: Ok.java
• Stress test program: PerformanceOk.java

java-sdk-demo/src/main/java/org/fisco/bcos/sdk/demo
                                              ├── contract
                                              │   ├── Ok.java
                                              │   └── sol
                                              │       └── Ok.sol
                                              └── perf
                                                  └── PerformanceOk.java

Performance Boosting Tips

When you have painstakingly completed a stress test and found that the results are not very satisfactory, then you should go back and start over from the "Parameter Adjustment" section. Here are some key skills that will give you more control over the stress testing process.

1. Environment : During the stress testing process, closely observe the environment parameters of the stress testing client and chain. include:
   CPU utilization, to ensure that the cores of multiple CPUs are as full as possible;

Network traffic to determine whether the network speed has become a bottleneck;

Hard disk IO situation (especially the IO Wait indicator), to determine whether the bottleneck of storage capacity has been reached;

The memory is generally not a big problem, but in extreme cases, if the memory is insufficient, the stress test process may hang.

The above observation methods can all use basic Linux (or corresponding operating system) basic commands. As a stress test executor, you must be proficient in the operating system and have a sufficient understanding of the information printed by the command.

2. Optimize contract logic : For example, reduce contract interface parameters, use complex data structures such as arrays and mappings with caution, simplify calculation logic and judgment logic in contracts, reduce unnecessary events (Event), and consider whether it is possible to consider large data. Only put the hash of the data on the chain; confirm whether the DAG parallel contract engine can be used. Another big killer is to use FISCO BCOS precompiled contracts to implement contract logic, which is very useful for developers who pursue extreme performance.
3. attention to the log : including the log of the stress test client, the log of the node, and the system log, especially whether there are errors or warnings in it. Once an exception occurs, it should be dealt with immediately, otherwise it may affect the stress test results. There is usually detailed description information in the program log. You can refer to the description for local analysis.
4. Efficient communication : If you want to consult the pressure test in the community, it is recommended to fully check the configuration mentioned above, record the software and hardware indicators of the pressure test process, clarify the contract complexity, and first consider whether there is a serial , Parallel related problems and optimization space, and collect related log information, and then use multiple screenshots or integrated documents to send problems to community assistants or other expert partners at one time for efficient and in-depth discussion.

「FAQ」

---

Q Under what circumstances does FISCO BCOS achieve a pressure test of more than 20,000 TPS?
A consensus nodes are deployed on a single server, each server has 24 cores and 32G memory, and the configuration of 10G network can achieve more than tens of thousands of TPS on the basis of reasonable optimization of chain parameters and parallel computing of contracts.

Q I can only press 1K in my environment, how to improve it?
A Different contracts have a great impact on the results. Some contracts may only run a few hundred TPS on the best hardware, or some pits have been stepped on, such as whether the contract is called synchronously or the trading pool is too small Wait. You can refer to this article [Pressure Testing Skills] for continuous optimization.

Q My CPU is not running full, what is the reason?
A The most common reason is that the QPS is not enough, and other reasons may include: too few transactions in block packaging, and the number of transactions in large blocks can be adjusted; parallel contracts are not used; the storage capacity in the contract logic is too large, Storage becomes the bottleneck.

Q I increased the QPS, but there are many wrong transactions, how to adjust it?
A QPS is much larger than the actual processed TPS, causing a large number of transactions to be cached in the transaction pool to time out. In the stress test, only the QPS is only slightly larger than the TPS.

Q using the Raft consensus algorithm, is the performance higher than that of PBFT?
A consensus algorithm has little impact on TPS, and the bottleneck is execution and storage. In general, the stress measurement process is the ultimate pursuit of performance. It is not only a work of strength, but also requires a sufficient understanding of the operating system and the principles of the blockchain system, familiar with various parameter configurations, and how to collect data, Analyzing problems, finding bottlenecks and breaking through bottlenecks, there may be many repetitions in a stress test, and finally getting the desired results will be an achievement and joy. May your app take off in speed and scale.

Reference link

Pressure test program directory:

https://github.com/FISCO-BCOS/java-sdk-demo/tree/main/src/main/java/org/fisco/bcos/sdk/demo/perf
ParallelOk.sol：

https://github.com/FISCO-BCOS/java-sdk-demo/blob/main/src/main/java/org/fisco/bcos/sdk/demo/contract/sol/ParallelOk.sol

ParallelOkPerf.java：

https://github.com/FISCO-BCOS/java-sdk-demo/blob/main/src/main/java/org/fisco/bcos/sdk/demo/perf/ParallelOkPerf.java

application development document:

https://fisco-bcos-documentation.readthedocs.io/zh_CN/latest/docs/tutorial/sdk_application.html

Ok.sol：

https://github.com/FISCO-BCOS/java-sdk-demo/blob/main/src/main/java/org/fisco/bcos/sdk/demo/contract/sol/Ok.sol

PerformanceOk.java：

https://github.com/FISCO-BCOS/java-sdk-demo/blob/main/src/main/java/org/fisco/bcos/sdk/demo/perf/PerformanceOk.java

compiles the contract to generate java code:

https://fisco-bcos-documentation.readthedocs.io/zh_CN/latest/docs/tutorial/sdk_application.html#id6

FISCO BCOS configuration file:

https://fisco-bcos-documentation.readthedocs.io/zh_CN/latest/docs/manual/configuration.html

precompiled contract:

https://fisco-bcos-documentation.readthedocs.io/zh_CN/latest/docs/manual/precompiled_contract.html

Parallel contract development method:

https://fisco-bcos-documentation.readthedocs.io/zh_CN/latest/docs/manual/transaction_parallel.html

For more dry goods, please pay attention to the FISCO BCOS open source community public account, visit the FISCO BCOS code repository to download all source codes of the project: https://github.com/FISCO-BCOS/FISCO-BCOS , welcome Click on the upper right of the page Angular star collection , get the latest version.