[Users and products] Competitive product comparison program

background

In order to better actively perceive user-level competing product PK indicators, such as: Is the opening speed Is Huobi's WebSocket connection fast or Binance's fast?

At present, these are all question sentences. An important part of the woodpecker perception map is the comparison of competing product data. Now we rely on third-party testing platforms (such as Boce, Tingyun), but the data cannot meet our needs well, and the data must be direct In our own hands, safest , which is the so-called "data in hand, I have the world."

the goal

• Dockerized deployment
• Plug-in management
• Task scheduling collection
• Custom indicator

Architecture diagram

Operating environment

Based on Docker operation, the runtime is Node.js, which can be completed:

1. Collect data based on Puppeteer headless browser, such as: performance indicators, WebSocket indicators, interface indicators, etc.
2. Completion based on Ping: detection of domain name availability, TTL, packet loss rate

Purchasing multiple node servers, such as: Alibaba Cloud ECS 1U1G Beijing, Hangzhou, Shanghai and other nodes to install Woodpecker Docker Agent, in fact, can be understood as the concept of "broiler". We manage the plug-ins and status of these Agent installations in the remote Master. When the task is triggered, it is distributed to the corresponding node, and the node completes the work of data collection and reporting.

Note that the Agent is only running a Docker container in ECS, and the Master manages/operates this container. It will not operate on the host and will not override its authority (such as: SSH ECS, we think these operations are handled by OPS category).

Master

1. Plug-in management

In the form of code (Node.js) sub-module management, select the plug-in in the task, and configure the required parameters of the plug-in, the plug-in list is as follows:

1. Ping plugin

   • Configuration: domain name/IP, timeout, times

2. Website page performance indicator plugin

   • Configuration: link, view width and height, UserAgent
3. Huobi Global WebSocket Test
4. Binance WebSocket test
5. In fact, there is a lot of...

2. Agent management

View the status and distribution of all Agents, install/uninstall plug-ins, and manage their geographic distribution, labels and other information.

3. Task Management

Send data to the corresponding Agent in a task queue. The task configuration is roughly as follows:

• mission name
• Owning Agent, such as: bj01, sg03, hz03
• Task interval: single, 1s, 1m, 1h, 1h

• Execution plugin:

  1. Ping - { url: 'www.baidu.com', timeout: 1000, count: 5 }
  2. Web Performance - { url: 'https://www.huobi.me/zh-cn/exchange/' }
  3. Web Performance - { url: 'https://www.baidu.com' }
  4. Global WebSocket
  5. Binance WebSocket

Agent

Use Docker to run a container in the background, such as:

docker run \\
    -d \\
    --name woodpecker-agent \\
    -p 8080:8080 \\
    -e ACCESS\_KEY\=生成的令牌 \\
    hub.Woodpecker.vip/pk/agent:v1.0.0

OpenAPI provided by the Agent:

• IP:8080/open/v1/status-current Agent status
• IP:8080/open/v1/plugin/install-Install plug-in, upload plug-in data to Agent
• IP: 8080/open/v1/task/exec-Use data to execute tasks

Woodpecker

The log collection service is collected by the Agent and pushed to the service, and stored by the service.

note

1. All of the above use intranet transmission, such as: Master -> Agent -> Woodpecker -> DB
2. Frequent collection of third-party websites may face the risk of IP being blocked