GOREPLAY traffic recording and playback actual combat

Introduction to GoReplay

As the complexity of an application grows, the amount of work required to test it also grows exponentially. GoReplay provides us with a simple idea of reusing existing traffic for testing. GoReplay is a simple traffic recording plug-in developed by golang, which supports multiple methods of filtering, current limiting amplification, rewriting and other features. GoReplay can be completely non-invasive to the code, does not need to change your production infrastructure, and has nothing to do with language. It is not a proxy, but directly monitors the traffic on the network card.

How GoReplay works: The listener server captures the traffic and sends it to the replay server or saves it to a file, or saves it to kafka. Then the replay server will transfer the traffic to the configured address

Use process

Requirement: Receiving the requirement of the algorithm side, it is necessary to record the real production environment traffic and replay it to any environment at any time.

Since some scenes on the algorithm side are written in non-Java languages, the existing traffic recording platform cannot support it temporarily, and new recording components are needed to support the pressure measurement requirements, so goreplay was chosen.

GoReplay supports storing the recorded data in a local file, and then reading it from the file during playback. Considering the complexity of storing and distributing files during each recording and playback, we hope to use a more convenient way to manage data.
GoReplay also natively supports the storage of recorded data in Kafka, but when it is used, it is found to have greater limitations; when using Kafka to store data, it must be recorded and replayed at the same time. Its architecture is as follows

Process 1-4 cannot be split, can only be carried out at the same time

This will make the traffic recording and playback function very tasteless. We need to replay the recorded data at any time, and also support the replay of a recorded data multiple times. Now that it has stored traffic data in Kafka, we can consider revamping GoReplay to allow it to support our needs.

The reconstructed traffic recording and playback architecture diagram:

In the figure, stages 1-2 and 3-5 are independent of each other

In other words, the flow recording process and the playback process can be separated. You only need to record the offset of Kafka at the beginning and end of the recording to know what data the recording task contains. We can easily organize each piece of recorded data into a recording task, and then perform traffic playback when needed.

Transformation and integration

Kafka offset support transformation

Brief process:
The definition InputKafkaConfig in the source code

type InputKafkaConfig struct {
    producer sarama.AsyncProducer
    consumer sarama.Consumer
    Host     string `json:"input-kafka-host"`
    Topic    string `json:"input-kafka-topic"`
    UseJSON  bool   `json:"input-kafka-json-format"`
}

The modified definition of InputKafkaConfig

type InputKafkaConfig struct {
    producer  sarama.AsyncProducer
    consumer  sarama.Consumer
    Host      string `json:"input-kafka-host"`
    Topic     string `json:"input-kafka-topic"`
    UseJSON   bool   `json:"input-kafka-json-format"`
    StartOffset    int64  `json:"input-kafka-offset"`
    EndOffset int64  `json:"input-kafka-end-offset"`
}

In the source code, a fragment of data read from Kafka:
As you can see, the offset it selects is Newest

for index, partition := range partitions {
        consumer, err := con.ConsumePartition(config.Topic, partition, sarama.OffsetNewest)

        go func(consumer sarama.PartitionConsumer) {
            defer consumer.Close()

            for message := range consumer.Messages() {
                i.messages <- message
            }
        }(consumer)

    }

Modified fragment of reading data from Kafka:

for index, partition := range partitions {
        consumer, err := con.ConsumePartition(config.Topic, partition, config.StartOffset)
        offsetEnd := config.EndOffset - 1

        go func(consumer sarama.PartitionConsumer) {
            defer consumer.Close()

            for message := range consumer.Messages() {
                // 比较消息的offset, 当超过这一批数据的最大值的时候，关闭通道
                if offsetFlag && message.Offset > offsetEnd {
                    i.quit <- struct{}{}
                    break
                }
                i.messages <- message
            }
        }(consumer)
    }

At this time, just specify the range of kafka offset when starting the playback task. We can achieve the effect we want.

Integrated into the stress testing platform

Simply fill in the selection operation through the page, and then generate a startup command to replace the lengthy command writing

StringBuilder builder = new StringBuilder("nohup /opt/apps/gor/gor");
// 拼接参数 组合命令
builder.append(" --input-kafka-host ").append("'").append(kafkaServer).append("'");
builder.append(" --input-kafka-topic ").append("'").append(kafkaTopic).append("'");
builder.append(" --input-kafka-start-offset ").append(record.getStartOffset());
builder.append(" --input-kafka-end-offset ").append(record.getEndOffset());
builder.append(" --output-http ").append(replayDTO.getTargetAddress());
builder.append(" --exit-after ").append(replayDTO.getMonitorTimes()).append("s");
if (StringUtils.isNotBlank(replayDTO.getExtParam())) {
  builder.append(" ").append(replayDTO.getExtParam());
}
builder.append(" > /opt/apps/gor/replay.log 2>&1 &");
String completeParam = builder.toString();

The stress testing platform controls the start and stop of the GoReplay process through the interface exposed Java agent

String sourceAddress = replayDTO.getSourceAddress();
String[] split = sourceAddress.split(COMMA);
for (String ip : split) {
  String uri = String.format(HttpTrafficRecordServiceImpl.BASE_URL + "/gor/start", ip,                                                  HttpTrafficRecordServiceImpl.AGENT_PORT);
  // 重新创建对象
  GoreplayRequest request = new GoreplayRequest();
  request.setConfig(replayDTO.getCompleteParam());
  request.setType(0);
  try {
    restTemplate.postForObject(uri, request, String.class);
  } catch (RestClientException e) {
    LogUtil.error("start gor fail，please check it！", e);
    MSException.throwException("start gor fail，please check it！", e);
  }
}