[Problem Memo] Record a DNS resolution exception handling

background

• System upgrade and transformation, there is currently gateway A, for smooth upgrade, first deploy gateway B, and add a layer of forwarding layer (IAS) in front of gateways A and B.

• When the scheduled task calls the service interface under gateway B, the interface returns:

   {
    "timestamp":"2022-05-09T10:02:35.889+08:00",
    "path":"/xxx/yyy",
    "status":404,
    "error":"Not Found",
    "message":null,
    "requestId":"06f3daf5-67392626"
  }

• The request log of the forwarding layer (IAS) does not find the request record, nor does the request log of gateway A.

identify the problem

1. Confirm whether IAS and gateway B are normal: do not use the domain name, request the interface through IP, and the interface call is successful. Conclusion: normal.
2. Confirm whether the DNS is normal: log in to the container, execute curl in the container, and the interface call is successful. Conclusion: normal.

To sum up: The initial suspicion is that there is a problem with the program calling the service. The service uses the spring-webflux WebClient call interface of ---0389549549c65f00c05b530fe33a5433---, and needs to check WebClient whether there is a DNS cache.

question confirmed

Through Google keyword search, a github issue was successfully located: Document How To Switch DNS Resolver Issue #1431 reactor/reactor-netty GitHub , and then through the link in the question, locate the official website Reactor Netty Reference Guide (projectreactor.io ) , according to the description of the official document, we know that webclient the bottom layer used reactor-netty has cached DNS, the default longest cache time is the maximum value of Int, that 21亿秒 .

If it is the cause of the cache, the service call should be normal after restarting. After trying to restart, it is found that the service can indeed be called.

problem solved

Modify DNS cache duration

According to the documentation, it is possible to set the DNS cache time

 webClient = WebClient.builder()
        .clientConnector(new ReactorClientHttpConnector(
                HttpClient.create().followRedirect(true)
        ))
        .build();

Change it to the following

 webClient = WebClient.builder()
        .clientConnector(new ReactorClientHttpConnector(
                HttpClient.create()
                        .resolver(spec -> spec.queryTimeout(Duration.ofSeconds(dnsCacheMaxTimeToLive)))
                        .followRedirect(true)
        ))
        .build();

But the current project springboot version is 2.1.3.RELEASE , which corresponds to the default introduced reactor-netty version is 0.8.5.RELEASE , this version does not have resolver method.

upgraded version

• reactor-netty ，指定版本1.0.15 ，启动报错NoSuchMethodException， reactor-netty跟springboot netty-codec-http Version conflict.

  • Upgrade springboot version to 2.3.0.RELEASE .
  • By the way, upgrade the spring version to 5.3.18 to solve the spring zero-day vulnerability problem. Since springboot has introduced spring by default, you only need to modify the version introduced by default: configure maven's properties : <spring-framework.version>5.3.18</spring-framework.version> to solve the spring zero-day vulnerability problem.
  • Since spring-boot-dependencies is configured by default with netty.version , the same as above, configure maven's properties : <netty.version>4.1.72.Final</netty.version> default version of springboot
  • Introduced io.projectreactor:reactor-core , version 3.4.17 .

• After the above upgrade is performed, the system starts successfully, but an error is reported when calling an external interface: DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144 . The new version of springboot limits the data packet size to 256K by default. Refer to the following article to solve the problem that the spring-cloud-gateway cache area is not enough . Since I am WebClient requesting an external interface to report an error, I created WebClient Time to increase the limit:

   webClient = WebClient.builder()
        .clientConnector(new ReactorClientHttpConnector(
                HttpClient.create()
                        .resolver(spec -> spec.queryTimeout(Duration.ofSeconds(dnsCacheMaxTimeToLive)))
                        .followRedirect(true)
        ))
        .codecs(configurer -> configurer
                .defaultCodecs()
                .maxInMemorySize(maxInMemorySize))
        .build();