CDH6.3.2 + flink1.12.4集成环境。HBASE sink 24小时之后插入失败

Question

查看了hosts文件正常
查看了主机的连通性没有问题
配置文件使用的集群的客户端配置。
hbase client版本使用的是hbase-2.1.0-cdh6.3.2
flink run启动后第一天表现正常，第2天(24小时之后)就不行了。

2021-12-12 15:12:26,986 ERROR org.apache.hadoop.hbase.client.AsyncProcess - Failed to get region location
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=11, exceptions:
Sun Dec 12 15:12:26 CST 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60105: Call to worker09.xs.com/10.52.40.212:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=10, waitTime=60003, rpcTimeout=60000 row 'portrayal:DM_tag_new,0041000251211202,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=worker09.xs.com,16020,1633584310562, seqNum=-1

at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:299)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:242)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:267)
at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:435)
at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:310)
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:595)
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:857)
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:755)
at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:273)
at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:228)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.doFlush(BufferedMutatorImpl.java:303)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:280)
at sink.CertHbaseSink.invoke(CertHbaseSink.java:59)
at sink.CertHbaseSink.invoke(CertHbaseSink.java:22)
at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:54)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:191)
at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:204)
at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:174)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:398)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:191)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:619)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:583)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:758)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:573)
at java.lang.Thread.run(Thread.java:748)

Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=60105: Call to worker09.xs.com/10.52.40.212:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=10, waitTime=60003, rpcTimeout=60000 row 'portrayal:DM_tag_new,0041000251211202,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=worker09.xs.com,16020,1633584310562, seqNum=-1

at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:159)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more

Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to worker09.xs.com/10.52.40.212:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=10, waitTime=60003, rpcTimeout=60000

at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:205)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
at org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:96)
at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:199)
at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:682)
at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:757)
at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:485)
... 1 more

Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=10, waitTime=60003, rpcTimeout=60000

at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:200)
... 4 more