java连redis-sentinel连不上,接下来如何排查?

Question

java连redis-sentinel连不上

Java连接redis-sentinel连不上

redis-sentinel是在k3s上部署的,使用helm部署的

用命令行查看一切正常

但是使用java连接报错,详细信息如下

环境准备

因为java的pod里面是没有redis的,所以下载一个redis-cli,然后拷贝的pod里面,再用命令行连接

[root@k8s-master01 ~]# kubectl get pods | grep spacex-file-8c8d67b87-ltbwp
spacex-file-8c8d67b87-ltbwp        1/1     Running     0                  15d
[root@k8s-master01 ~]# kubectl get pods | grep redis
redis-node-0                       2/2     Running     0                  13d
redis-node-1                       2/2     Running     0                  13d
redis-node-2                       2/2     Running     0                  13d
[root@k8s-master01 ~]# 

安装redis到pod中

[root@k8s-master01 ~]# kubectl cp redis-7.4.0.tar.gz spacex-file-8c8d67b87-ltbwp:/
[root@k8s-master01 ~]# kubectl exec -it kubectl  spacex-file-8c8d67b87-ltbwp -- sh
[root@k8s-master01 ~]# kubectl exec -it  spacex-file-8c8d67b87-ltbwp -- sh
/redis-7.4.0 # tar xzf redis-7.4.0.tar.gz
/redis-7.4.0 # cd redis-7.4.0/
/redis-7.4.0 # apk add --no-cache make gcc g++ musl-dev
/redis-7.4.0 # make
/redis-7.4.0 # make install
/redis-7.4.0 # redis-cli -p 6379 -h redis-node-0.redis-headless.default.svc.cluster.local -a poinuat@beijing
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
redis-node-0.redis-headless.default.svc.cluster.local:6379> ping
PONG
redis-node-0.redis-headless.default.svc.cluster.local:6379> 

直接连接主节点正常

/redis-7.4.0 # redis-cli -p 6379 -h redis-node-0.redis-headless.default.svc.cluster.local -a poinuat@beijing
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
redis-node-0.redis-headless.default.svc.cluster.local:6379> ping
PONG
redis-node-0.redis-headless.default.svc.cluster.local:6379> 
redis-node-0.redis-headless.default.svc.cluster.local:6379> 
redis-node-0.redis-headless.default.svc.cluster.local:6379> 
redis-node-0.redis-headless.default.svc.cluster.local:6379> set "testKey" "testValue"
OK
redis-node-0.redis-headless.default.svc.cluster.local:6379> get "testKey"
"testValue"
redis-node-0.redis-headless.default.svc.cluster.local:6379> set "testKey" "testValue3"
OK
redis-node-0.redis-headless.default.svc.cluster.local:6379> get "testKey"
"testValue3"
redis-node-0.redis-headless.default.svc.cluster.local:6379> 

连接从节点正常

从节点1:

/redis-7.4.0 # redis-cli -p 6379 -h redis-node-1.redis-headless.default.svc.cluster.local -a poinuat@beijing
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
redis-node-1.redis-headless.default.svc.cluster.local:6379> set "testKey" "testValue3"
(error) READONLY You can't write against a read only replica.
redis-node-1.redis-headless.default.svc.cluster.local:6379> get "testKey"
"testValue3"
redis-node-1.redis-headless.default.svc.cluster.local:6379> ping
PONG
redis-node-1.redis-headless.default.svc.cluster.local:6379> exit

从节点2:

/redis-7.4.0 # redis-cli -p 6379 -h redis-node-2.redis-headless.default.svc.cluster.local -a poinuat@beijing
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
redis-node-2.redis-headless.default.svc.cluster.local:6379> set "testKey" "testValue3"
(error) READONLY You can't write against a read only replica.
redis-node-2.redis-headless.default.svc.cluster.local:6379> get "testKey"
"testValue3"
redis-node-2.redis-headless.default.svc.cluster.local:6379> ping
PONG
redis-node-2.redis-headless.default.svc.cluster.local:6379> 

连接sentinel正常,查看sentinel状态正常

查看主节点信息

/redis-7.4.0 # redis-cli -h redis-node-0.redis-headless.default.svc.cluster.local -p 26379 -a "poinuat@beijing" sentinel get-master-addr-by-name mymaster
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
1) "redis-node-0.redis-headless.default.svc.cluster.local"
2) "6379"

查看从节1点信息

/redis-7.4.0 # redis-cli -h redis-node-1.redis-headless.default.svc.cluster.local -p 26379 -a "poinuat@beijing" sentinel slaves mymaster
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
1)  1) "name"
    2) "redis-node-1.redis-headless.default.svc.cluster.local:6379"
    3) "ip"
    4) "redis-node-1.redis-headless.default.svc.cluster.local"
    5) "port"
    6) "6379"
    7) "runid"
    8) "12967c54f9ea44b5058fc1487aaeb8a7abcd4a41"
    9) "flags"
   10) "slave"
   11) "link-pending-commands"
   12) "0"
   13) "link-refcount"
   14) "1"
   15) "last-ping-sent"
   16) "0"
   17) "last-ok-ping-reply"
   18) "63"
   19) "last-ping-reply"
   20) "63"
   21) "down-after-milliseconds"
   22) "60000"
   23) "info-refresh"
   24) "3684"
   25) "role-reported"
   26) "slave"
   27) "role-reported-time"
   28) "1204053512"
   29) "master-link-down-time"
   30) "0"
   31) "master-link-status"
   32) "ok"
   33) "master-host"
   34) "redis-node-0.redis-headless.default.svc.cluster.local"
   35) "master-port"
   36) "6379"
   37) "slave-priority"
   38) "100"
   39) "slave-repl-offset"
   40) "405401889"
   41) "replica-announced"
   42) "1"
2)  1) "name"
    2) "redis-node-2.redis-headless.default.svc.cluster.local:6379"
    3) "ip"
    4) "redis-node-2.redis-headless.default.svc.cluster.local"
    5) "port"
    6) "6379"
    7) "runid"
    8) "8dc03b4b7cac176094e42accf924b2951e74f389"
    9) "flags"
   10) "slave"
   11) "link-pending-commands"
   12) "0"
   13) "link-refcount"
   14) "1"
   15) "last-ping-sent"
   16) "0"
   17) "last-ok-ping-reply"
   18) "999"
   19) "last-ping-reply"
   20) "999"
   21) "down-after-milliseconds"
   22) "60000"
   23) "info-refresh"
   24) "2014"
   25) "role-reported"
   26) "slave"
   27) "role-reported-time"
   28) "1204003261"
   29) "master-link-down-time"
   30) "0"
   31) "master-link-status"
   32) "ok"
   33) "master-host"
   34) "redis-node-0.redis-headless.default.svc.cluster.local"
   35) "master-port"
   36) "6379"
   37) "slave-priority"
   38) "100"
   39) "slave-repl-offset"
   40) "405402347"
   41) "replica-announced"
   42) "1"
/redis-7.4.0 # 

从节点2信息:

/redis-7.4.0 # redis-cli -h redis-node-2.redis-headless.default.svc.cluster.local -p 26379 -a "poinuat@beijing" sentinel slaves mymaster
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
1)  1) "name"
    2) "redis-node-2.redis-headless.default.svc.cluster.local:6379"
    3) "ip"
    4) "redis-node-2.redis-headless.default.svc.cluster.local"
    5) "port"
    6) "6379"
    7) "runid"
    8) "8dc03b4b7cac176094e42accf924b2951e74f389"
    9) "flags"
   10) "slave"
   11) "link-pending-commands"
   12) "0"
   13) "link-refcount"
   14) "1"
   15) "last-ping-sent"
   16) "0"
   17) "last-ok-ping-reply"
   18) "407"
   19) "last-ping-reply"
   20) "407"
   21) "down-after-milliseconds"
   22) "60000"
   23) "info-refresh"
   24) "8467"
   25) "role-reported"
   26) "slave"
   27) "role-reported-time"
   28) "1204099255"
   29) "master-link-down-time"
   30) "0"
   31) "master-link-status"
   32) "ok"
   33) "master-host"
   34) "redis-node-0.redis-headless.default.svc.cluster.local"
   35) "master-port"
   36) "6379"
   37) "slave-priority"
   38) "100"
   39) "slave-repl-offset"
   40) "405431763"
   41) "replica-announced"
   42) "1"
2)  1) "name"
    2) "redis-node-1.redis-headless.default.svc.cluster.local:6379"
    3) "ip"
    4) "redis-node-1.redis-headless.default.svc.cluster.local"
    5) "port"
    6) "6379"
    7) "runid"
    8) "12967c54f9ea44b5058fc1487aaeb8a7abcd4a41"
    9) "flags"
   10) "slave"
   11) "link-pending-commands"
   12) "0"
   13) "link-refcount"
   14) "1"
   15) "last-ping-sent"
   16) "0"
   17) "last-ok-ping-reply"
   18) "407"
   19) "last-ping-reply"
   20) "407"
   21) "down-after-milliseconds"
   22) "60000"
   23) "info-refresh"
   24) "4135"
   25) "role-reported"
   26) "slave"
   27) "role-reported-time"
   28) "1204099255"
   29) "master-link-down-time"
   30) "0"
   31) "master-link-status"
   32) "ok"
   33) "master-host"
   34) "redis-node-0.redis-headless.default.svc.cluster.local"
   35) "master-port"
   36) "6379"
   37) "slave-priority"
   38) "100"
   39) "slave-repl-offset"
   40) "405433095"
   41) "replica-announced"
   42) "1"
/redis-7.4.0 # 

查看所有主节点的状态

/redis-7.4.0 # redis-cli -h redis-node-2.redis-headless.default.svc.cluster.local -p 26379 -a "poinuat@beijing" sentinel masters
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
1)  1) "name"
    2) "mymaster"
    3) "ip"
    4) "redis-node-0.redis-headless.default.svc.cluster.local"
    5) "port"
    6) "6379"
    7) "runid"
    8) "73d4c220f2a5632f265708b1213a84899d337209"
    9) "flags"
   10) "master"
   11) "link-pending-commands"
   12) "0"
   13) "link-refcount"
   14) "1"
   15) "last-ping-sent"
   16) "0"
   17) "last-ok-ping-reply"
   18) "319"
   19) "last-ping-reply"
   20) "319"
   21) "down-after-milliseconds"
   22) "60000"
   23) "info-refresh"
   24) "1795"
   25) "role-reported"
   26) "master"
   27) "role-reported-time"
   28) "1204235578"
   29) "config-epoch"
   30) "0"
   31) "num-slaves"
   32) "2"
   33) "num-other-sentinels"
   34) "2"
   35) "quorum"
   36) "3"
   37) "failover-timeout"
   38) "180000"
   39) "parallel-syncs"
   40) "1"
/redis-7.4.0 # 

sentinel的配置文件,保护模式也是关的

上网查了一些资料,说保护模式必须关闭,不然会有问题,查了一下配置也没发现问题

[root@k8s-master01 ~]# kubectl exec -it  redis-node-1 -c sentinel  -- ls  /opt/bitnami/redis-sentinel/etc 
]sentinel.conf
[root@k8s-master01 ~]# kubectl exec -it  redis-node-1 -c sentinel  -- ls  /opt/bitnami/redis-sentinel/etc/sentinel.conf
/opt/bitnami/redis-sentinel/etc/sentinel.conf
[root@k8s-master01 ~]# kubectl exec -it  redis-node-1 -c sentinel  -- cat  /opt/bitnami/redis-sentinel/etc/sentinel.conf
dir "/tmp"
port 26379
sentinel monitor mymaster redis-node-0.redis-headless.default.svc.cluster.local 6379 3
sentinel down-after-milliseconds mymaster 60000

# User-supplied sentinel configuration:
protected-mode no

# End of sentinel configuration
sentinel auth-pass mymaster poinuat@beijing
requirepass "poinuat@beijing"
sentinel myid 33535e4e17bf8f9f9ff9ce8f9ddf609e558ff4f2

sentinel known-sentinel mymaster redis-node-2.redis-headless.default.svc.cluster.local 26379 9fe32540b27937ed9f341b0f610a0d8df405bb63

sentinel known-replica mymaster redis-node-1.redis-headless.default.svc.cluster.local 6379

sentinel announce-hostnames yes
sentinel resolve-hostnames yes
sentinel announce-port 26379
sentinel announce-ip "redis-node-1.redis-headless.default.svc.cluster.local"

# Generated by CONFIG REWRITE
latency-tracking-info-percentiles 50 99 99.9
user default on sanitize-payload #0354006f12c761180d787948be8da86bec5c0739d45661914268e434006b37af ~* &* +@all
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
sentinel known-sentinel mymaster redis-node-0.redis-headless.default.svc.cluster.local 26379 2a09ba7abbb41ee71e79087310d75f9809c3c815
sentinel current-epoch 0

sentinel known-replica mymaster redis-node-2.redis-headless.default.svc.cluster.local 6379
[root@k8s-master01 ~]# 

pod描述

Last login: Fri Jan 17 03:41:21 2025 from tul.lan
[root@k8s-master01 ~]# kubectl describe pods redis-node-0
Name:             redis-node-0
Namespace:        default
Priority:         0
Service Account:  redis
Node:             k8s-master03/10.10.7.25
Start Time:       Fri, 03 Jan 2025 06:20:24 -0500
Labels:           app.kubernetes.io/component=node
                  app.kubernetes.io/instance=redis
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=redis
                  app.kubernetes.io/version=7.4.1
                  controller-revision-hash=redis-node-598c7d5948
                  helm.sh/chart=redis-20.5.0
                  statefulset.kubernetes.io/pod-name=redis-node-0
Annotations:      checksum/configmap: 2900b919c5bf287786a93865472095f1777114e320add542918dd241fa4720a4
                  checksum/health: 1eace71641fb5bd753e90012419e82c10b1180d694f91d83743b727a5aacca73
                  checksum/scripts: fc33fd958526ffdcd4e3fb24bf520effb44c02438dceb453de8d527c300d396e
                  checksum/secret: c9290c9a2640a506c0ce05773b81cbeab686b9bdcacce268476c0e546db2088e
Status:           Running
IP:               10.42.1.114
IPs:
  IP:           10.42.1.114
Controlled By:  StatefulSet/redis-node
Containers:
  redis:
    Container ID:  containerd://d13c0c9f545caf9139f088363b1ac9fb8e11ddc15eb1fc3ee05e8836b0ccf60c
    Image:         docker.io/bitnami/redis:7.4.1-debian-12-r2
    Image ID:      sha256:356e68861bb2255bc69c493e16f9ddb89dad9d58048e619ffa841fcae3434457
    Port:          6379/TCP
    Host Port:     0/TCP
    Command:
      /bin/bash
    Args:
      -c
      /opt/bitnami/scripts/start-scripts/start-node.sh
    State:          Running
      Started:      Fri, 03 Jan 2025 06:20:38 -0500
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:                150m
      ephemeral-storage:  2Gi
      memory:             192Mi
    Requests:
      cpu:                100m
      ephemeral-storage:  50Mi
      memory:             128Mi
    Liveness:             exec [sh -c /health/ping_liveness_local.sh 5] delay=20s timeout=5s period=5s #success=1 #failure=5
    Readiness:            exec [sh -c /health/ping_readiness_local.sh 1] delay=20s timeout=1s period=5s #success=1 #failure=5
    Startup:              exec [sh -c /health/ping_liveness_local.sh 5] delay=10s timeout=5s period=10s #success=1 #failure=22
    Environment:
      BITNAMI_DEBUG:               false
      REDIS_MASTER_PORT_NUMBER:    6379
      ALLOW_EMPTY_PASSWORD:        no
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'redis'>  Optional: false
      REDIS_MASTER_PASSWORD:       <set to the key 'redis-password' in secret 'redis'>  Optional: false
      REDIS_TLS_ENABLED:           no
      REDIS_PORT:                  6379
      REDIS_SENTINEL_TLS_ENABLED:  no
      REDIS_SENTINEL_PORT:         26379
      REDIS_DATA_DIR:              /data
    Mounts:
      /data from redis-data (rw)
      /health from health (rw)
      /opt/bitnami/redis-sentinel/etc from sentinel-data (rw)
      /opt/bitnami/redis/etc from empty-dir (rw,path="app-conf-dir")
      /opt/bitnami/redis/mounted-etc from config (rw)
      /opt/bitnami/scripts/start-scripts from start-scripts (rw)
      /tmp from empty-dir (rw,path="tmp-dir")
  sentinel:
    Container ID:  containerd://7766761172f6f23ff8a154ef51736f28b8ef073c8e0a765d85f6be23b9cd51bf
    Image:         docker.io/bitnami/redis-sentinel:7.4.1-debian-12-r2
    Image ID:      sha256:f95222d30756f138dfa26d760a90101569ba662503fd0d7a0bd234db74f31095
    Port:          26379/TCP
    Host Port:     0/TCP
    Command:
      /bin/bash
    Args:
      -c
      /opt/bitnami/scripts/start-scripts/start-sentinel.sh
    State:          Running
      Started:      Fri, 03 Jan 2025 06:20:38 -0500
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:                150m
      ephemeral-storage:  2Gi
      memory:             192Mi
    Requests:
      cpu:                100m
      ephemeral-storage:  50Mi
      memory:             128Mi
    Liveness:             exec [sh -c /health/ping_sentinel.sh 5] delay=20s timeout=5s period=10s #success=1 #failure=6
    Readiness:            exec [sh -c /health/ping_sentinel.sh 1] delay=20s timeout=1s period=5s #success=1 #failure=6
    Startup:              exec [sh -c /health/ping_sentinel.sh 5] delay=10s timeout=5s period=10s #success=1 #failure=22
    Environment:
      BITNAMI_DEBUG:               false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'redis'>  Optional: false
      REDIS_SENTINEL_TLS_ENABLED:  no
      REDIS_SENTINEL_PORT:         26379
    Mounts:
      /data from redis-data (rw)
      /health from health (rw)
      /opt/bitnami/redis-sentinel/etc from sentinel-data (rw)
      /opt/bitnami/redis-sentinel/mounted-etc from config (rw)
      /opt/bitnami/scripts/start-scripts from start-scripts (rw)
      /tmp from empty-dir (rw,path="tmp-dir")
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  redis-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  redis-data-redis-node-0
    ReadOnly:   false
  start-scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redis-scripts
    Optional:  false
  health:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redis-health
    Optional:  false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redis-configuration
    Optional:  false
  sentinel-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  empty-dir:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      
    SizeLimit:   <unset>
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>
[root@k8s-master01 ~]# 

java连接报错

[root@k8s-master01 ~]# kubectl cp testRedis-1.0-SNAPSHOT-jar-with-dependencies.jar spacex-file-8c8d67b87-ltbwp:/
[root@k8s-master01 ~]# kubectl exec -it spacex-file-8c8d67b87-ltbwp  -- java -jar testRedis-1.0-SNAPSHOT-jar-with-dependencies.jar
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
redis.clients.jedis.exceptions.JedisConnectionException: All sentinels down, cannot determine where is mymaster master is running...
        at redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:269)
        at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:194)
        at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:172)
        at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:153)
        at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:143)
        at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:118)
        at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:106)
        at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:76)
        at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:60)
        at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:70)
        at cn.aerospacex.TestRedis.main(TestRedis.java:20)
[root@k8s-master01 ~]# 

java代码

依赖:

 <!-- Jedis 依赖 -->
        <dependency>
            <groupId>redis.clients</groupId>
            <artifactId>jedis</artifactId>
            <version>4.4.3</version>
            <scope>compile</scope>
        </dependency>

package cn.aerospacex;

import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPoolConfig;
import redis.clients.jedis.JedisSentinelPool;

import java.util.HashSet;
import java.util.Set;

public class TestRedis {
    public static void main(String[] args) {
        System.setProperty("redis.clients.jedis.debug", "true");
        // 配置 Sentinel 地址
        Set<String> sentinels = new HashSet<>();
        sentinels.add("redis-node-0.redis-headless.default.svc.cluster.local:26379");
        sentinels.add("redis-node-1.redis-headless.default.svc.cluster.local:26379");
        sentinels.add("redis-node-2.redis-headless.default.svc.cluster.local:26379");

        // 创建 JedisSentinelPool
        try (JedisSentinelPool pool = new JedisSentinelPool("mymaster", sentinels, new JedisPoolConfig(), "poinuat@beijing")) {
            // 获取 Redis 连接
            try (Jedis jedis = pool.getResource()) {
                jedis.auth("poinuat@beijing");
                jedis.set("key", "value");
                System.out.println("Value: " + jedis.get("key"));
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}