1. Migration tool comparison
- redis-migrate-tool
redis-migrate-tool is a real-time data migration tool between Redis heterogeneous clusters open sourced by Vipshop, but it has not been updated for two years. I personally think this is a relatively complete tool, especially for data school. For detailed function introduction, please refer to GitHub: Reference address
- RedisShake
RedisShake is a secondary development tool developed by Alibaba Cloud based on the open source redis-port of the pea pod to support real-time synchronization of Redis heterogeneous clusters. Compared with redis-migrate-tool, I think its advantage lies in supporting the synchronization of prefix keys. Supports multi-DB synchronization, while redis-migrate-tool can only synchronize in full, and if the source is sub-database, when it is synchronized to the target Redis, it will be synchronized to the db0 database. This is not for the business of the sub-database scenario. Yes, for the detailed function introduction of RedisShake, please refer to GitHub: Reference address
- redis-port
redis-port is a Redis data migration tool used by pea pods to make it easier for everyone to migrate from redis to Codis. It has not been updated for a long time. For its functions, please refer to GitHub: reference address
2. Migration tool installation and configuration
- Installation of RedisShake
Download the binary package directly to install:
Configuration file modification
redis-shake.conf configuration file:
# 当前配置文件的版本号，请不要修改该值。 conf.version = 1 # ------------------------------------------------------ # id id = redis-shake # log file，日志文件，不配置将打印到stdout (e.g. /var/log/redis-shake.log ) log.file = # log level: "none", "error", "warn", "info", "debug". default is "info". log.level = info # pid path，进程文件存储地址（e.g. /var/run/)，不配置将默认输出到执行下面, # 注意这个是目录，真正的pid是`{pid_path}/{id}.pid` pid_path = # pprof port. system_profile = 9310 # restful port, set -1 means disable, in `restore` mode RedisShake will exit once finish restoring RDB only if this value # is -1, otherwise, it'll wait forever. # restful port，查看metric端口, -1表示不启用，如果是`restore`模式，只有设置为-1才会在完成RDB恢复后退出，否则会一直block。 http_profile = 9320 # parallel routines number used in RDB file syncing. default is 64. # 启动多少个并发线程同步一个RDB文件。 parallel = 32 # 源端redis的类型，支持standalone，sentinel，cluster和proxy四种模式，注意：目前proxy只用于rump模式。 source.type = sentinel # 源redis地址。对于sentinel或者开源cluster模式，输入格式为"master名字:拉取角色为master或者slave@sentinel的地址"，别的cluster # 架构，比如codis, twemproxy, aliyun proxy等需要配置所有master或者slave的db地址。 source.address = mymaster:master@10.10.20.101:26379;10.10.20.102:26379;10.10.20.103:26379 # password of db/proxy. even if type is sentinel. source.password_raw = # auth type, don't modify it source.auth_type = auth # tls enable, true or false. Currently, only support standalone. # open source redis does NOT support tls so far, but some cloud versions do. source.tls_enable = false # input RDB file. # used in `decode` and `restore`. # if the input is list split by semicolon(;), redis-shake will restore the list one by one. # 如果是decode或者restore，这个参数表示读取的rdb文件。支持输入列表，例如：rdb.0;rdb.1;rdb.2 # redis-shake将会挨个进行恢复。 source.rdb.input = local # 假如db节点/输入的rdb有5个，但rdb.parallel=3，那么一次只会 # 并发拉取3个db的全量数据，直到某个db的rdb拉取完毕并进入增量，才会拉取第4个db节点的rdb， # 以此类推，最后会有len(source.address)或者len(rdb.input)个增量线程同时存在。 source.rdb.parallel = 0 # for special cloud vendor: ucloud # used in `decode` and `restore`. # ucloud集群版的rdb文件添加了slot前缀，进行特判剥离: ucloud_cluster。 source.rdb.special_cloud = # 目的redis的类型，支持standalone，sentinel，cluster和proxy四种模式。 target.type = cluster target.address = 10.10.20.101:7000;10.10.20.102:7000;10.10.20.103:7000 # password of db/proxy. even if type is sentinel. target.password_raw = Redis1234{} # auth type, don't modify it target.auth_type = auth # all the data will be written into this db. < 0 means disable. target.db = 0 # tls enable, true or false. Currently, only support standalone. # open source redis does NOT support tls so far, but some cloud versions do. target.tls_enable = false # output RDB file prefix. # used in `decode` and `dump`. # 如果是decode或者dump，这个参数表示输出的rdb前缀，比如输入有3个db，那么dump分别是: # ${output_rdb}.0, ${output_rdb}.1, ${output_rdb}.2 target.rdb.output = local_dump # e.g., target.version = 4.0 target.version = # 用于处理过期的键值，当迁移两端不一致的时候，目的端需要加上这个值 fake_time = # 当源目的有重复key，是否进行覆写 # rewrite表示源端覆盖目的端。 # none表示一旦发生进程直接退出。 # ignore表示保留目的端key，忽略源端的同步key。该值在rump模式下没有用。 key_exists = rewrite # 指定的db被通过，比如0;5;10将会使db0, db5, db10通过, 其他的被过滤 filter.db.whitelist = # 指定的db被过滤，比如0;5;10将会使db0, db5, db10过滤，其他的被通过 filter.db.blacklist = # 支持按前缀过滤key，只让指定前缀的key通过，分号分隔。比如指定abc，将会通过abc, abc1, abcxxx filter.key.whitelist = # 支持按前缀过滤key，不让指定前缀的key通过，分号分隔。比如指定abc，将会阻塞abc, abc1, abcxxx filter.key.blacklist = # filter given slot, multiple slots are separated by ';'. # e.g., 1;2;3 # used in `sync`. # 指定过滤slot，只让指定的slot通过 filter.slot = # filter lua script. true means not pass. However, in redis 5.0, the lua # converts to transaction(multi+{commands}+exec) which will be passed. # 控制不让lua脚本通过，true表示不通过 filter.lua = false # 正常key如果不大，那么都是直接调用restore写入到目的端，如果key对应的value字节超过了给定 # 的值，那么会分批依次一个一个写入。如果目的端是Codis，这个需要置为1，具体原因请查看FAQ。 # 如果目的端大版本小于源端，也建议设置为1。 big_key_threshold = 524288000 # enable metric # used in `sync`. # 是否启用metric metric = true # print in log # 是否将metric打印到log中 metric.print_log = false # sender information. # sender flush buffer size of byte. # used in `sync`. # 发送缓存的字节长度，超过这个阈值将会强行刷缓存发送 sender.size = 104857600 # sender flush buffer size of oplog number. # used in `sync`. flush sender buffer when bigger than this threshold. # 发送缓存的报文个数，超过这个阈值将会强行刷缓存发送，对于目的端是cluster的情况，这个值 # 的调大将会占用部分内存。 sender.count = 4095 # 用于metric统计时延的队列 sender.delay_channel_size = 65535 # enable keep_alive option in TCP when connecting redis. # the unit is second. # 0 means disable. # TCP keep-alive保活参数，单位秒，0表示不启用。 keep_alive = 0 # 每次scan的个数，不配置则默认100. scan.key_number = 50 # 有些版本具有特殊的格式，与普通的scan命令有所不同，我们进行了特殊的适配。目前支持腾讯云的集群版"tencent_cluster" # 和阿里云的集群版"aliyun_cluster"，注释主从版不需要配置，只针对集群版。 scan.special_cloud = # used in `rump`. # we support to fetching data from given file which marks the key list. # 有些云版本，既不支持sync/psync，也不支持scan，我们支持从文件中进行读取所有key列表并进行抓取：一行一个key。 scan.key_file = # limit the rate of transmission. Only used in `rump` currently. # e.g., qps = 1000 means pass 1000 keys per second. default is 500,000(0 means default) qps = 200000 # enable resume from break point, please visit xxx to see more details. # 断点续传开关 resume_from_break_point = false # replace hash tag. # used in `sync`. replace_hash_tag = falseDescription of main configuration items:
# 源Redis的类型，哨兵模式采用sentinel source.type = sentinel # 源Redis的地址信息，如果是哨兵， 格式：哨兵名称:主从标识@节点地址信息 source.address = mymaster:master@10.10.20.101:26379;10.10.20.102:26379;10.10.20.103:26379 # 源Redis的密码信息，没有密码认证，留空即可 source.password_raw = # 源Redis的认证标识，不管有无密码认证， 都不用修改 source.auth_type = auth # 目标Redis的类型，这里所填的目标是集群 target.type = cluster # 目标Redis的地址信息，多个地址以;分号分隔，注意这里要填的统一为主节点或从节点信息 target.address = 10.10.20.101:7000;10.10.20.102:7000;10.10.20.103:7000 # 目标Redis的密码信息 target.password_raw = Redis1234{} # 目标Redis的认证标识， 不用修改 target.auth_type = auth # 目标Redis的存储数据库，如果为负数，代表不启用该功能，如果指定某个库，会将源数据全部写入 target.db = 0 # 当存在相同的keys，处理方案： # rewrite表示源端覆盖目的端。 # none表示一旦发生进程直接退出。 # ignore表示保留目的端key，忽略源端的同步key。该值在rump模式下没有用。 key_exists = rewrite
3. Data synchronization
Data synchronization command
./redis-shake -conf=redis-shake.conf -type=syncThe type can support sync, restore, dump, decode, rump. Select sync for full + incremental synchronization
Log result information
The synchronization is divided into three stages:
A. Wait for the source end save rdb to complete, the log is as follows
2019/06/06 15:14:56 [INFO] dbSyncer[0] + waiting source rdb 2019/06/06 15:14:57 [INFO] dbSyncer[0] - waiting source rdb 2019/06/06 15:14:57 [INFO] dbSyncer[0] + waiting source rdbB. During the full synchronization phase, the percentage is displayed:
2019/06/06 15:15:41 [INFO] dbSyncer[0] total=924836132 - 9155943 [ 0%] entry=2109 2019/06/06 15:15:42 [INFO] dbSyncer[0] total=924836132 - 16107663 [ 1%] entry=4411 2019/06/06 15:15:43 [INFO] dbSyncer[0] total=924836132 - 22914262 [ 2%] entry=6750 2019/06/06 15:15:44 [INFO] dbSyncer[0] total=924836132 - 29707595 [ 3%] entry=9060 2019/06/06 15:15:45 [INFO] dbSyncer[0] total=924836132 - 35741354 [ 3%] entry=11067 2019/06/06 15:15:46 [INFO] dbSyncer[0] total=924836132 - 42911547 [ 4%] entry=13480C. Incremental synchronization, after the words
sync rdb doneappear, the current dbSyncer enters incremental synchronization:2019/07/09 16:34:05 [INFO] dbSyncer[0] sync: +forwardCommands=1 +filterCommands=0 +writeBytes=4 2019/07/09 16:34:06 [INFO] dbSyncer[0] sync: +forwardCommands=0 +filterCommands=0 +writeBytes=0 2019/07/09 16:34:07 [INFO] dbSyncer[0] sync: +forwardCommands=0 +filterCommands=0 +writeBytes=0 2019/07/09 16:34:08 [INFO] dbSyncer[0] sync: +forwardCommands=0 +filterCommands=0 +writeBytes=0 2019/07/09 16:34:09 [INFO] dbSyncer[0] sync: +forwardCommands=0 +filterCommands=0 +writeBytes=0 2019/07/09 16:34:10 [INFO] dbSyncer[0] sync: +forwardCommands=0 +filterCommands=0 +writeBytes=0 2019/07/09 16:34:11 [INFO] dbSyncer[0] sync: +forwardCommands=0 +filterCommands=0 +writeBytes=0Where forwardCommands represents the number of commands sent, filterCommands represents the number of filtered commands, such as opinfo or specified filter will be filtered, and writeBytes represents the number of bytes sent.
4. Data verification processing
- Data verification tool
redis-full-check performs data verification by fully comparing the data in the source and destination redis.
Verify execution command
./redis-full-check -s '10.10.20.102:6379' -p '' -t '10.10.20.101:7000;10.10.20.102:7000;10.10.20.103:7000' -a 'Redis1234{}' --targetdbtype=1 --metric --result=result.1Configure the synchronization source address and target address information, turn on the metric, the analysis information will be output, and the result will summarize the verification results.
Detailed parameter description:
-s, --source=SOURCE 源redis库地址（ip:port），如果是集群版，那么需要以分号（;）分割不同的db，只需要配置主或者从的其中之一。例如：10.1.1.1:1000;10.2.2.2:2000;10.3.3.3:3000。 -p, --sourcepassword=Password 源redis库密码 --sourceauthtype=AUTH-TYPE 源库管理权限，开源reids下此参数无用。 --sourcedbtype= 源库的类别，0：db(standalone单节点、主从)，1: cluster（集群版），2: 阿里云 --sourcedbfilterlist= 源库需要抓取的逻辑db白名单，以分号（;）分割，例如：0;5;15表示db0,db5和db15都会被抓取 -t, --target=TARGET 目的redis库地址（ip:port） -a, --targetpassword=Password 目的redis库密码 --targetauthtype=AUTH-TYPE 目的库管理权限，开源reids下此参数无用。 --targetdbtype= 参考sourcedbtype --targetdbfilterlist= 参考sourcedbfilterlist -d, --db=Sqlite3-DB-FILE 对于差异的key存储的sqlite3 db的位置，默认result.db --comparetimes=COUNT 比较轮数 -m, --comparemode= 比较模式，1表示全量比较，2表示只对比value的长度，3只对比key是否存在，4全量比较的情况下，忽略大key的比较 --id= 用于打metric --jobid= 用于打metric --taskid= 用于打metric -q, --qps= qps限速阈值 --interval=Second 每轮之间的时间间隔 --batchcount=COUNT 批量聚合的数量 --parallel=COUNT 比较的并发协程数，默认5 --log=FILE log文件 --result=FILE 不一致结果记录到result文件中，格式：'db diff-type key field' --metric=FILE metric文件 --bigkeythreshold=COUNT 大key拆分的阈值，用于comparemode=4 -f, --filterlist=FILTER 需要比较的key列表，以分号（;）分割。例如："abc*|efg|m*"表示对比'abc', 'abc1', 'efg', 'm', 'mxyz'，不对比'efgh', 'p'。 -v, --versionView the comparison result
View result information:
[root@localhost redis-full-check-1.4.8]# less result.1Verification result:
0 lack_target single23 1 lack_target test 9 value single2The first column represents the database number, the second column represents the conflict type, and the third column represents the conflict key.
Conflict types include:
lack_soure: The field exists in the source key, and the field does not exist with the destination key.lack_target: The field does not exist with the source key, and the field exists at the destination key.value: The field exists in the source key and the destination key, but the value corresponding to the field is different.To view more specific information, you can query through sqlite:
[root@localhost redis-full-check-1.4.8]# sqlite3 result.db.3 SQLite version 3.7.17 2013-05-20 00:56:22 Enter ".help" for instructions Enter SQL statements terminated with a ";" sqlite> .tables FINAL_RESULT field key sqlite> select * from key; 1|single23|string|lack_target|0|2|0 2|test|string|lack_target|1|4|0 3|single2|string|value|9|1|9Information column correspondence: id, key, type, conflict_type, db, source_len, target_len
- *
This article was created and shared by mirson. If you need further communication, please join the QQ group: 19310171 or visit www.softart.cn
java
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用。你还可以使用@来通知其他用户。