Technology Sharing | A case of PITR recovery based on mongo cluster

Author: Ren Kun

Now living in Zhuhai, he has served as a full-time Oracle and MySQL DBA. Now he is mainly responsible for the maintenance of MySQL, mongoDB and Redis.

Source of this article: original submission

* Produced by the Aikesheng open source community, original content is not allowed to be used without authorization, please contact the editor and indicate the source for reprinting.

---

1 background

There are many cases on the Internet based on mongo single-instance PITR, and official documents also have recovery steps for mongo cluser, but there are almost no cases of PITR based on mongo cluster. Based on the experimental environment, this article simulates the online environment to complete a PITR for the mongo cluster.

Distribution of original cluster instances:

172.16.129.170 shard1 27017 shard2 27018 config 37017 mongos 47017
172.16.129.171 shard1 27017 shard2 27018 config 37017 mongos 47017
172.16.129.172 shard1 27017 shard2 27018 config 37017 mongos 47017

Considering that the actual data in the online environment will be relatively large, here we only restore the shard to a single instance, which can be used for development and query data. Instance distribution after recovery:

172.16.129.173 shard1 27017 shard2 27018 config 37017 mongos 47017
172.16.129.174 config 37017
172.16.129.175 config 37017

The mongo version is percona 4.2.13. A hot backup regular backup script is deployed for each shard instance and config server, and an oplog regular backup script is deployed for each shard instance and config server. Build test data, log in to the cluster through mongos, create a hash fragment table and insert 10 pieces of data:

use admin
db.runCommand({"enablesharding":"renkun"})
sh.shardCollection("renkun.user", { id: "hashed" } )
use renkun
var tmp = [];
for(var i =0; i<10; i++){
tmp.push({ 'id':i, "name":"Kun " + i});
}
db.user.insertMany(tmp);

Perform physical hot backup on shard1, shard2 and config server, and continue to insert data:

use renkun
var tmp = [];
for(var i =10; i<20; i++){
tmp.push({ 'id':i, "name":"Kun " + i});
}
db.user.insertMany(tmp);

Perform oplog backup on shard, shard2 and config server. All the above operations are included in the oplog backup file. At this time, the user table has 20 pieces of data, and the id is incremented from 0-20. Our requirement is to restore the cluster to the snapshot point at max(id)=15.

The steps given by the official file are to restore config server, shard and mongo in turn, but there is no operation to roll forward oplog on the official file.

In our case, we need to parse out the time point to be restored from the oplog file of the shard, so adjust the order, first restore the shard and then restore the config server.

2 Recover shard single instance

Transfer the shard's physical backup and oplog backup to the target machine, and directly unzip the physical backup to the data directory to start.

2.1 Confirm the snapshot point to be restored

Perform the following operations on the oplog backups of the two shards

bsondump oplog.rs.bson > oplog.rs.json
more oplog.rs.json | egrep "\"op\"\:\"\i\",\"ns\":\"renkun\.user\"" | grep "\"Kun
15\""

Query on the oplog backup of shard2

{"ts":{"$timestamp":{"t":1623408268,"i":6}},"t":{"$numberLong":"1"},"h":
{"$numberLong":"0"},"v":{"$numberInt":"2"},"op":"i","ns":"renkun.user","ui":{"$binary":
{"base64":"uhlG0e4sRB+RUfFOzpMCEQ==","subType":"04"}},"wall":{"$date":
{"$numberLong":"1623408268694"}},"o":{"_id":{"$oid":"60c33e8c1c3edd59f25eecb5"},"id":
{"$numberDouble":"15.0"},"name":"Kun 15"}}

It is confirmed that the corresponding timestamp when inserting id=15 is 1623408268:6, but the timestamp parameter specified by the --oplogLimit of mongorestore is an open interval, that is, the oplog entry corresponding to the specified timestamp will not be replayed, so if you want to restore to At this point in time, one more bit must be added to it, so it becomes 1623408268:7.

2.2 Create a temporary account

Log in to the shard instance with the root account, create a user with the role of __system, and name it internal_restore. This user is not only used to roll forward oplog, but also to modify admin.system.version and delete local db. The root account does not have permission to perform these operations by default.

use admin
db.createUser( { user: "internal_restore", pwd: "internal_restore", roles: [
"__system" ] })

Use internal_restore to log in to two shard instances and delete local db.

use local
db.dropDatabase()

2.3 Roll forward oplog to a specified point in time

mongorestore ‐h 127.0.0.1 ‐uinternal_restore ‐p"internal_restore" ‐‐port 27017 ‐‐
oplogReplay ‐‐oplogLimit "1623408268:7" ‐‐authenticationDatabase admin
/data/backup/202106111849_27017/local/oplog.rs.bson
mongorestore ‐h 127.0.0.1 ‐uinternal_restore ‐p"internal_restore" ‐‐port 27018 ‐‐
oplogReplay ‐‐oplogLimit "1623408268:7" ‐‐authenticationDatabase admin
/data/backup/202106111850_27018/local/oplog.rs.bson

Log in to shard1 and shard2 to view the data:

‐‐shard1 27017
> db.user.find().sort({"id":1})
{ "_id" : ObjectId("60c330322424943565780766"), "id" : 3, "name" : "Kun 3" }
{ "_id" : ObjectId("60c330322424943565780769"), "id" : 6, "name" : "Kun 6" }
{ "_id" : ObjectId("60c33032242494356578076b"), "id" : 8, "name" : "Kun 8" }
{ "_id" : ObjectId("60c33e8c1c3edd59f25eecb1"), "id" : 11, "name" : "Kun 11" }
{ "_id" : ObjectId("60c33e8c1c3edd59f25eecb2"), "id" : 12, "name" : "Kun 12" }
‐‐shard2 27018
> db.user.find().sort({"id":1})
{ "_id" : ObjectId("60c330322424943565780763"), "id" : 0, "name" : "Kun 0" }
{ "_id" : ObjectId("60c330322424943565780764"), "id" : 1, "name" : "Kun 1" }
{ "_id" : ObjectId("60c330322424943565780765"), "id" : 2, "name" : "Kun 2" }
{ "_id" : ObjectId("60c330322424943565780767"), "id" : 4, "name" : "Kun 4" }
{ "_id" : ObjectId("60c330322424943565780768"), "id" : 5, "name" : "Kun 5" }
{ "_id" : ObjectId("60c33032242494356578076a"), "id" : 7, "name" : "Kun 7" }
{ "_id" : ObjectId("60c33032242494356578076c"), "id" : 9, "name" : "Kun 9" }
{ "_id" : ObjectId("60c33e8c1c3edd59f25eecb0"), "id" : 10, "name" : "Kun 10" }
{ "_id" : ObjectId("60c33e8c1c3edd59f25eecb3"), "id" : 13, "name" : "Kun 13" }
{ "_id" : ObjectId("60c33e8c1c3edd59f25eecb4"), "id" : 14, "name" : "Kun 14" }
{ "_id" : ObjectId("60c33e8c1c3edd59f25eecb5"), "id" : 15, "name" : "Kun 15" }

The data has been restored, a total of 16 items, max(id)=15

2.4 Modify admin.system.version

The shard has changed from the original 3-node replication set to a single instance, and the host ip has also changed, so the corresponding metadata information must be modified. Log in to the two shard instances as the internal_restore user and execute respectively:

use admin
db.system.version.deleteOne( { _id: "minOpTimeRecovery" } )
db.system.version.find( {"_id" : "shardIdentity" } )
db.system.version.updateOne(
 { "_id" : "shardIdentity" },
{ $set :
{ "configsvrConnectionString" :
"configdb/172.16.129.173:37017,172.16.129.174:37017,172.16.129.175:37017"}
}
)

Before being deleted and modified, the two records stored the old config server information respectively. The details are as follows:

{ "_id" : "shardIdentity", "shardName" : "repl", "clusterId" :
ObjectId("60c2c9b44497a4f2e02510fd"), "configsvrConnectionString" :
"configdb/172.16.129.170:37017,172.16.129.171:37017,172.16.129.172:37017" }
{ "_id" : "minOpTimeRecovery", "configsvrConnectionString" :
"configdb/172.16.129.170:37017,172.16.129.171:37017,172.16.129.172:37017", "minOpTime" :
{ "ts" : Timestamp(1623383008, 6), "t" : NumberLong(1) }, "minOpTimeUpdaters" : 0,
"shardName" : "repl" }

So far, the two shard servers have been restored. The next step is to restore the config server.

3 Restore config server

Transfer the physical backup to the target machine and unzip it directly to the data directory to use. First start the config server as a single instance, after logging in with the root account, create a user with the __system role (same as above).

3.1 Roll forward oplog

mongorestore ‐h 127.0.0.1 ‐uinternal_restore ‐p"internal_restore" ‐‐port 37017 ‐‐
oplogReplay ‐‐oplogLimit "1623408268:7" ‐‐authenticationDatabase admin
/data/backup/202106111850_37017/local/oplog.rs.bson

3.2 Modify metadata

Log in to the instance as interal_restore and modify the shard metadata information of the cluster. The specific operations are as follows:

use local
db.dropDatabase()
use config
db.shards.find()
db.shards.updateOne({ "_id" : "repl" }, { $set : { "host" : "172.16.129.173:27017" } })
db.shards.updateOne({ "_id" : "repl2" }, { $set : { "host" : "172.16.129.173:27018" } })
db.shards.find()

3.3 Start the cluster

Close the config server, start in cluster mode, and enable the following parameters in the corresponding configuration file:

sharding:
clusterRole: configsvr
replication:
oplogSizeMB: 10240
replSetName: configdb

Execute after login

rs.initiate()
rs.add("172.16.129.174:37017")
rs.add("172.16.129.175:37017")

At this time, the config server becomes a 3-node cluster, and the recovery is complete.

4 configure mongos

You can directly copy the mongos configuration file of the original environment, just modify the sharding and bindIp parameters.

sharding:
configDB: "configdb/172.16.129.173:37017,172.16.129.174:37017,172.16.129.175:37017"
net:
port: 47017
bindIp: 127.0.0.1,172.16.129.173

After startup, log in to mongos to query the user table, a total of 16 records, max(id)=15, which meets the expected result.

mongos> use renkun
switched to db renkun
mongos> db.user.find().sort({"id":1})
{ "_id" : ObjectId("60c330322424943565780763"), "id" : 0, "name" : "Kun 0" }
{ "_id" : ObjectId("60c330322424943565780764"), "id" : 1, "name" : "Kun 1" }
{ "_id" : ObjectId("60c330322424943565780765"), "id" : 2, "name" : "Kun 2" }
{ "_id" : ObjectId("60c330322424943565780766"), "id" : 3, "name" : "Kun 3" }
{ "_id" : ObjectId("60c330322424943565780767"), "id" : 4, "name" : "Kun 4" }
{ "_id" : ObjectId("60c330322424943565780768"), "id" : 5, "name" : "Kun 5" }
{ "_id" : ObjectId("60c330322424943565780769"), "id" : 6, "name" : "Kun 6" }
{ "_id" : ObjectId("60c33032242494356578076a"), "id" : 7, "name" : "Kun 7" }
{ "_id" : ObjectId("60c33032242494356578076b"), "id" : 8, "name" : "Kun 8" }
{ "_id" : ObjectId("60c33032242494356578076c"), "id" : 9, "name" : "Kun 9" }
{ "_id" : ObjectId("60c33e8c1c3edd59f25eecb0"), "id" : 10, "name" : "Kun 10" }
{ "_id" : ObjectId("60c33e8c1c3edd59f25eecb1"), "id" : 11, "name" : "Kun 11" }
{ "_id" : ObjectId("60c33e8c1c3edd59f25eecb2"), "id" : 12, "name" : "Kun 12" }
{ "_id" : ObjectId("60c33e8c1c3edd59f25eecb3"), "id" : 13, "name" : "Kun 13" }
{ "_id" : ObjectId("60c33e8c1c3edd59f25eecb4"), "id" : 14, "name" : "Kun 14" }
{ "_id" : ObjectId("60c33e8c1c3edd59f25eecb5"), "id" : 15, "name" : "Kun 15" }

At this point, a complete mongo cluster pitr operation is officially completed.

5 Summary

Mongo 4.X began to introduce transactions, especially 4.2 that supports cross-shard transactions. According to the official documents, you must use designated tools to restore the data involved in the transaction. The above operations are temporarily not applicable.