利用 ChangeStream 实现 Amazon DocumentDB 表级别容灾复制

前言
与 MongoDB 兼容的Amazon DocumentDB，使用完全托管式文档数据库服务轻松扩展 JSON 工作负载，通过独立扩展计算和存储，支持每秒数以百万计文档的读取请求；自动化硬件预置、修补、设置和其他数据库管理任务；通过自动复制、连续备份和严格的网络隔离实现 99.999999999% 的持久性；将现有 MongoDB 驱动程序和工具与 Apache 2.0 开源 MongoDB 3.6 和 4.0 API 搭配使用。鉴于上述性能优势，越来越多的企业已经或即将使用 DocumentDB 来管理JSON文档数据库。

亚马逊云科技开发者社区为开发者们提供全球的开发技术资源。这里有技术文档、开发案例、技术专栏、培训视频、活动与竞赛等。帮助中国开发者对接世界最前沿技术，观点，和项目，并将中国优秀开发者或技术推荐给全球云社区。如果你还没有关注/收藏，看到这里请一定不要匆匆划过，点这里让它成为你的技术宝库！

对很多行业而言，需要保证数据与业务的持续性，存在关键业务与数据的容灾诉求。亚马逊云科技于2021年6月推出了面向 Amazon DocumentDB（兼容 MongoDB）的全局集群（Global Cluster）。全局集群是一项新功能，可在发生区域范围的中断时提供灾难恢复，同时通过允许从最近的 Amazon DocumentDB 集群读取来实现低延迟全局读取。客户可以将业务发生 Region 内的 DocumentDB 通过该功能同步至其他Region，轻松实现数据层的跨区域容灾。但由于 Global Cluster 全局集群功能是基于存储的快速复制，所以很遗憾，截止本文发稿时，DocumentDB Global Cluster全局集群仅支持实例级别的数据同步与复制，暂不支持 Database 或者 Collection 级别的数据容灾。

亚马逊云科技还有另一款数据库产品 Amazon Data Migration Server（DMS），可以实现 Database 或者 Collection 级别的数据同步，以低延迟与较低的 RPO 指标实现数据的跨区域同步与复制，以实现容灾的需求。但在面对容灾场景中的数据保护诉求，DMS 暂不支持对删除类型的操作进行过滤。

在本文中，我们将向您介绍使用 Amazon Managed Streaming for Apache Kafka（MSK）作为消息中间件暂存 DocumentDB 的改变流事件Change Stream Events，来实现跨 Region 的数据库同步，并拦截删除类型的操作的整体解决方案。本例中，我们采用 us-east-1 弗吉尼亚北部区域作为主区域 Primary Region，已有 DocumentDB 主实例，us-west-2俄勒冈区域作为灾备区域 DR Region，已有 DocumentDB 灾备实例，使用了 python 作为编程语言，除 python 外您还可以使用其他主流编程语言譬如 Java，Node.JS 实现业务逻辑，但由于驱动原因，暂不支持 Ruby；另外请使用 Amazon DocumentDB v4.0 以上版本。参考架构图如下图所示：

主 region 的 stream-capture 主机环境设置
1.在主 region 的 stream-capture 主机上设置OS参数环境 Code部分：

设置环境变量，请替换红色的文字部分为您实际的值，本文中默认采用 bar.foo 为改变流监控 collection，您可以替换为您自定义的其他 DB 与 collection

##设置环境变量，请替换红色的文字部分为您实际的值，本文中默认采用 bar.foo 为改变流监控 collection，您可以替换为您自定义的其他DB 与 collection
echo -e "USERNAME="Your Primary MongoDB User"\nexport USERNAME\nPASSWORD="Your Primary MongoDB password"\nexport PASSWORD\nmongo_host="Primary MongoDB Cluster URI"\nexport mongo_host\nstate_tbl="YOUR STATE COLLECTION"\nexport state_tbl\nstate_db="YOUR STATE DB"\nexport state_db\nwatched_db_name="bar"\nexport watched_db_name\nwatched_tbl_name="foo"\nexport watched_tbl_name\nevents_remain=1\nexport events_remain\nDocuments_per_run=100000\nexport Documents_per_run\nkfk_host="YOUR KFK URI\nexport kfk_host\nkfk_topic="changevents"\nexport kfk_topic\nbucket_name="YOUR S3 BUCKET"\nexport bucket_name\nS3_prefix=""\nexport S3_prefix"" >> .bash_profile
##应用环境变量
source .bash_profile

2. 在主 region 的 stream-capture 主机上安装 pymongo 与 boto3 请参考如何在 Amazon Linux 2 上使用 Boto 3 库创建 Python 3 虚拟环境

完成 python3 与 boto3 的安装与配置，本文不再复述

##安装 pymongo
sudo pip install pymongo

3. 在主 region 的 stream-capture 主机上安装 MongoDB 客户端与证书

##下载 SSL 证书到 /tmp 下
wget -P /tmp https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem

##配置 MongoDB 的 YUM REPO
sudo echo -e "[mongodb-org-5.0]\nname=MongoDB Repository\nbaseurl=https://repo.mongodb.org/yum/amazon/2/mongodb-org/5.0/x86_64/\ngpgcheck=1\nenabled=\ngpgkey=https://www.mongodb.org/static/pgp/server-5.0.asc" >> /etc/yum.repos.d/mongodb-org-5.0.repo
##安装M ongoDB 客户端
sudo yum install -y mongodb-org-shell

创建 MSK 的 Topic 用以接受改变流事件
请参照本文档【开始使用MSK第3步：创建主题】来创建 MSK 的 topic，本文不再复述。请将步骤12中的–topic MSKTutorialTopic替换–topic changevents 之后，执行第步骤12

我们将可以看到如下消息：

Created topic changevents.

启用Amazon DocumentDB改变流
1.使用 mongosh 客户端登陆主 DocumentDB 集群

Mongo --host $mongo_host:27017 --ssl --sslCAFile
/tmp/rds-combined-ca-bundle.pem --username $USERNAME --password $PASSWORD

2.对 bar.foo 启用改变流

db.adminCommand({modifyChangeStreams: 1,database: "bar",collection: "foo", enable: true});

3.确认成功
{ "ok" : 1 }

主 region 的改变流捕获程序

#!/bin/env python

import json
import logging
import os
import time
import boto3
import datetime
from pymongo import MongoClient
from pymongo.errors import OperationFailure
from kafka import KafkaProducer

db_client = None
kafka_client = None                                           
s3_client = None        
                                 
logging.basicConfig(Level=logging.ERROR)

# The error code returned when data for the requested resume token has been deleted
err_code_136 = 136


def get_db_client():

    # Use a global variable if CX has interest in Lambda function instead of long-running python
    global db_client

    if db_client is None:
        logging.debug('Creating a new DB client.')

        try:

            username = os.environ[‘USERNAME’]
            password = os.environ[‘PASSWORD’]
            cluster_uri = os.environ['mongo_host'] 
            db_client = MongoClient(cluster_uri, ssl=True, retryWrites=False, ssl_ca_certs='/tmp/rds-combined-ca-bundle.pem')
            # Make an attemp for connecting
            db_client.admin.command('ismaster')
            db_client["admin"].authenticate(name=username, password=password)
            logging.debug('Successfully created a new DB client.')
        except Exception as err:
            logging.error('Failed to create a new DB client: {}'.format(err))
            raise

    return db_client


def get_state_tbl_client():

    """Return a DocumentDB client for the collection in which we store processing state."""

        try:

            db_client = get_db_client()
            state_db_name = os.environ['state_db']
            state_tbl_name = os.environ['state_tbl']
            state_tbl = db_client[state_db_name][state_tbl_name]
        except Exception as err:
            logging.error('Failed to create new state collection client: {}'.format(err))
            raise

    return state_tbl


def get_last_position():

            last_position = None
            logging.debug('Locate the last position.’)
        try:

            state_tbl = get_state_tbl_client()
            if "watched_tbl_name" in os.environ:
            position_point = state_tbl.find_one({'currentposition': True, 'watched_db': str(os.environ['watched_db_name']), 
                'watched_tbl': str(os.environ['watched_tbl_name']), 'db_level': False})
            else:
            position_point = state_tbl.find_one({'currentposition': True, 'db_level': True, 
                'watched_db': str(os.environ['watched_db_name'])})
           
            if position_point is not None:
            if 'lastProcessed' in position_point: 
                last_position = position_point['lastProcessed']
            else:
            if "watched_tbl_name" in os.environ:
                state_tbl.insert_one({'watched_db': str(os.environ['watched_db_name']),
                    'watched_tbl': str(os.environ['watched_tbl_name']), 'currentposition': True, 'db_level': False})
            else:
                state_tbl.insert_one({'watched_db': str(os.environ['watched_db_name']), 'currentposition': True, 
                    'db_level': True})

        except Exception as err:
            logging.error('Failed to locate the last processed id: {}'.format(err))
            raise

    return last_position


def save_last_position(resume_token):

            """Save the resume token by the last successfully processed change event."""

            logging.debug('Saving last processed id.')
        try:

            state_tbl = get_state_tbl_client()
            if "watched_tbl_name" in os.environ:
            state_tbl.update_one({'watched_db': str(os.environ['watched_db_name']), 
                'watched_tbl': str(os.environ['watched_tbl_name'])},{'$set': {'lastProcessed': resume_token}})
            else:
            state_tbl.update_one({'watched_db': str(os.environ['watched_db_name']), 'db_level': True, },
                {'$set': {'lastProcessed': resume_token}})

        except Exception as err:
            logging.error('Failed to save last processed id: {}'.format(err))
            raise


def conn_kfk_producer():

            # Use a global variable if CX has interest in Lambda function instead of long-running python
            global kafka_client
    
            if kafka_client is None:
            logging.debug('Creating a new Kafka client.')

        try:

            kafka_client = KafkaProducer(bootstrap_servers=os.environ['kfk_host'])
        except Exception as err:
            logging.error('Failed to create a new Kafka client: {}'.format(err))
            raise
    
    return kafka_client


def produce_msg(producer_instance, topic_name, key, value):

    """Produce change events to MSK."""
    
        try:

            topic_name = os.environ['kfk_topic']
            producer_instance = KafkaProducer(key_serializer=lambda key: json.dumps(key).encode('utf-8’),value_serializer=lambda value: json.dumps(value).encode('utf-8’),retries=3)
            producer_instance.send(topic_name, key, value)
            producer_instance.flush()
        except Exception as err:
            logging.error('Error in publishing message: {}'.format(err))
            raise


def write_S3(event, database, collection, doc_id):

            global s3_client

            if s3_client is None:
            s3_client = boto3.resource('s3')  

        try:
            logging.debug('Publishing message to S3.') #, str(os.environ['S3_prefix'])
            if "S3_prefix" in os.environ:
            s3_client.Object(os.environ['bucket_name'], str(os.environ['S3_prefix']) + '/' + database + '/' +
                collection + '/' + datetime.datetime.now().strftime('%Y/%m/%d/') + doc_id).put(Body=event)
            else: 
            s3_client.Object(os.environ['bucket_name'], database + '/' + collection + '/' + 
                datetime.datetime.now().strftime('%Y/%m/%d/') + doc_id).put(Body=event)

        except Exception as err:
            logging.error('Error in publishing message to S3: {}'.format(err))
            raise

def main(event, context):
    """Read change events from DocumentDB and push them to MSK&S3."""
    
            events_processed = 0
            watcher = None
            kafka_client = None

        try:
        
            # Kafka client set up    
            if "kfk_host" in os.environ:
            kafka_client = conn_kfk_producer()  
            logging.debug('Kafka client set up.')    

            # DocumentDB watched collection set up
            db_client = get_db_client()
            watched_db = os.environ['watched_db_name']
            if "watched_tbl_name" in os.environ:
            watched_tbl = os.environ['watched_tbl_name']
            watcher = db_client[watched_db][watched_tbl]
            else: 
            watcher = db_client[watched_db]
            logging.debug('Watching table {}'.format(watcher))

            # DocumentDB sync set up
            state_sync_count = int(os.environ['events_remain'])
            last_position = get_last_position()
            logging.debug("last_position: {}".format(last_position))

            with watcher.watch(full_document='updateLookup', resume_after=last_position) as change_stream:
            i = 0
            state = 0

            while change_stream.alive and i < int(os.environ['Documents_per_run']):
            
                i += 1
                change_event = change_stream.try_next()
                logging.debug('Event: {}'.format(change_event))
                
                
                if change_event is None:
                        Time.sleep(0.5)
                        Continue
                else:
                    op_type = change_event['operationType']
                    op_id = change_event['_id']['_data']

                    if op_type == insert':             
                        doc_body = change_event['fullDocument']
                        doc_id = str(doc_body.pop("_id", None))
                        insert_body = doc_body
                        readable = datetime.datetime.fromtimestamp(change_event['clusterTime'].time).isoformat()
                        doc_body.update({'operation':op_type,'timestamp':str(change_event['clusterTime'].time),'timestampReadable':str(readable)})
                        doc_body.update({'insert_body':json.dumps(insert_body)})
                        doc_body.update({'db':str(change_event['ns']['db']),'coll':str(change_event['ns']['coll'])})
                        payload = {'_id':doc_id}
                        payload.update(doc_body)
                        # Publish event to MSK
                            produce_msg(kafka_client, kfk_topic, op_id, payload)


                    if op_type == 'update':             
                        doc_id = str(documentKey["_id"])
                        readable = datetime.datetime.fromtimestamp(change_event['clusterTime'].time).isoformat()
                        doc_body.update({'operation':op_type,'timestamp':str(change_event['clusterTime'].time),'timestampReadable':str(readable)})
                        doc_body.update({'updateDescription':json.dumps(updateDescription)})
                        doc_body.update({'db':str(change_event['ns']['db']),'coll':str(change_event['ns']['coll'])})
                        payload = {'_id':doc_id}
                        payload.update(doc_body)
                        # Publish event to MSK
                            produce_msg(kafka_client, kfk_topic, op_id, payload)


                    if op_type == 'delete':
                        doc_id = str(change_event['documentKey']['_id'])
                        readable = datetime.datetime.fromtimestamp(change_event['clusterTime'].time).isoformat()
                        doc_body.update({'operation':op_type,'timestamp':str(change_event['clusterTime'].time),'timestampReadable':str(readable)})
                        doc_body.update({'db':str(change_event['ns']['db']),'coll':str(change_event['ns']['coll'])})
                        payload = {'_id':doc_id}
                        payload.update(doc_body)
                        # Append event for S3
                        if "bucket_name" in os.environ:
                            write_S3(op_id, json.dumps(payload))

                        logging.debug('Processed event ID {}'.format(op_id))

                    events_processed += 1

    except OperationFailure as of:
            if of.code == err_code_136:
            # Data for the last processed ID has been deleted in the change stream,
            # Store the last known good state so our next invocation
            # starts from the most recently available data
            save_last_position(None)
        raise

    except Exception as err:
            logging.error(‘Positionpoint lost: {}'.format(err))
        raise

    else:
        
        if events_processed > 0:

            save_last_position(change_stream.resume_token)
            logging.debug('Synced token {} to state collection'.format(change_stream.resume_token))
            return{
                'statusCode': 200,
                'description': 'Success',
                'detail': json.dumps(str(events_processed)+ ' records processed successfully.')
            }
        else:
                return{
                    'statusCode': 201,
                    'description': 'Success',
                    'detail': json.dumps('No records to process.')
                }

    finally:

        # Close Kafka client
        if "kfk_host" in os.environ:                                                 
            kafka_client.close()

容灾 region 的 stream-apply 主机环境设置
Code 部分：

##设置环境变量，请替换红色的文字部分为您实际的值
echo -e "DR_USERNAME="Your DR MongoDB User"\nexport DR_USERNAME\nDR_PASSWORD="Your DR MongoDB Password"\nexport DR_PASSWORD\nDR_mongo_host="Your DR MongoDB cluster URI"\nexport DR_mongo_host\nkfk_host="YOUR KFK URI\nexport kfk_host\nkfk_topic="changevents"\nexport kfk_topic \nDocuments_per_run=100000\nexport Documents_per_run" >> .bash_profile
##应用环境变量
source .bash_profile

容灾 region 的改变流应用程序
在 stream-apply 主机上部署下列 python 代码并运行

Python Code：

#!/bin/env python

import json
import logging
import os
import string
import sys
import time
import boto3
import datetime
from pymongo import MongoClient
from kafka import KafkaConsumer
                                                
db_client = None 
kafka_client = None                                                  

"""ERROR level for deployment."""                                
logging.basicConfig(Level=logging.ERROR)

def get_db_client():
    global db_client

    if db_client is None:
            logging.debug('Creating a new DB client.')

        try:

            username = os.environ[‘DR_USERNAME’]
            password = os.environ[‘DR_PASSWORD’]
            cluster_uri = os.environ[‘DR_mongo_host'] 
            db_client = MongoClient(cluster_uri, ssl=True, retryWrites=False, ssl_ca_certs='/tmp/rds-combined-ca-bundle.pem')
            # Make an attemp for connecting
            db_client.admin.command('ismaster')
            db_client["admin"].authenticate(name=username, password=password)
            logging.debug('Successfully created a new DB client.')
        except Exception as err:
            logging.error('Failed to create a new DB client: {}'.format(err))
            raise

    return db_client

def conn_kfk_comsumer():
    global kafka_client
    
            if kafka_client is None:
            logging.debug('Creating a new Kafka client.')

        try:

            kafka_client = KafkaConsumer(bootstrap_servers=os.environ['kfk_host'])
        except Exception as err:
            logging.error('Failed to create a new Kafka client: {}'.format(err))
            raise
    
    return kafka_client

def poll_msg(consumer, topic_name, key, value):
    """Poll documentdb changes from MSK."""
    
        try:

                topic_name = os.environ['kfk_topic']
                consumer = KafkaConsumer(topic_name, bootstrap_servers= os.environ['kfk_host'], auto_offset_reset=‘latest’, group_id=‘docdb’, key_deserializer=lambda key: json.loads(key).decode('utf-8’), value_deserializer=lambda value: json.loads(value).decode('utf-8’))
                consumer.subscribe(topic_name, key, value)
                consumer.poll(max_records=1)
        except Exception as err:
                logging.error('Error in polling message: {}'.format(err))
                raise


def apply2mongodb(message,db_client)

    try:
        
                # Kafka client set up    
                if "kfk_host" in os.environ:
                kafka_client = conn_kfk_consumer()  
                logging.debug('Kafka client set up.')    

                db_client = get_db_client()

                partition = KafkaConsumer.assignment()
                next_offset = KafkaConsumer.position(partition)
            
                if next_offset is None:
                    Time.sleep(0.5)
                    Continue
                else:
                poll_msg(kafka_client, kfk_topic, op_id, payload)
                for message in consumer:
                event_body = message.value()
              op_type = json.loads(event_body[‘operation'])

                if op_type == 'insert':
                    coll = json.loads(event_body['coll'])
                    coll_client = db_client(coll)
                    insert_body = json.loads(event_body[‘insert_body'])
                    payload = {'_id':ObjectId(json.loads(event_body['_id']))}
                  payload.update(insert_body)
                    coll_client.insert_one(payload)

                if op_type == 'update':
                    coll = json.loads(event_body['coll'])
                    coll_client = db_client(coll)
                    update_body = json.loads(event_body[‘updateDescription']['updatedFields'])
                    update_set = {"$set":update_body}
                    payload = {'_id':(json.loads(event_body['_id']))}
                    coll_client.update_one(payload,update_set)

                    events_processed += 1

def main(event, context):
     events_processed = 0
    kafka_client = None

    try:

                # DocumentDB watched collection set up
                db_client = get_db_client()
                dr_db = os.environ['DR_mongo_host']
                dr_db_client = db_client(dr_db)
                while i < int(os.environ['Documents_per_run']):
                apply2mongodb(message,dr_db_client)
                i += 1

        else:

                if events_processed > 0:

                logging.debug(' {} events been processed successfully'.format(events_processed))
                return{
                'statusCode': 200,
                'description': 'Success',
                'detail': json.dumps(str(events_processed)+ ' events processed successfully.')
            }
        else:
                return{
                    'statusCode': 201,
                    'description': 'Success',
                    'detail': json.dumps('No records to process.')
                }

    finally:

        # Close Kafka client
        if "kfk_host" in os.environ:                                                 
            kafka_client.close()

结果验证

1. 分别登陆主region与容灾region的DocumentDB
   主region：

mongo --host $mongo_host:27017 --ssl --sslCAFile
/tmp/rds-combined-ca-bundle.pem --username $USERNAME --password $PASSWORD

容灾region：

mongo --host $DR_mongo_host:27017 --ssl --sslCAFile
/tmp/rds-combined-ca-bundle.pem --username $USERNAME --password $PASSWORD

2. 在主 region 插入数据

use bar;
db.foo.insertOne({
"x":1}) ;

3. 在灾备 region 观察

use bar;
db.foo.find();
##得到结果
{"_id":ObjectId(9416e4a253875177a816b3d6),"x":1}

3. 在主 region 更新数据

db.foo.updateOne({
"x":1},
{
$set:{"x":2}}
);

4. 在灾备 region 观察

db.foo.find();
##得到结果
{"_id":ObjectId(9416e4a253875177a816b3d6),"x":2}

5.在主 region 非监控表 exa 插入数据y=1

db.exa.insertOne({
"y":1});

6.在主 region 观察有哪些表，发现新增加了 exa 这张表

show tables;
exa
foo

7. 在灾备 region 观察，并没有 exa 出现，因为 exa 并不在我们的 watched collection 里，不会捕捉相关的改变流

show tables;
foo

8. 在主region的foo表删除x记录

db.foo.deleteOne({
"x":2}) ;

##观察得到结果，主 region DocumentDB foo 表已被清空
db.foo.find();
##得到结果为空

9. 在灾备 region 验证 foo 表内容

db.foo.find();
##得到结果
{"_id":ObjectId(9416e4a253875177a816b3d6),"x":2}
##删除操作被拦截

10.下载 S3 中的文件，并打开，其中内容为

{"_id":"ObjectId(9416e4a253875177a816b3d6)", "operation":"delete", "timestamp":1658233222,"timestampReadable":"2022-07-19 20:20:22", "db":"bar","coll":"foo"}
##验证了本条 delete 命令被拦截并保存在 S3 中。

总结
我们在此文中，使用了 MSK 来异步保存 DocumentDB 的 insert/update 改变流，拦截 delete 类型的改变流存储在 S3 中备查。如果需要进一步对删除事件做出分析，可以引入 Amazon Glue 与Amazon Athena 对存储于 S3 中的日志文件即席查询。MSK 中的改变流事件，我们将其应用在灾备区域的 DocumentDB，做到数据只增不减，避免主 region 的数据库因为意外误操作导致的数据损失或者高时间成本数据恢复操作。

参考资源
Amazon Linux 2 上使用 Boto 3 库创建 Python 3 虚拟环境

https://aws.amazon.com/cn/premiumsupport/knowledge-center/ec2...

创建MSK的Topic

https://docs.aws.amazon.com/zh_cn/msk/latest/developerguide/c...

本篇作者

付晓明 Amazon 解决方案架构师，负责云计算解决方案的咨询与架构设计，同时致力于数据库，边缘计算方面的研究和推广。在加入亚马逊云科技之前曾在金融行业IT部门负责互联网券商架构的设计，对分布式，高并发，中间件等具有丰富经验。

刘冰冰 Amazon 数据库解决方案架构师，负责基于Amazon 的数据库解决方案的咨询与架构设计，同时致力于大数据方面的研究和推广。在加入Amazon 之前曾在Oracle工作多年，在数据库云规划、设计运维调优、DR解决方案、大数据和数仓以及企业应用等方面有丰富的经验。

文章来源：https://dev.amazoncloud.cn/column/article/630994cd86218f3ca3e...