ES 快照到 S3 并从 Windows 共享目录恢复(qbit)

qbit

前言

  • 业务需要将 Elasticsearch 快照到 AWS S3,再将快照拷贝到 Windows 系统,并恢复到 Elasticsearch。如下图所示:
    image.png
  • 环境

    Elasticsearch 7.10.1
    Windows Server 2019
    Ubuntu 20.04 (ES 宿主)

ES 集群1 安装 S3 插件

  • 官方文档: https://www.elastic.co/guide/...
  • 应在 ES 集群的所有节点上安装插件
  • 下载 S3 插件SHA hash,并上传到 ES 服务器
  • 查看并校验

    # ll -ah
    total 4.6M
    drwxr-xr-x 2 qhy qhy 4.0K Mar  9 01:55 ./
    drwxr-xr-x 7 qhy qhy 4.0K Mar  9 01:50 ../
    -rw-rw-r-- 1 qhy qhy 4.6M Mar  9 01:55 repository-s3-7.10.1.zip
    -rw-rw-r-- 1 qhy qhy  154 Mar  9 01:55 repository-s3-7.10.1.zip.sha512
    
    # shasum -a 512 -c repository-s3-7.10.1.zip.sha512 
    repository-s3-7.10.1.zip: OK
  • 安装

    # /usr/share/elasticsearch/bin/elasticsearch-plugin install file:///home/qhy/es_down/repository-s3-7.10.1.zip
    -> Installing file:///home/qhy/es_down/repository-s3-7.10.1.zip
    -> Downloading file:///home/qhy/es_down/repository-s3-7.10.1.zip
    [=================================================] 100%   
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @     WARNING: plugin requires additional permissions     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    java.lang.RuntimePermission accessDeclaredMembers
    java.lang.RuntimePermission getClassLoader
    java.lang.reflect.ReflectPermission suppressAccessChecks
    java.net.SocketPermission * connect,resolve
    java.util.PropertyPermission es.allow_insecure_settings read,write
    See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
    for descriptions of what these permissions allow and the associated risks.
    
    Continue with installation? [y/N]y
    -> Installed repository-s3
  • 重启 ES 服务

ES 集群1 添加 KEY

  • 官方文档: https://www.elastic.co/guide/...
  • 应在 ES 集群的所有节点上添加 key
  • 查看有哪些 key

    /usr/share/elasticsearch/bin/elasticsearch-keystore list
    
  • 添加 access_key

    /usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.access_key
  • 添加 secret_key

    /usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.secret_key
  • 重载配置

    POST _nodes/reload_secure_settings

创建 S3 快照仓库

PUT _snapshot/my_s3_repository
{
  "type": "s3",
  "settings": {
    "endpoint": "s3.cn-northwest-1.amazonaws.com.cn",
    "bucket": "zt-hadoop-cn-northwest-1",
    "base_path": "es_snapshot", 
    "max_snapshot_bytes_per_sec": "200mb",  # 调整快照创建的速度,默认 40mb
    "max_restore_bytes_per_sec": "200mb"    # 调整快照恢复的速度,默认无限制
  }
}
  • 查看所有注册的快照仓库
GET _snapshot/_all
  • 删除快照仓库
DELETE _snapshot/my_s3_repository
  • 调整集群恢复分片速度和并发数
PUT _cluster/settings 
{
  "transient": {
    "indices.recovery.max_bytes_per_sec": "200mb",  # 默认 40mb
    "cluster.routing.allocation.node_concurrent_recoveries": "5"    # 默认 2
  }
}
  • 查看集群配置(包括默认配置)
GET _cluster/settings?flat_settings&include_defaults

创建快照

PUT /_snapshot/my_s3_repository/snapshot_zt
{
  "indices": "zt_product_doc_index_20210223_3",
  "ignore_unavailable": true,
  "include_global_state": false
}
  • 查看一个my_s3_repository仓库下的所有快照
GET _snapshot/my_s3_repository/_all
  • 查看 snapshot_zt 快照的概要状态
GET _snapshot/my_s3_repository/snapshot_zt
  • 查看 snapshot_zt 快照的详细状态
GET _snapshot/my_s3_repository/snapshot_zt/_status
  • 删除快照
DELETE _snapshot/my_s3_repository/snapshot_zt

从 S3 恢复快照

  • 官方文档: https://www.elastic.co/guide/...

    POST /_snapshot/my_s3_repository/snapshot_zt/_restore
    {
      "indices": "zt_product_doc_index_20210223_3",
      "index_settings": {
    "index.number_of_replicas": 0
      },
      "rename_pattern": "zt_product_doc_index_20210223_3",
      "rename_replacement": "zt_product_doc_index_20210223_3_restore"
    }
  • 增加副本

    PUT zt_product_doc_index_20210223_3_restore/_settings
    {
      "index.number_of_replicas" : "1"
    }

从 Windows 共享目录恢复快照

  • 已将 S3 上的文件拷贝到 Windows 的共享目录,并挂载到 ES 集群2 服务器的 /mnt/winshare 目录

    # tree -d /mnt/winshare/
    /mnt/winshare/
    └── es_snapshot
      └── indices
          └── kM6SVcCrTUGjP-634aDUYg
              ├── 0
              ├── 1
              └── 2
  • Linux 挂载 Windows 共享目录参见: Ubuntu 20.04 读写 Windows 10 共享目录
  • elasticsearch.yml 文件中添加如下配置,并重启 ES 服务

    path.repo:
    - /mnt/winshare
  • 创建 Windows 系统的快照仓库

    PUT /_snapshot/my_backup
    {
    "type": "fs",
    "settings": {
      "location": "/mnt/winshare/es_snapshot"
    }
    }
  • 官方文档

    POST /_snapshot/my_backup/snapshot_zt/_restore
    {
    "indices": "zt_product_doc_index_20210223_3",
    "index_settings": {
      "index.number_of_replicas": 0
    },
    "rename_pattern": "zt_product_doc_index_20210223_3",
    "rename_replacement": "zt_product_doc_index_20210223_3_smb"
    }
  • 增加副本

    PUT zt_product_doc_index_20210223_3_smb/_settings
    {
    "index.number_of_replicas" : "1"
    }
本文出自 qbit snap
阅读 1.1k

qbit snap
开箱即用,拿走不谢。
206 声望
267 粉丝
0 条评论
你知道吗?

206 声望
267 粉丝
文章目录
宣传栏