Found a data backup and recovery artifact! Open source, free, fast, efficient, and safe

Introduction

restic is a fast, efficient and safe backup program. It supports three major operating systems (Linux, macOS, Windows) and some smaller operating systems (FreeBSD, OpenBSD). It is written in the go programming language, uses AES-256 to encrypt data, and uses Poly1305-AES to authenticate data.

github address: https://github.com/restic/restic

Design Principles

Restic is a program that can perform backups correctly, and its design follows the following principles:

• Simple : Backup should be a smooth process, otherwise you may want to skip it. Restic should be easy to configure and use so that in case of data loss, you can directly restore it. Likewise, restoring data should not be complicated.
• Fast : Use restic to back up data should only be limited by the network or hard disk bandwidth, so that you can back up files every day. If it takes too much time, no one will make a backup. The recovery backup should only transfer the data required by the file to be recovered, so this process is also fast.
• verifiable : recovery is more important than backup, so restic allows you to easily verify whether all data can be recovered.
• Security : Restic uses encryption technology to ensure the confidentiality and integrity of your data. Assume that the storage location of the backup data is not in a trusted environment (for example, other people such as system administrators can access your backup shared space). Restic aims to protect your data from such attackers.
• Efficient : As data grows, additional snapshots should only occupy the actual incremental storage. More importantly, before actually writing duplicate data to the storage backend, it should be deduplicated to save valuable backup space.

installation

CentOS

[root@centos7 ~]# yum install yum-plugin-copr -y
[root@centos7 ~]# yum copr enable copart/restic -y
Loaded plugins: copr, fastestmirror
copr done
[root@centos7 ~]# yum install restic -y

If there is an error in the above installation, please execute the following command to solve the source problem

[root@centos7 ~]# yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/copart/restic/repo/epel-7/copart-restic-epel-7.repo
Loaded plugins: fastestmirror
adding repo from: https://copr.fedorainfracloud.org/coprs/copart/restic/repo/epel-7/copart-restic-epel-7.repo
grabbing file https://copr.fedorainfracloud.org/coprs/copart/restic/repo/epel-7/copart-restic-epel-7.repo to /etc/yum.repos.d/copart-restic-epel-7.repo
repo saved to /etc/yum.repos.d/copart-restic-epel-7.repo

macOS

# brew
$ brew install restic

# macprots
$ sudo port install restic

For more installation methods, please refer to: https://restic.readthedocs.io/en/latest/020_installation.html#stable-releases

Configure backup repository

The location where the backup is saved is called the "repository". The repository can be stored locally or on a remote server or service. The following storage methods are supported:

For automatic backups, restic accepts the repository location RESTIC_REPOSITORY in the environment variable. Restic can also read the repository location RESTIC_REPOSITORY_FILE from the file specified by the --repository-file option or environment variable.

For the password, there are several options:

• Set the environment variable RESTIC_PASSWORD
• Specify the file path with the password by option --password-file or environment variable RESTIC_PASSWORD_FILE
• Configure the program to be called when a password is required through the option --password-command or environment variables RESTIC_PASSWORD_COMMAND

Create a local repository

Take the creation of a local repository as an example

[root@centos7 ~]# restic init --repo /restic/backup_dir
enter password for new repository: 
enter password again: 
created restic repository dff64d39c6 at /restic/backup_dir

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
#提示很明白，让你记住在此处输入的密码，丢掉密码就是丢掉了数据

For other ways to create a repository, please refer to the official documentation:
https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html

Backup practice

Back up the contents of the directory data to the repository

[root@centos7 ~]# restic -r /restic/backup_dir --verbose backup ./data
open repository
enter password for repository: 
repository dff64d39 opened successfully, password is correct
created new cache in /root/.cache/restic
lock repository
load index files
no parent snapshot found, will read all files
start scan on [./data]
start backup on [./data]
scan finished in 1.455s: 2922 files, 71.126 MiB

Files:        2922 new,     0 changed,     0 unmodified
Dirs:           99 new,     0 changed,     0 unmodified
Data Blobs:   2889 new
Tree Blobs:     99 new
Added to the repo: 72.083 MiB

processed 2922 files, 71.126 MiB in 0:05  #备份的文件数及大小
snapshot 4d20711e saved                   #创建了文件快照

--verbose #输出过程信息

You can also back up individual files

[root@centos7 ~]# ls ./data
goInception-linux-amd64-v1.2.3.tar.gz  httpd-2.4.6-95.el7.centos.x86_64.rpm  mingongge.z02
httpd-2.4.46                           mingongge.file                        mingongge.zip
httpd-2.4.46.tar.gz                    mingongge.z01
[root@centos7 ~]# restic -r /restic/backup_dir --verbose backup ./data/mingongge.zip
open repository
enter password for repository: 
repository dff64d39 opened successfully, password is correct
lock repository
load index files
no parent snapshot found, will read all files
start scan on [./data/mingongge.zip]
start backup on [./data/mingongge.zip]
scan finished in 0.249s: 1 files, 942.793 KiB

Files:           1 new,     0 changed,     0 unmodified
Dirs:            1 new,     0 changed,     0 unmodified
Data Blobs:      0 new
Tree Blobs:      2 new
Added to the repo: 750 B

processed 1 files, 942.793 KiB in 0:00
snapshot 3e5b7dea saved

If you execute the backup command in the first step again, you will find that it no longer adds content, just adds another snapshot for the current data. In fact, restic has the function of scanning files (scanning and comparing files one by one), so the same data will only be backed up and stored once.

File detection function

Scanning the entire contents of each file is a waste of resources, so restic also uses change detection rules based on file metadata to determine whether the file may not have been changed since the last backup. If it is, it will not scan the file again.

On Unix (including Linux and Mac), given that the file is in the same location as the file in the previous backup, the following file metadata attributes must match to assume that its content has not changed:

• Modify the timestamp (mtime)
• Metadata change timestamp (ctime)
• File size
• inode number (internal number used to refer to files in the file system)

Therefore, based on the above reasons, some parameters are introduced as follows:

--force            #关闭更改检测，重新扫描全部文件
--ignore-ctime    #要求 mtime 匹配，但允许 ctime 不同
--ignore-inode    #要求 mtime 匹配，但允许 inode number 和 ctime 不同

Exclude file parameters

--exclude      #指定一次或多次排除一个或多个项
--iexclude    #与exclude相同，但忽略路径的情况
--exclude-caches    #指定一次排除包含特殊文件的文件夹
--exclude-file      #指定一次排除包含特殊文件的文件夹
--iexclude-file      #与exclude-file相同，但忽略路径的情况
--exclude-if-present foo      #排除文件夹包含名为foo的文件
--exclude-larger-than size    #指定一次以排除大于给定大小的文件

For more related functions, please refer to: https://restic.readthedocs.io/en/latest/040_backup.html

Repository usage

Now that the data is backed up to the storage library, we also need to use the storage library. Here are the related operations.

List all snapshots of the repository

This function is the same as using the ls command on the system at ordinary times. View the displayed function

[root@centos7 ~]# restic -r /restic/backup_dir/ snapshots
enter password for repository: 
repository dff64d39 opened successfully, password is correct
ID        Time                 Host        Tags        Paths
-------------------------------------------------------------------------------
4d20711e  2021-06-04 03:40:47  centos7                 /root/data
3e5b7dea  2021-06-04 03:46:34  centos7                 /root/data/mingongge.zip
94c62288  2021-06-04 03:51:21  centos7                 /root/data
-------------------------------------------------------------------------------
3 snapshots

#还可以使用下面的参数进行过滤匹配查看
--path="dir_name"
--host hostname
#通过相同的过滤器（主机、路径、标签）对输出进行分组
--group-by

For more information, please refer to: https://restic.readthedocs.io/en/latest/045_working_with_repos.html

Detect repository data

[root@centos7 ~]# restic -r /restic/backup_dir/ check
using temporary cache in /tmp/restic-check-cache-294136679
enter password for repository: 
repository dff64d39 opened successfully, password is correct
created new cache in /tmp/restic-check-cache-294136679
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  3 / 3 snapshots
no errors were found

Data Recovery

This is the point, and restoring data is the king.

Create an environment that simulates data deletion

[root@centos7 ~]# cd data/
[root@centos7 data]# ll
total 33796
-rw-r--r--  1 root root 13034487 Aug 30  2020 goInception-linux-amd64-v1.2.3.tar.gz
drwxr-sr-x 11 root   40     4096 Dec 24 22:35 httpd-2.4.46
-rw-r--r--  1 root root  9363314 Aug  5  2020 httpd-2.4.46.tar.gz
-rw-r--r--  1 root root  2846172 Oct 14  2020 httpd-2.4.6-95.el7.centos.x86_64.rpm
-rw-r--r--  1 root root        0 Jan 16 11:32 mingongge.file
-rw-r--r--  1 root root  4194304 Jan 16 16:24 mingongge.z01
-rw-r--r--  1 root root  4194304 Jan 16 16:24 mingongge.z02
-rw-r--r--  1 root root   965420 Jan 16 16:24 mingongge.zip
[root@centos7 data]# rm -rf ./*
[root@centos7 data]# ll
total 0

Data recovery

Recover accidentally deleted data directly from the snapshot

[root@centos7 ~]# restic -r /restic/backup_dir/ restore 4d20711e --target /root/
enter password for repository: 
repository dff64d39 opened successfully, password is correct
restoring <Snapshot 4d20711e of [/root/data] at 2021-06-04 03:40:47.878873654 -0400 EDT by root@centos7> to /root/
[root@centos7 ~]# ll /root/data/
total 33796
-rw-r--r--  1 root root 13034487 Aug 30  2020 goInception-linux-amd64-v1.2.3.tar.gz
drwxr-sr-x 11 root   40     4096 Dec 24 22:35 httpd-2.4.46
-rw-r--r--  1 root root  9363314 Aug  5  2020 httpd-2.4.46.tar.gz
-rw-r--r--  1 root root  2846172 Oct 14  2020 httpd-2.4.6-95.el7.centos.x86_64.rpm
-rw-r--r--  1 root root        0 Jan 16 11:32 mingongge.file
-rw-r--r--  1 root root  4194304 Jan 16 16:24 mingongge.z01
-rw-r--r--  1 root root  4194304 Jan 16 16:24 mingongge.z02
-rw-r--r--  1 root root   965420 Jan 16 16:24 mingongge.zip

The above recovery method is quite designated to restore data from a certain snapshot. Sometimes there are many snapshots. If you don't know how to quickly restore from which snapshot, you can use the latest parameter to restore data from the last backup snapshot.

[root@centos7 ~]# rm -rf /root/data/*
[root@centos7 ~]# ll /root/data/
total 0
[root@centos7 ~]# restic -r /restic/backup_dir/ restore latest --target /root/
enter password for repository: 
repository dff64d39 opened successfully, password is correct
restoring <Snapshot 94c62288 of [/root/data] at 2021-06-04 03:51:21.232686491 -0400 EDT by root@centos7> to /root/
[root@centos7 ~]# ll /root/data/
total 33796
-rw-r--r--  1 root root 13034487 Aug 30  2020 goInception-linux-amd64-v1.2.3.tar.gz
drwxr-sr-x 11 root   40     4096 Dec 24 22:35 httpd-2.4.46
-rw-r--r--  1 root root  9363314 Aug  5  2020 httpd-2.4.46.tar.gz
-rw-r--r--  1 root root  2846172 Oct 14  2020 httpd-2.4.6-95.el7.centos.x86_64.rpm
-rw-r--r--  1 root root        0 Jan 16 11:32 mingongge.file
-rw-r--r--  1 root root  4194304 Jan 16 16:24 mingongge.z01
-rw-r--r--  1 root root  4194304 Jan 16 16:24 mingongge.z02
-rw-r--r--  1 root root   965420 Jan 16 16:24 mingongge.zip

Other management introduction

Delete snapshot

The snapshot and the backup file name here are the same. If there are more snapshots, they may need to be deleted regularly. Generally, the data backup time is basically a 30-day cycle. Except for particularly important data, it may be longer. This is based on actual conditions. The situation is fixed.

[root@centos7 ~]# restic -r /restic/backup_dir/ snapshots
enter password for repository: 
repository dff64d39 opened successfully, password is correct
ID        Time                 Host        Tags        Paths
-------------------------------------------------------------------------------
4d20711e  2021-06-04 03:40:47  centos7                 /root/data
3e5b7dea  2021-06-04 03:46:34  centos7                 /root/data/mingongge.zip
94c62288  2021-06-04 03:51:21  centos7                 /root/data
-------------------------------------------------------------------------------
3 snapshots
[root@centos7 ~]# restic -r /restic/backup_dir/ forget 4d20711e
enter password for repository: 
repository dff64d39 opened successfully, password is correct
[0:00] 100.00%  1 / 1 files deleted
[root@centos7 ~]# restic -r /restic/backup_dir/ snapshots
enter password for repository: 
repository dff64d39 opened successfully, password is correct
ID        Time                 Host        Tags        Paths
-------------------------------------------------------------------------------
3e5b7dea  2021-06-04 03:46:34  centos7                 /root/data/mingongge.zip
94c62288  2021-06-04 03:51:21  centos7                 /root/data
-------------------------------------------------------------------------------
2 snapshots

#这个方式和恢复相同，都是指定快照ID

This method of deletion only deletes the snapshot, but the referenced file is still in the storage library, which means that the size of the storage library has not changed. You need to use the prune parameter to clear the data.

[root@centos7 ~]# restic -r /restic/backup_dir/ prune
enter password for repository: 
repository dff64d39 opened successfully, password is correct
loading indexes...
loading all snapshots...
finding data that is still in use for 2 snapshots
[0:00] 100.00%  2 / 2 snapshots
searching used packs...
collecting packs for deletion and repacking
[0:00] 100.00%  19 / 19 packs processed

to repack:            0 blobs / 0 B
this removes          0 blobs / 0 B
to delete:            0 blobs / 0 B
total prune:          0 blobs / 0 B
remaining:         2990 blobs / 72.175 MiB
unused size after prune: 0 B (0.00% of remaining size)

done

You can also use the following command to perform the above two steps at the same time, which is equivalent to a merge operation

restic forget --keep-last 1 --prune
#--keep-last 1 不删除最后一次的快照

For more information about snapshot deletion, please refer to: https://restic.readthedocs.io/en/latest/060_forget.html

Password management of the repository

We can use the key command to set multiple passwords for the same repository, and we can also use add, list, remove, and passwd to manage passwords.

• View password

[root@centos7 ~]# restic -r /restic/backup_dir/ key list
enter password for repository: 
repository dff64d39 opened successfully, password is correct
 ID        User  Host     Created
---------------------------------------------
*d216779f  root  centos7  2021-06-04 03:28:34
---------------------------------------------

• Add password

[root@centos7 ~]# restic -r /restic/backup_dir/ key add
enter password for repository: 
repository dff64d39 opened successfully, password is correct
enter new password: 
enter password again: 
saved new key as <Key of root@centos7, created on 2021-06-04 04:43:18.024358447 -0400 EDT m=+18.001857421>
[root@centos7 ~]# restic -r /restic/backup_dir/ key list
enter password for repository: 
repository dff64d39 opened successfully, password is correct
 ID        User  Host     Created
---------------------------------------------
*d216779f  root  centos7  2021-06-04 03:28:34
 33d0b428  root  centos7  2021-06-04 04:43:18
---------------------------------------------

• delete

  [root@centos7 ~]# restic -r /restic/backup_dir/ key remove 33d0b428
  enter password for repository: 
  repository dff64d39 opened successfully, password is correct
  removed key 33d0b428cba5c62585f1190432e61d46b88b4a6418c693d09ec47db596eace1f
  [root@centos7 ~]# restic -r /restic/backup_dir/ key list
  enter password for repository: 
  repository dff64d39 opened successfully, password is correct
   ID        User  Host     Created
  ---------------------------------------------
  *d216779f  root  centos7  2021-06-04 03:28:34
  ---------------------------------------------

• change the password

  [root@centos7 ~]# restic -r /restic/backup_dir/ key passwd
  enter password for repository: 
  repository dff64d39 opened successfully, password is correct
  enter new password: 
  enter password again: 
  saved new key as <Key of root@centos7, created on 2021-06-04 04:51:13.658184739 -0400 EDT m=+27.022974479>
  [root@centos7 ~]# restic -r /restic/backup_dir/ key list
  enter password for repository: 
  repository dff64d39 opened successfully, password is correct
   ID        User  Host     Created
  ---------------------------------------------
  *a62168f6  root  centos7  2021-06-04 04:51:13
  ---------------------------------------------
  #密码已更新完成，ID变化上可以看出来

common problem

https://restic.readthedocs.io/en/latest/faq.html

Interested readers can install and experience more related functions and operations. Migrant brother, I experienced it in a circle and felt that this restic is indeed a very powerful, fast, safe and efficient backup artifact. The key is that it is open source and free. It is too fragrant and highly recommended.