harbor仓库私服双主复制高可用

高可用策略

主流的策略有那么几种:
1.harbor做双主复制
2.harbor集群挂载分布式cephfs存储
3.在k8s集群上部署harbor

• 1.第二种和第三种都是多个节点，然后挂载的分布式存储，然后为了保证数据的统一性使用单独的mysql数据库，这样以来存在mysql数据和镜像仓库数据单点存放，故障恢复难度大，安装操作复杂的问题
• 2.双主复制不存在这些问题，数据多点存放，而且扩容更改高可用模式操作简单，可以更换成主主从等模式。

二．安装docker-compose

#cat -a /etc/sysctl.conf <<-EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

然后重新加载 sysctl.conf 即可

#sysctl -p

最好在ifcnf-eth0中配置dns参数，hosts千万不要配置域名和ip。

安装docker-compose

方法一

curl -L https://github.com/docker/compose/releases/download/1.18.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
* 查看版本
docker-compose version

方法二

CentOS:
yum install epel-release -y
yum install python-pip -y
Ubuntu:
apt-get install python-pip -y
# 通用命令
pip --version
pip install --upgrade pip
pip install -U -i https://pypi.tuna.tsinghua.edu.cn/simple docker-compose 
docker-compose version
这里采用方法2

二．安装Harbor

Harbor私有仓库的安装

2个节点安装步骤一致
• 下载Harbor安装文件
从 github harbor 官网 release 页面下载指定版本的安装包。
1、在线安装包（不一定好使，自己找源）

$ wget https://github.com/vmware/harbor/releases/download/v1.1.2/harbor-online-installer-v1.1.2.tgz
$ tar xvf harbor-online-installer-v1.1.2.tgz

2、离线安装包

$ wget https://github.com/vmware/harbor/releases/download/v1.1.2/harbor-offline-installer-v1.1.2.tgz
$ tar xvf harbor-offline-installer-v1.1.2.tgz

• 本次离线安装。

推荐使用第二种，因为第一种在线安装可能由于官网源的网络波动导致安装失败。
• 配置Harbor
解压缩之后，目录下回生成harbor.conf文件，该文件就是Harbor的配置文件。

# cat harbor.cfg 
_version = 1.5.0
hostname = repository.skong.com
ui_url_protocol = https
max_job_workers = 50 
customize_crt = on
ssl_cert = /data/harbor-data/cert/repository.crt
ssl_cert_key = /data/harbor-data/cert/repository.key
secretkey_path = /data/harbor-data/
admiral_url = NA
log_rotate_count = 50
100k, size 100M and size 100G 
log_rotate_size = 200M
http_proxy =
https_proxy =
no_proxy = 127.0.0.1,localhost,ui
email_identity = 
email_server = smtp.mydomain.com
email_server_port = 25
email_username = [e-mail]
email_password = abc
email_from = admin <[e-mail]>
email_ssl = false
email_insecure = false
harbor_admin_password = Harbor123456
auth_mode = db_auth
ldap_url = ldaps://ldap.mydomain.com
ldap_basedn = ou=people,dc=mydomain,dc=com
ldap_uid = uid 
ldap_scope = 2 

ldap_timeout = 5
ldap_verify_cert = true
ldap_group_basedn = ou=group,dc=mydomain,dc=com
ldap_group_filter = objectclass=group
ldap_group_gid = cn
ldap_group_scope = 2
token_expiration = 30
project_creation_restriction = everyone
db_host = mysql
db_password = root123
db_port = 3306
db_user = root
redis_url = redis:6379
clair_db_host = postgres
clair_db_password = password
clair_db_port = 5432
clair_db_username = postgres
clair_db = postgres
uaa_endpoint = uaa.mydomain.org
uaa_clientid = id
uaa_clientsecret = secret
uaa_verify_cert = true
uaa_ca_cert = /path/to/ca.pem
registry_storage_provider_name = filesystem
registry_storage_provider_config =
################################################################

• 创建harbor data目录：

#mkdir -pv /data/harbor-data/cert

# cat docker-compose.yml
version: '2'
services:
  log:
    image: vmware/harbor-log:v1.5.0
    container_name: harbor-log 
    restart: always
    volumes:
      - /data/harbor-data/log/harbor/:/var/log/docker/:z
      - ./common/config/log/:/etc/logrotate.d/:z
    ports:
      - 127.0.0.1:1514:10514
    networks:
      - harbor 
  registry:
    image: vmware/registry-photon:v2.6.2-v1.5.0
    container_name: registry
    restart: always
    volumes:
      - /data/harbor-data/registry:/storage:z
      - ./common/config/registry/:/etc/registry/:z
    networks:
      - harbor 
    environment:
      - GODEBUG=netdns=cgo
    command:
      ["serve", "/etc/registry/config.yml"]
    depends_on:
      - log
    logging:
      driver: "syslog"
      options:  
        syslog-address: "tcp://127.0.0.1:1514"
        tag: "registry"
  mysql:
    image: vmware/harbor-db:v1.5.0
    container_name: harbor-db
    restart: always
    volumes:
      - /data/harbor-data/database:/var/lib/mysql:z
    networks:
      - harbor 
    env_file:
      - ./common/config/db/env
    depends_on:
      - log
    logging:
      driver: "syslog"
      options:  
        syslog-address: "tcp://127.0.0.1:1514"
        tag: "mysql"
  adminserver:
    image: vmware/harbor-adminserver:v1.5.0
    container_name: harbor-adminserver
    env_file:
      - ./common/config/adminserver/env
    restart: always
    volumes:
      - /data/harbor-data/config/:/etc/adminserver/config/:z
      - /data/harbor-data/secretkey:/etc/adminserver/key:z
      - /data/harbor-data/:/data/:z
    networks:
      - harbor 
    depends_on:
      - log
    logging:
      driver: "syslog"
      options:  
        syslog-address: "tcp://127.0.0.1:1514"
        tag: "adminserver"
  ui:
    image: vmware/harbor-ui:v1.5.0
    container_name: harbor-ui
    env_file:
      - ./common/config/ui/env
    restart: always
    volumes:
      - ./common/config/ui/app.conf:/etc/ui/app.conf:z
      - ./common/config/ui/private_key.pem:/etc/ui/private_key.pem:z
      - ./common/config/ui/certificates/:/etc/ui/certificates/:z
      - /data/harbor-data/secretkey:/etc/ui/key:z
      - /data/harbor-data/ca_download/:/etc/ui/ca/:z
      - /data/harbor-data/psc/:/etc/ui/token/:z
    networks:
      - harbor 
    depends_on:
      - log
      - adminserver
      - registry
    logging:
      driver: "syslog"
      options:  
        syslog-address: "tcp://127.0.0.1:1514"
        tag: "ui"
  jobservice:
    image: vmware/harbor-jobservice:v1.5.0
    container_name: harbor-jobservice
    env_file:
      - ./common/config/jobservice/env
    restart: always
    volumes:
      - /data/harbor-data/job_logs:/var/log/jobs:z
      - ./common/config/jobservice/config.yml:/etc/jobservice/config.yml:z
    networks:
      - harbor 
    depends_on:
      - redis
      - ui
      - adminserver
    logging:
      driver: "syslog"
      options:  
        syslog-address: "tcp://127.0.0.1:1514"
        tag: "jobservice"
  redis:
    image: vmware/redis-photon:v1.5.0
    container_name: redis
    restart: always
    volumes:
      - /data/harbor-data/redis:/data
    networks:
      - harbor 
    depends_on:
      - log
    logging:
      driver: "syslog"
      options:  
        syslog-address: "tcp://127.0.0.1:1514"
        tag: "redis"
  proxy:
    image: vmware/nginx-photon:v1.5.0
    container_name: nginx
    restart: always
    volumes:
      - ./common/config/nginx:/etc/nginx:z
    networks:
      - harbor 
    ports:
      - 80:80
      - 443:443
      - 4443:4443
    depends_on:
      - mysql
      - registry
      - ui
      - log
    logging:
      driver: "syslog"
      options:  
        syslog-address: "tcp://127.0.0.1:1514"
        tag: "proxy"
networks:
  harbor:
    external: false

*启动 Harbor
修改完配置文件后，在的当前目录执行./install.sh，Harbor服务就会根据当期目录下的docker-compose.yml开始下载依赖的镜像，检测并按照顺序依次启动各
./install.sh

四．配置https域名证书等

X509存在一些问题在对v3_ca

#vim /etc/pki/tls/openssl.cnf
[ v3_ca ]
# Extensions for a typical CA
subjectAltName = IP:192.168.0.64    #添加

因为要配置https，需要生成自签名的证书

#cd  /data/harbor-data/cert
#openssl req -nodes -subj "/C=CN/ST=BeiJing/L=ChaoYao/CN=basic-repository.skong.com" -newkey rsa:2048 -keyout basic-repository.key -out basic-repository.csr

#openssl x509 -req -days 3650 -in basic-repository.csr -signkey basic-repository.key -out basic-repository.crt

#openssl x509 -req -in basic-repository.csr -CA basic-repository.crt -CAkey basic-repository.key -CAcreateserial -out basic-repository.crt -days 10000

*如下目录是nginx容器的cert目录：（不一定会自己生成）

#mkdir /data/harbor_install/harbor/common/config/nginx/cert/

# ls /etc/docker/certs.d/
basic-registry.skong.com  basic-repository.skong.com  redhat.com  redhat.io  registry.access.redhat.com  registry.skong.com  repository.skong.com
#scp –a basic-repository.crt docker-IP:/etc/docker/cert.d/ basic-repository.skong.com

*在传完证书的docker服务器上执行：

#mkdir –pv  /etc/docker/cert.d/basic-repository.skong.com
#service  docker  restart

# docker login -u admin -p Harbor123456 repository.skong.com
1、停止Harbor
# docker-compose down -v
Stopping nginx ... done
Stopping harbor-jobservice ... done
......
Removing harbor-log ... done
Removing network harbor_harbor

#docker-compose stop

2、启动Harbor

# docker-compose up -d
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... 
......
Creating nginx
Creating harbor-jobservice ... done

#docker-compose start

#docker-compose up –d

3.当需要修改配置文件刷新配置时需要执行： #./prepare

Test：
# docker login -u admin -p Harbor123456 repository.skong.com
# ls /etc/docker/certs.d/
#docker pull basic-registry.skong.com/skong/dubbo:latest
# docker images
#docker tag basic-registry.skong.com/skong/dubbo:latest repository.skong.com/basic/dubbo:latest
# docker push repository.skong.com/basic/dubbo:latest