离线环境在Kubernetes安装KubeSphere v3.1.0

KubeSphere官网地址：https://kubesphere.com.cn
离线安装kubernetes集群参考sealyun官网地址：https://www.sealyun.com

前置要求

1. 先安装好kubernetes，离线安装kubernetes集群参考地址：https://www.sealyun.com

2. 修改Kubernetes集群中所有节点中docker的daemon.json文件

   vi /etc/docker/daemon.json

3. 添加Harbor私有仓库地址
   如果不配置后续在安装KubeSphere组件时候会发送htts请求，这里我们是使用的http，因为在内网离线环境中不需要使用https

   {
     "insecure-registries":
           ["192.168.28.150:8001"]
   }

4. 重启docker服务，让配置生效

   systemctl daemon-reload
   systemctl restart docker

5. 登录Harbor

   docker login 192.168.28.150:8001 -u 用户名 -p 密码

安装NFS服务端

1.安装包
  在可以联网的机器上下载NFS安装包，以后在服务端和客户端安装使用。
yum -y install nfs-utils --downloadonly --downloaddir /home/nfs
  下载好后打包上传到要安装的服务器中，执行下面命令开始安装
rpm -Uvh *.rpm --nodeps --force

2.编辑配置文件
⚠️配置文件中的*是允许所有网段，根据自己实际情况写明网段
cat >/etc/exports <<EOF
/mnt/kubesphere *(insecure,rw,async,no_root_squash) 
EOF

3.创建目录并修改权限
⚠️这里为了方便实验授予了挂载目录权限为777，请根据实际情况修改目录权限和所有者
mkdir /mnt/kubesphere && chmod 777 /mnt/kubesphere

4.启动服务
systemctl enable nfs-server rpcbind && systemctl start nfs-server rpcbind

配置NFS客户端

在所有node节点中安装NFS客户端

rpm -Uvh *.rpm --nodeps --force

在k8s集群中的任意节点中执行如下所以操作

创建文件 vi storageclass.yaml内容如下：

cat >storageclass.yaml <<EOF
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
   name: nfs-provisioner-runner
   namespace: default
rules:
   -  apiGroups: [""]
      resources: ["persistentvolumes"]
      verbs: ["get", "list", "watch", "create", "delete"]
   -  apiGroups: [""]
      resources: ["persistentvolumeclaims"]
      verbs: ["get", "list", "watch", "update"]
   -  apiGroups: ["storage.k8s.io"]
      resources: ["storageclasses"]
      verbs: ["get", "list", "watch"]
   -  apiGroups: [""]
      resources: ["events"]
      verbs: ["watch", "create", "update", "patch"]
   -  apiGroups: [""]
      resources: ["services", "endpoints"]
      verbs: ["get","create","list", "watch","update"]
   -  apiGroups: ["extensions"]
      resources: ["podsecuritypolicies"]
      resourceNames: ["nfs-provisioner"]
      verbs: ["use"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-client-provisioner
spec:
  selector:
    matchLabels:
      app: nfs-client-provisioner
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccount: nfs-provisioner
      containers:
        - name: nfs-client-provisioner
          image: 192.168.28.150:8001/kubesphere-install/kubesphere/nfs-client-provisioner:v3.1.0-k8s1.11
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: nfs-client
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: 192.168.28.160
            - name: NFS_PATH
              value: /mnt/kubesphere
      volumes:
        - name: nfs-client
          nfs:
            server: 192.168.28.160
            path: /mnt/kubesphere
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-storage
provisioner: fuseim.pri/ifs
reclaimPolicy: Retain
EOF

1. 应用storageclass.yaml

   kubectl apply -f storageclass.yaml

2. 设置默认strorageclass

   kubectl patch storageclass nfs-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

3. 验证

   kubectl get pods
   
   NAME                                      READY   STATUS    RESTARTS   AGE
   nfs-client-provisioner-7b9746695c-nrz4n   1/1     Running   0          2m38s

   检查默认存储

   kubectl get sc
   
   NAME                    PROVISIONER      AGE
   nfs-storage (default)   fuseim.pri/ifs   7m22s

准安装镜像

当您在离线环境中安装 KubeSphere 时，需要事先准备一个包含所有必需镜像的镜像包。

• 使用以下命令从能够访问互联网的机器上下载镜像清单文件 images-list.txt：

  curl -L -O https://github.com/kubesphere/ks-installer/releases/download/v3.1.0/images-list.txt

该文件根据不同的模块列出了 ##+modulename 下的镜像。您可以按照相同的规则把自己的镜像添加到这个文件中。要查看完整文件，请参见附录。

• 下载 offline-installation-tool.sh。

  curl -L -O https://github.com/kubesphere/ks-installer/releases/download/v3.1.0/offline-installation-tool.sh

• 使 .sh 文件可执行。

  chmod +x offline-installation-tool.sh

• 您可以执行命令 ./offline-installation-tool.sh -h 来查看如何使用脚本：

  root@master:/home/ubuntu# ./offline-installation-tool.sh -h
  Usage:
   
  ./offline-installation-tool.sh [-l IMAGES-LIST] [-d IMAGES-DIR] [-r PRIVATE-REGISTRY] [-v KUBERNETES-VERSION ]
   
  Description:
  -b                     : save kubernetes' binaries.
  -d IMAGES-DIR          : the dir of files (tar.gz) which generated by `docker save`. default: ./kubesphere-images
  -l IMAGES-LIST         : text file with list of images.
  -r PRIVATE-REGISTRY    : target private registry:port.
  -s                     : save model will be applied. Pull the images in the IMAGES-LIST and save images as a tar.gz file.
  -v KUBERNETES-VERSION  : download kubernetes' binaries. default: v1.17.9
  -h                     : usage message

• 在 offline-installation-tool.sh 中拉取镜像。

  ./offline-installation-tool.sh -s -l images-list.txt -d ./kubesphere-images

  推送镜像至私有仓库

  将打包的镜像文件传输至您的本地机器，并运行以下命令把它推送至仓库。

./offline-installation-tool.sh -l images-list.txt -d ./kubesphere-images -r 192.168.28.150:8001/kubesphere-install

命令中的域名是 192.168.28.150:8001/kubesphere-install。请确保使用您自己仓库的地址。

下载部署文件

与在现有 Kubernetes 集群上在线安装 KubeSphere 相似，您也需要事先下载 cluster-configuration.yaml 和 kubesphere-installer.yaml。

• 执行以下命令下载这两个文件，并将它们传输至您充当任务机的机器，用于安装。

  curl -L -O https://github.com/kubesphere/ks-installer/releases/download/v3.1.0/cluster-configuration.yaml
  curl -L -O https://github.com/kubesphere/ks-installer/releases/download/v3.1.0/kubesphere-installer.yaml

• 编辑 cluster-configuration.yaml 添加您的私有镜像仓库。例如，本教程中的仓库地址是 192.168.28.150:8001/kubesphere-install，将它用作 .spec.local_registry 的值，如下所示：

spec:
  persistence:
    storageClass: "" # 上面步骤已经安装了nfs作为默认存储，这里就不需要指定了
  authentication:
    jwtSecret: ""
  local_registry: 192.168.28.150:8001/kubesphere-install # Add this line manually; make sure you use your own registry address.

您可以在该 YAML 文件中启用可插拔组件，体验 KubeSphere 的更多功能。有关详情，请参考启用可插拔组件。

• 编辑完成后保存 cluster-configuration.yaml。使用以下命令将 ks-installer 替换为您自己仓库的地址。

sed -i "s#^\s*image: kubesphere.*/ks-installer:.*#        image: 192.168.28.150:8001/kubesphere-install/kubesphere/ks-installer:v3.1.0#" kubesphere-installer.yaml

命令中的仓库地址是 192.168.28.150:8001/kubesphere-install。请确保使用您自己仓库的地址。

开始安装

确定完成上面所有步骤后，您可以执行以下命令。

kubectl apply -f kubesphere-installer.yaml
kubectl apply -f cluster-configuration.yaml

查看安装日志：

kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f

验证安装
安装完成后，您会看到以下内容：

#####################################################
###              Welcome to KubeSphere!           ###
#####################################################

Console: http://192.168.28.140:30880
Account: admin
Password: P@88w0rd

NOTES：
  1. After logging into the console, please check the
     monitoring status of service components in
     the "Cluster Management". If any service is not
     ready, please wait patiently until all components 
     are ready.
  2. Please modify the default password after login.

#####################################################
https://kubesphere.io             2021-02-09 13:50:43
#####################################################

现在，您可以通过 http://{IP}:30880 使用默认帐户和密码 admin/P@88w0rd 访问 KubeSphere 的 Web 控制台。
要访问控制台，请确保在您的安全组中打开端口 30880。

附录

KubeSphere v3.1.0 镜像清单

##k8s-images
kubesphere/kube-apiserver:v1.20.4
kubesphere/kube-scheduler:v1.20.4
kubesphere/kube-proxy:v1.20.4
kubesphere/kube-controller-manager:v1.20.4
kubesphere/kube-apiserver:v1.19.8
kubesphere/kube-scheduler:v1.19.8
kubesphere/kube-proxy:v1.19.8
kubesphere/kube-controller-manager:v1.19.8
kubesphere/kube-apiserver:v1.18.6
kubesphere/kube-scheduler:v1.18.6
kubesphere/kube-proxy:v1.18.6
kubesphere/kube-controller-manager:v1.18.6
kubesphere/kube-apiserver:v1.17.9
kubesphere/kube-scheduler:v1.17.9
kubesphere/kube-proxy:v1.17.9
kubesphere/kube-controller-manager:v1.17.9
kubesphere/pause:3.1
kubesphere/pause:3.2
kubesphere/etcd:v3.4.13
calico/cni:v3.16.3
calico/kube-controllers:v3.16.3
calico/node:v3.16.3
calico/pod2daemon-flexvol:v3.16.3
coredns/coredns:1.6.9
kubesphere/k8s-dns-node-cache:1.15.12
openebs/provisioner-localpv:2.3.0
openebs/linux-utils:2.3.0
kubesphere/nfs-client-provisioner:v3.1.0-k8s1.11
##csi-images
csiplugin/csi-neonsan:v1.2.0
csiplugin/csi-neonsan-ubuntu:v1.2.0
csiplugin/csi-neonsan-centos:v1.2.0
csiplugin/csi-provisioner:v1.5.0
csiplugin/csi-attacher:v2.1.1
csiplugin/csi-resizer:v0.4.0
csiplugin/csi-snapshotter:v2.0.1
csiplugin/csi-node-driver-registrar:v1.2.0
csiplugin/csi-qingcloud:v1.2.0
##kubesphere-images
kubesphere/ks-apiserver:v3.1.0
kubesphere/ks-console:v3.1.0
kubesphere/ks-controller-manager:v3.1.0
kubesphere/ks-installer:v3.1.0
kubesphere/kubectl:v1.17.0
kubesphere/kubectl:v1.18.0
kubesphere/kubectl:v1.19.0
kubesphere/kubectl:v1.20.0
redis:5.0.5-alpine
alpine:3.10.4
haproxy:2.0.4
nginx:1.14-alpine
minio/minio:RELEASE.2019-08-07T01-59-21Z
minio/mc:RELEASE.2019-08-07T23-14-43Z
mirrorgooglecontainers/defaultbackend-amd64:1.4
kubesphere/nginx-ingress-controller:v0.35.0
osixia/openldap:1.3.0
csiplugin/snapshot-controller:v2.0.1
kubesphere/kubefed:v0.7.0
kubesphere/tower:v0.2.0
kubesphere/prometheus-config-reloader:v0.42.1
kubesphere/prometheus-operator:v0.42.1
prom/alertmanager:v0.21.0
prom/prometheus:v2.26.0
prom/node-exporter:v0.18.1
kubesphere/ks-alerting-migration:v3.1.0
jimmidyson/configmap-reload:v0.3.0
kubesphere/notification-manager-operator:v1.0.0
kubesphere/notification-manager:v1.0.0
kubesphere/metrics-server:v0.4.2
kubesphere/kube-rbac-proxy:v0.8.0
kubesphere/kube-state-metrics:v1.9.7
openebs/provisioner-localpv:2.3.0
thanosio/thanos:v0.18.0
grafana/grafana:7.4.3
##kubesphere-logging-images
kubesphere/elasticsearch-oss:6.7.0-1
kubesphere/elasticsearch-curator:v5.7.6
kubesphere/fluentbit-operator:v0.5.0
kubesphere/fluentbit-operator:migrator
kubesphere/fluent-bit:v1.6.9
elastic/filebeat:6.7.0
kubesphere/kube-auditing-operator:v0.1.2
kubesphere/kube-auditing-webhook:v0.1.2
kubesphere/kube-events-exporter:v0.1.0
kubesphere/kube-events-operator:v0.1.0
kubesphere/kube-events-ruler:v0.2.0
kubesphere/log-sidecar-injector:1.1
docker:19.03
##istio-images
istio/pilot:1.6.10
istio/proxyv2:1.6.10
jaegertracing/jaeger-agent:1.17
jaegertracing/jaeger-collector:1.17
jaegertracing/jaeger-es-index-cleaner:1.17
jaegertracing/jaeger-operator:1.17.1
jaegertracing/jaeger-query:1.17
kubesphere/kiali:v1.26.1
kubesphere/kiali-operator:v1.26.1
##kubesphere-devops-images
kubesphere/ks-jenkins:2.249.1
jenkins/jnlp-slave:3.27-1
kubesphere/s2ioperator:v3.1.0
kubesphere/s2irun:v2.1.1
kubesphere/builder-base:v2.1.0
kubesphere/builder-nodejs:v2.1.0
kubesphere/builder-maven:v2.1.0
kubesphere/builder-go:v2.1.0
kubesphere/s2i-binary:v2.1.0
kubesphere/tomcat85-java11-centos7:v2.1.0
kubesphere/tomcat85-java11-runtime:v2.1.0
kubesphere/tomcat85-java8-centos7:v2.1.0
kubesphere/tomcat85-java8-runtime:v2.1.0
kubesphere/java-11-centos7:v2.1.0
kubesphere/java-8-centos7:v2.1.0
kubesphere/java-8-runtime:v2.1.0
kubesphere/java-11-runtime:v2.1.0
kubesphere/nodejs-8-centos7:v2.1.0
kubesphere/nodejs-6-centos7:v2.1.0
kubesphere/nodejs-4-centos7:v2.1.0
kubesphere/python-36-centos7:v2.1.0
kubesphere/python-35-centos7:v2.1.0
kubesphere/python-34-centos7:v2.1.0
kubesphere/python-27-centos7:v2.1.0
kubesphere/notification:flyway_v2.1.2
kubesphere/notification:v2.1.2
##openpitrix-images
kubesphere/openpitrix-jobs:v3.1.0
##weave-scope-images
weaveworks/scope:1.13.0
##kubeedge-images
kubeedge/cloudcore:v1.6.1
kubesphere/edge-watcher:v0.1.0
kubesphere/kube-rbac-proxy:v0.5.0
kubesphere/edge-watcher-agent:v0.1.0
##example-images-images
kubesphere/examples-bookinfo-productpage-v1:1.16.2
kubesphere/examples-bookinfo-reviews-v1:1.16.2
kubesphere/examples-bookinfo-reviews-v2:1.16.2
kubesphere/examples-bookinfo-reviews-v3:1.16.2
kubesphere/examples-bookinfo-details-v1:1.16.2
kubesphere/examples-bookinfo-ratings-v1:1.16.3
busybox:1.31.1
joosthofman/wget:1.0
kubesphere/netshoot:v1.0
nginxdemos/hello:plain-text
wordpress:4.8-apache
mirrorgooglecontainers/hpa-example:latest
java:openjdk-8-jre-alpine
fluent/fluentd:v1.4.2-2.0
perl:latest