Istioctl设计分析

前言

为了方便用户安装istio和操作相应的manifest，社区将原本的纯helm安装模式演变成如今的cli和operator模式(https://docs.google.com/docum...)。同时为了保证后向兼容和良好的代码复用，cli复用operator提供的命令行api，operator命令行基于helm的chart渲染进行实现。基本的模式结构如下图所示：

因此，本文从istioctl提供的install和manifest指令出发，按照istioctl->operator->helm的方向进行分析，重点描述operator中cmd的实现流程。

Install

istioctl

命令行参数与实例

此节可参考https://istio.io/latest/docs/...官方文档。

代码入口

istioctl本质上是利用cobra编写的命令行程序，在/istio/istioctl/cmd/root.go中将InstallCmd加入RootCmd中，而InstallCmd由operator实现。

installCmd := mesh.InstallCmd(loggingOptions)
hideInheritedFlags(installCmd, FlagNamespace, FlagIstioNamespace, FlagCharts)
rootCmd.AddCommand(installCmd)

operator

代码流程与重要函数

整体结构如下图所示。

接下来，以istioctl install --set installPackagePath=/istio/manifests --set profile=empty -f test0.yaml -f test1.yaml为例，分析各个函数。其中test0.yaml和test1.yaml的内容如下图所示。

# test0.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: minimal
  hub: docker.io/istio
  tag: 1.1.4
  components:
    pilot:
      enabled: true

# test1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  hub: docker.io/istio
  tag: 1.1.5
  components:
    pilot:
      enabled: true

ParseYAMLFiles

ParseYAMLFiles从左到右读取并覆盖-f标志所传入的IstioOperator yaml，并提取出其中的profile字段。因此提取出的profile为minimal，test0.yaml和test1.yaml的初步处理结果temp0.yaml如下所示。

# temp0.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: minimal
  hub: docker.io/istio
  tag: 1.1.5
  components:
    pilot:
      enabled: true

ReadYaml(And)Profile

ReadYamlProfile根据--set profile=empty对ParseYAMLFiles返回的profile结果进行覆盖，因此最终的profile为empty。

GetProfileYAML

GetProfileYAML从--set installPackagePath=/istio/manifests指定的目录读取empty yaml，因为istio所提供的所有内置profile都基于default profile，因此利用empty yaml去覆盖default profile，最终得到temp1.yaml。

#empty yaml

# The empty profile has everything disabled
# This is useful as a base for custom user configuration
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    base:
      enabled: false
    pilot:
      enabled: false
    ingressGateways:
    - name: istio-ingressgateway
      enabled: false

#default yaml

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
spec:
  hub: gcr.io/istio-testing
  tag: latest

  # You may override parts of meshconfig by uncommenting the following lines.
  meshConfig:
    defaultConfig:
      proxyMetadata: {}
    enablePrometheusMerge: true
    # Opt-out of global http2 upgrades.
    # Destination rule is used to opt-in.
    # h2_upgrade_policy: DO_NOT_UPGRADE

  # Traffic management feature
  components:
    base:
      enabled: true
    pilot:
      enabled: true

    # Istio Gateway feature
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
    egressGateways:
    - name: istio-egressgateway
      enabled: false

    # Istio CNI feature
    cni:
      enabled: false
    
    # Remote and config cluster configuration for an external istiod
    istiodRemote:
      enabled: false

  # Global values passed through to helm global.yaml.
  # Please keep this in sync with manifests/charts/global.yaml
  values:
    defaultRevision: ""
    global:
      istioNamespace: istio-system
      istiod:
        enableAnalysis: false
      logging:
        level: "default:info"
      logAsJson: false
      pilotCertProvider: istiod
      jwtPolicy: third-party-jwt
      proxy:
        image: proxyv2
        clusterDomain: "cluster.local"
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 2000m
            memory: 1024Mi
        logLevel: warning
        componentLogLevel: "misc:error"
        privileged: false
        enableCoreDump: false
        statusPort: 15020
        readinessInitialDelaySeconds: 1
        readinessPeriodSeconds: 2
        readinessFailureThreshold: 30
        includeIPRanges: "*"
        excludeIPRanges: ""
        excludeOutboundPorts: ""
        excludeInboundPorts: ""
        autoInject: enabled
        tracer: "zipkin"
      proxy_init:
        image: proxyv2
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 10m
            memory: 10Mi
      # Specify image pull policy if default behavior isn't desired.
      # Default behavior: latest images will be Always else IfNotPresent.
      imagePullPolicy: ""
      operatorManageWebhooks: false
      tracer:
        lightstep: {}
        zipkin: {}
        datadog: {}
        stackdriver: {}
      imagePullSecrets: []
      oneNamespace: false
      defaultNodeSelector: {}
      configValidation: true
      multiCluster:
        enabled: false
        clusterName: ""
      omitSidecarInjectorConfigMap: false
      network: ""
      defaultResources:
        requests:
          cpu: 10m
      defaultPodDisruptionBudget:
        enabled: true
      priorityClassName: ""
      useMCP: false
      sds:
        token:
          aud: istio-ca
      sts:
        servicePort: 0
      meshNetworks: {}
      mountMtlsCerts: false
    base:
      enableCRDTemplates: false
      validationURL: ""
    pilot:
      autoscaleEnabled: true
      autoscaleMin: 1
      autoscaleMax: 5
      replicaCount: 1
      image: pilot
      traceSampling: 1.0
      env: {}
      cpu:
        targetAverageUtilization: 80
      nodeSelector: {}
      keepaliveMaxServerConnectionAge: 30m
      enableProtocolSniffingForOutbound: true
      enableProtocolSniffingForInbound: true
      deploymentLabels:
      podLabels: {}
      configMap: true

    telemetry:
      enabled: true
      v2:
        enabled: true
        metadataExchange:
          wasmEnabled: false
        prometheus:
          wasmEnabled: false
          enabled: true
        stackdriver:
          enabled: false
          logging: false
          monitoring: false
          topology: false
          configOverride: {}

    istiodRemote:
      injectionURL: ""
      
    gateways:
      istio-egressgateway:
        env: {}
        autoscaleEnabled: true
        type: ClusterIP
        name: istio-egressgateway
        secretVolumes:
          - name: egressgateway-certs
            secretName: istio-egressgateway-certs
            mountPath: /etc/istio/egressgateway-certs
          - name: egressgateway-ca-certs
            secretName: istio-egressgateway-ca-certs
            mountPath: /etc/istio/egressgateway-ca-certs

      istio-ingressgateway:
        autoscaleEnabled: true
        type: LoadBalancer
        name: istio-ingressgateway
        env: {}
        secretVolumes:
          - name: ingressgateway-certs
            secretName: istio-ingressgateway-certs
            mountPath: /etc/istio/ingressgateway-certs
          - name: ingressgateway-ca-certs
            secretName: istio-ingressgateway-ca-certs
            mountPath: /etc/istio/ingressgateway-ca-certs

#temp1.yaml

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
spec:
  components:
    base:
      enabled: false
    cni:
      enabled: false
    egressGateways:
    - enabled: false
      name: istio-egressgateway
    ingressGateways:
    - enabled: false
      name: istio-ingressgateway
    istiodRemote:
      enabled: false
    pilot:
      enabled: false
  hub: gcr.io/istio-testing
  meshConfig:
    defaultConfig:
      proxyMetadata: {}
    enablePrometheusMerge: true
  tag: latest
  values:
    base:
      enableCRDTemplates: false
      validationURL: ""
    defaultRevision: ""
    gateways:
      istio-egressgateway:
        autoscaleEnabled: true
        env: {}
        name: istio-egressgateway
        secretVolumes:
        - mountPath: /etc/istio/egressgateway-certs
          name: egressgateway-certs
          secretName: istio-egressgateway-certs
        - mountPath: /etc/istio/egressgateway-ca-certs
          name: egressgateway-ca-certs
          secretName: istio-egressgateway-ca-certs
        type: ClusterIP
      istio-ingressgateway:
        autoscaleEnabled: true
        env: {}
        name: istio-ingressgateway
        secretVolumes:
        - mountPath: /etc/istio/ingressgateway-certs
          name: ingressgateway-certs
          secretName: istio-ingressgateway-certs
        - mountPath: /etc/istio/ingressgateway-ca-certs
          name: ingressgateway-ca-certs
          secretName: istio-ingressgateway-ca-certs
        type: LoadBalancer
    global:
      configValidation: true
      defaultNodeSelector: {}
      defaultPodDisruptionBudget:
        enabled: true
      defaultResources:
        requests:
          cpu: 10m
      imagePullPolicy: ""
      imagePullSecrets: []
      istioNamespace: istio-system
      istiod:
        enableAnalysis: false
      jwtPolicy: third-party-jwt
      logAsJson: false
      logging:
        level: default:info
      meshNetworks: {}
      mountMtlsCerts: false
      multiCluster:
        clusterName: ""
        enabled: false
      network: ""
      omitSidecarInjectorConfigMap: false
      oneNamespace: false
      operatorManageWebhooks: false
      pilotCertProvider: istiod
      priorityClassName: ""
      proxy:
        autoInject: enabled
        clusterDomain: cluster.local
        componentLogLevel: misc:error
        enableCoreDump: false
        excludeIPRanges: ""
        excludeInboundPorts: ""
        excludeOutboundPorts: ""
        image: proxyv2
        includeIPRanges: '*'
        logLevel: warning
        privileged: false
        readinessFailureThreshold: 30
        readinessInitialDelaySeconds: 1
        readinessPeriodSeconds: 2
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 100m
            memory: 128Mi
        statusPort: 15020
        tracer: zipkin
      proxy_init:
        image: proxyv2
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 10m
            memory: 10Mi
      sds:
        token:
          aud: istio-ca
      sts:
        servicePort: 0
      tracer:
        datadog: {}
        lightstep: {}
        stackdriver: {}
        zipkin: {}
      useMCP: false
    istiodRemote:
      injectionURL: ""
    pilot:
      autoscaleEnabled: true
      autoscaleMax: 5
      autoscaleMin: 1
      configMap: true
      cpu:
        targetAverageUtilization: 80
      deploymentLabels: null
      enableProtocolSniffingForInbound: true
      enableProtocolSniffingForOutbound: true
      env: {}
      image: pilot
      keepaliveMaxServerConnectionAge: 30m
      nodeSelector: {}
      podLabels: {}
      replicaCount: 1
      traceSampling: 1
    telemetry:
      enabled: true
      v2:
        enabled: true
        metadataExchange:
          wasmEnabled: false
        prometheus:
          enabled: true
          wasmEnabled: false
        stackdriver:
          configOverride: {}
          enabled: false
          logging: false
          monitoring: false
          topology: false

OverlayIOP

OverlayIOP将ReadYaml(And)Profile所得到的temp0.yaml覆盖于temp1.yaml之上，得到final.yaml

#empty yaml

# The empty profile has everything disabled
# This is useful as a base for custom user configuration
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    base:
      enabled: false
    pilot:
      enabled: false
    ingressGateways:
    - name: istio-ingressgateway
      enabled: false

# final.yaml

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  creationTimestamp: null
  namespace: istio-system
spec:
  components:
    base:
      enabled: false
    cni:
      enabled: false
    egressGateways:
    - enabled: false
      name: istio-egressgateway
    ingressGateways:
    - enabled: false
      name: istio-ingressgateway
    istiodRemote:
      enabled: false
    pilot:
      enabled: true
  hub: docker.io/istio
  installPackagePath: /istio/manifests
  meshConfig:
    defaultConfig:
      proxyMetadata: {}
    enablePrometheusMerge: true
  profile: empty
  tag: 1.1.5
  values:
    base:
      enableCRDTemplates: false
      validationURL: ""
    defaultRevision: ""
    gateways:
      istio-egressgateway:
        autoscaleEnabled: true
        env: {}
        name: istio-egressgateway
        secretVolumes:
        - mountPath: /etc/istio/egressgateway-certs
          name: egressgateway-certs
          secretName: istio-egressgateway-certs
        - mountPath: /etc/istio/egressgateway-ca-certs
          name: egressgateway-ca-certs
          secretName: istio-egressgateway-ca-certs
        type: ClusterIP
      istio-ingressgateway:
        autoscaleEnabled: true
        env: {}
        name: istio-ingressgateway
        secretVolumes:
        - mountPath: /etc/istio/ingressgateway-certs
          name: ingressgateway-certs
          secretName: istio-ingressgateway-certs
        - mountPath: /etc/istio/ingressgateway-ca-certs
          name: ingressgateway-ca-certs
          secretName: istio-ingressgateway-ca-certs
        type: LoadBalancer
    global:
      configValidation: true
      defaultNodeSelector: {}
      defaultPodDisruptionBudget:
        enabled: true
      defaultResources:
        requests:
          cpu: 10m
      imagePullPolicy: ""
      imagePullSecrets: []
      istioNamespace: istio-system
      istiod:
        enableAnalysis: false
      jwtPolicy: third-party-jwt
      logAsJson: false
      logging:
        level: default:info
      meshNetworks: {}
      mountMtlsCerts: false
      multiCluster:
        clusterName: ""
        enabled: false
      network: ""
      omitSidecarInjectorConfigMap: false
      oneNamespace: false
      operatorManageWebhooks: false
      pilotCertProvider: istiod
      priorityClassName: ""
      proxy:
        autoInject: enabled
        clusterDomain: cluster.local
        componentLogLevel: misc:error
        enableCoreDump: false
        excludeIPRanges: ""
        excludeInboundPorts: ""
        excludeOutboundPorts: ""
        image: proxyv2
        includeIPRanges: '*'
        logLevel: warning
        privileged: false
        readinessFailureThreshold: 30
        readinessInitialDelaySeconds: 1
        readinessPeriodSeconds: 2
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 100m
            memory: 128Mi
        statusPort: 15020
        tracer: zipkin
      proxy_init:
        image: proxyv2
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 10m
            memory: 10Mi
      sds:
        token:
          aud: istio-ca
      sts:
        servicePort: 0
      tracer:
        datadog: {}
        lightstep: {}
        stackdriver: {}
        zipkin: {}
      useMCP: false
    istiodRemote:
      injectionURL: ""
    pilot:
      autoscaleEnabled: true
      autoscaleMax: 5
      autoscaleMin: 1
      configMap: true
      cpu:
        targetAverageUtilization: 80
      deploymentLabels: null
      enableProtocolSniffingForInbound: true
      enableProtocolSniffingForOutbound: true
      env: {}
      image: pilot
      keepaliveMaxServerConnectionAge: 30m
      nodeSelector: {}
      podLabels: {}
      replicaCount: 1
      traceSampling: 1
    telemetry:
      enabled: true
      v2:
        enabled: true
        metadataExchange:
          wasmEnabled: false
        prometheus:
          enabled: true
          wasmEnabled: false
        stackdriver:
          configOverride: {}
          enabled: false
          logging: false
          monitoring: false
          topology: false

GenerateConfig

配置部分总结如下，operator将绝大部分面向用户的配置复杂度都隐藏在了内置profile中，其中base profile是基础演变项，演变过程为base profile -> select profile -> user config -> set flag，越往后优先级越高。在具体实践中，内置profile能满足大多数需求，若有额外需求则使用user config解决，set flag不便于管理。

RenderCharts

前面提到，operator要使用helm的chart rendering，因此通过GenerateConfig生成的final.yaml可以映射到helm的values.yaml。并且，为了便于用户理解与维护helm chart，istio控制面的每一个组件都有一个chart与之对应，final.yaml中各个组件的配置最终映射到各个chart的values.yaml进行渲染。charts目录如下所示。

processRecursive

每个组件的chart渲染成功后生成所需的manifest，并利用k8s的api与apiserver进行交互，最后成功创建或更新istio正常运行所需的所有资源。

Manifest

Manifest命令基本与Install命令的流程类似，只是通过helm渲染后不需要与apiserver交互并创建资源，因此不再赘述。