HPA使用自定义指标进行扩缩容

kubernetes引入HPA进行扩缩容，HPA可以根据指标(metrics)，自动对workload进行弹性的扩缩容。

HPA资源在autoscaling这个group下面，HPA分为v1和v2版本：

• HPA v1仅支持基于CPU指标的伸缩；
• HPA v2beta1引入了基于Memory和自定义指标的伸缩；
• HPA v2beta2引入了自定义策略(behavior)的伸缩；
• HPA v2将v2beta2变成GA功能；

# kubectl api-versions|grep auto
autoscaling/v1
autoscaling/v2beta1
autoscaling/v2beta2
autoscaling/v2

一.HPA的指标类型

• Resouce: 提供Pod级别的CPU/Memory使用量；

  • 通过metrics-server提供指标，调用metrics.k8s.io；

• ContainerResource: 提供Container级别的CPU/Memory使用量；

  • 通过metrics-server提供指标，调用metrics.k8s.io；

• Object: 提供k8s集群内任意资源的相关指标；

  • 通过prometheus-adapter提供指标，调用custom.metrics.k8s.io；

• Pods: 提供k8s集群内任意资源的相关指标；

  • 通过prometheus-adapter提供指标，调用custom.metrics.k8s.io；
• External: 提供k8s集群外的指标数据；

二.HPA自定义指标的伸缩

1. 部署prometheus-adapter

自定义指标由prometheus-adapter查询prometheus得到，所以需要部署prometheus和prometheus-adatper组件。

这里使用helm chart的方式部署：

• prometheus: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
• prometheus-adatper: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-adapter

部署完成以后：

# kubectl get pod -n monitoring
NAME                                  READY   STATUS    RESTARTS   AGE
alertmanager-main-0                   2/2     Running   0          4d20h
kube-state-metrics-796659d8b6-vr5w8   3/3     Running   0          4d20h
node-exporter-rn9nj                   2/2     Running   0          4d20h
prometheus-adapter-5f4df978c8-4fxjq   1/1     Running   0          46s
prometheus-k8s-0                      3/3     Running   1          4d20h
prometheus-operator-7bc7bd7d7-jp754   2/2     Running   0          4d20h

2. 创建deploy和service

deploy: 初始为1个副本

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  labels:
    app: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - image: luxas/autoscale-demo:v0.1.2
        name: metrics-provider
        ports:
        - name: http
          containerPort: 8080

service:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: sample-app
  name: sample-app
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: sample-app
  type: ClusterIP

该sample-app仅暴露了一个指标：http_requests_total:

# curl http://$(kubectl get service sample-app -o jsonpath='{ .spec.clusterIP }')/metrics
# HELP http_requests_total The amount of requests served by the server in total
# TYPE http_requests_total counter
http_requests_total 6179

3. 创建serviceMonitor

创建serviceMonitor，以便让prometheus发现sample-app，拉到它的metrics:

kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
  name: sample-app
  labels:
    app: sample-app
spec:
  selector:
    matchLabels:
      app: sample-app
  endpoints:
  - port: http

4. 创建prometheusRule自定义指标

通过prometheusRule创建自定义指标：

• 创建了自定义指标：sample_app_http_requests_rate = rate(http_requests_total[5m])

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: k8s
    role: alert-rules
  name: sample-app-rules
  namespace: monitoring
spec:
  groups:
    - name: sample-app.expr.rules
      rules:
        - expr: |
            rate(http_requests_total[5m])
          record: sample_app_http_requests_rate
          labels:
            namespace: default
            service: sample-app

可以通过kubectl查询自定义指标：

# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/sample_app_http_requests_rate"|python -m json.tool
{
    "apiVersion": "custom.metrics.k8s.io/v1beta1",
    "items": [
        {
            "describedObject": {
                "apiVersion": "/v1",
                "kind": "Pod",
                "name": "sample-app-7cfb596f98-jmswq",
                "namespace": "default"
            },
            "metricName": "sample_app_http_requests_rate",
            "selector": null,
            "value": "33m"
        }
    ],
    "kind": "MetricValueList",
    "metadata": {
        "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/sample_app_http_requests_rate"
    }
}

5. 创建HPA

使用 sample_app_http_requests_rate 指标，创建HPA：

kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
  name: sample-app
spec:
  scaleTargetRef:
    # point the HPA at the sample application
    # you created above
    apiVersion: apps/v1
    kind: Deployment
    name: sample-app
  # autoscale between 1 and 10 replicas
  minReplicas: 1
  maxReplicas: 10
  metrics:
  # use a "Pods" metric, which takes the average of the
  # given metric across all pods controlled by the autoscaling target
  - type: Pods
    pods:
      # use the metric that you used above: pods/sample_app_http_requests_rate
      metricName: sample_app_http_requests_rate
      # target 500 milli-requests per second,
      # which is 1 request every two seconds
      targetAverageValue: 500m

6. 加大流量观察HPA

通过curl命令，加大流量：

# while true; do curl http://$(kubectl get service sample-app -o jsonpath='{ .spec.clusterIP }')/metrics; sleep 0.1; done

等待一会后，观察hpa对象，可以看到deploy已扩容为5个副本：

# kubectl get hpa sample-app
NAME         REFERENCE               TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
sample-app   Deployment/sample-app   697m/500m   1         10        5          45h

再次通过kubectl 查询自定义指标：

# kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/sample_app_http_requests_rate?selector=app%3Dsample-app | python -m json.tool
{
    "apiVersion": "custom.metrics.k8s.io/v1beta1",
    "items": [
        {
            "describedObject": {
                "apiVersion": "/v1",
                "kind": "Pod",
                "name": "sample-app-7cfb596f98-9vkrb",
                "namespace": "default"
            },
            "metricName": "sample_app_http_requests_rate",
            "selector": null,
            "value": "263m"
        },
        {
            "describedObject": {
                "apiVersion": "/v1",
                "kind": "Pod",
                "name": "sample-app-7cfb596f98-f4dqr",
                "namespace": "default"
            },
            "metricName": "sample_app_http_requests_rate",
            "selector": null,
            "value": "553m"
        },
        {
            "describedObject": {
                "apiVersion": "/v1",
                "kind": "Pod",
                "name": "sample-app-7cfb596f98-jmswq",
                "namespace": "default"
            },
            "metricName": "sample_app_http_requests_rate",
            "selector": null,
            "value": "1740m"
        },
        {
            "describedObject": {
                "apiVersion": "/v1",
                "kind": "Pod",
                "name": "sample-app-7cfb596f98-qg24g",
                "namespace": "default"
            },
            "metricName": "sample_app_http_requests_rate",
            "selector": null,
            "value": "234m"
        }
    ],
    "kind": "MetricValueList",
    "metadata": {
        "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/sample_app_http_requests_rate"
    }
}

参考：

1.github-doc:https://github.com/arielb135/HPA-with-prometheus-and-RabbitMQ/blob/master/README.md
2.HPA指标类型：https://mp.weixin.qq.com/s/ul2G1WFPmHo5QR0xI143jg