k8s中查看默认调度器开启的插件以及权重的含义

头像
ning1875
     北京
    阅读 17 分钟
    头图

    要点01：软件配置项的来源主要有2块：命令行参数 和 配置文件

    • k8s中的组件一般都是遵循上面的模式
    • 我们以kube-scheduler为例
    • 命令参数举例 ：可以看到--xxx=xxx的传参，当然所有参数都是有默认值的，如果你不传就走默认
    /usr/local/bin/kube-scheduler --log-dir=/var/log/k8s --logtostderr=false --alsologtostderr=true --config=/etc/k8s/kube-scheduler.yaml --kube-api-qps=500 --kube-api-burst=500 --authentication-kubeconfig=/etc/k8s/scheduler.conf --authorization-kubeconfig=/etc/k8s/scheduler.conf --kubeconfig=/etc/k8s/scheduler.conf --leader-elect=true --v=2
    • 上面的命令行参数中可以看到--config=xxx.yaml指定了 配置文件的路径
    • 比如我们可以查看 这个配置文件的内容 ：发现基本就是官方的默认配置
    cat /etc/k8s/kube-scheduler.yaml

apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: "/etc/k8s/scheduler.conf"
leaderElection:
  leaderElect: true
metricsBindAddress: 0.0.0.0:10251

    要点02：k8s组件提供了一个应用运行时查看生效配置的接口

    • 思考为何要提供？主要原因有下面几点

      • 来自命令行和配置文件的配置2块可能有些覆盖的地方
      • 配置项目太多了

      • 在没有配置热更新的情况下：查看变更是否生效：

        • 配置文件已经更改，但忘记是应用重启前还是重启后改的了
    • 所以讲究的项目里面都会留有一个http接口
    • 直接将当前应用内存中生效的配置项目打印出来，方便排查问题
    • 所以我们后面写golang的项目也可以仿照这个接口

    要点03 : 如何请求scheduler的配置接口

    k8s组件对应查看配置的接口就是configz

    访问组件接口需要鉴权，我们可以通过sa来实现

    如何在1.24集群中创建rbac

    apiVersion: rbac.authorization.k8s.io/v1 # api的version
kind: ClusterRole # 类型
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources: # 资源
  - nodes
  - nodes/metrics
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"] 
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus # 自定义名字
  namespace: kube-system # 命名空间
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef: # 选择需要绑定的Role
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects: # 对象
- kind: ServiceAccount
  name: prometheus
  namespace: kube-system

---

apiVersion: v1
kind: Secret
metadata:
  namespace: kube-system
  name: prometheus
  annotations:
    kubernetes.io/service-account.name: prometheus
type: kubernetes.io/service-account-token

    获取token命令

    root@k8s-master01:~# TOKEN=$(kubectl -n kube-system  get secret prometheus -o jsonpath='{.data.token}'| base64 --decode )
root@k8s-master01:~# echo $TOKEN
eyJhbGciOiJSUzI1NiIsImtpZCI6ImFVMS1mYlhobWIxcF92djBwbUIxZDhTVlFWd0VNa3VpNDlmOUhqcG9qSlkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InByb21ldGhldXMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI2OTg2NWUwYy0yOGE4LTQ3YTEtYWEzYy03NThmNDlkYjA1YWUiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06cHJvbWV0aGV1cyJ9.SlORtZtVvLptYWE3zvblaOMxHNMBrHVTTHra7fO1RrwxdK3Bzc42ETLvkzVfQmAQlWrq5yiB4HiKFLe4qY2KVwK3qLDS_sWADLI16Sv8-O1Dt0oOQ0UZD0VOSGY0XEq2EGUxgxnx_JWllgEuMd0rjxtAtyjFh9wjCo_07lFCj44BffGGFp6Kovd8Dl_CJKpORakaJW-haIvTmTlbFPbRKojRTyKvtNCVn0zIXsz8Esp7z9XZmtUvZmHqNlY7bAFtGM9qLWUY_PkM1C0lQ2ZDKASdhZpx6LJr1Wo4WSNILCVfECT0sd6TnFHbgd1NwBc0kcTct5VbST76AJwUpG5esA

    在http请求中把上面的TOKEN 作为beare-token传过去就可以了

    请求的curl命令如下(在kube-scheduler部署的节点上执行，通常是master)

    curl -k -s  https://localhost:10259/configz --header "Authorization: Bearer $TOKEN"    |python -m json.tool

    要点04 根据返回的json配置段来分析插件开启的情况

    • json数据如下
    {
    "componentconfig": {
        "AlgorithmSource": {
            "Policy": null,
            "Provider": "DefaultProvider"
        },
        "ClientConnection": {
            "AcceptContentTypes": "",
            "Burst": 100,
            "ContentType": "application/vnd.kubernetes.protobuf",
            "Kubeconfig": "/etc/k8s/scheduler.conf",
            "QPS": 50
        },
        "EnableContentionProfiling": true,
        "EnableProfiling": true,
        "Extenders": null,
        "HealthzBindAddress": "0.0.0.0:10251",
        "LeaderElection": {
            "LeaderElect": true,
            "LeaseDuration": "15s",
            "RenewDeadline": "10s",
            "ResourceLock": "leases",
            "ResourceName": "kube-scheduler",
            "ResourceNamespace": "kube-system",
            "RetryPeriod": "2s"
        },
        "MetricsBindAddress": "0.0.0.0:10251",
        "Parallelism": 16,
        "PercentageOfNodesToScore": 0,
        "PodInitialBackoffSeconds": 1,
        "PodMaxBackoffSeconds": 10,
        "Profiles": [
            {
                "PluginConfig": [
                    {
                        "Args": {
                            "MinCandidateNodesAbsolute": 100,
                            "MinCandidateNodesPercentage": 10
                        },
                        "Name": "DefaultPreemption"
                    },
                    {
                        "Args": {
                            "HardPodAffinityWeight": 1
                        },
                        "Name": "InterPodAffinity"
                    },
                    {
                        "Args": {
                            "AddedAffinity": null
                        },
                        "Name": "NodeAffinity"
                    },
                    {
                        "Args": {
                            "IgnoredResourceGroups": null,
                            "IgnoredResources": null
                        },
                        "Name": "NodeResourcesFit"
                    },
                    {
                        "Args": {
                            "Resources": [
                                {
                                    "Name": "cpu",
                                    "Weight": 1
                                },
                                {
                                    "Name": "memory",
                                    "Weight": 1
                                }
                            ]
                        },
                        "Name": "NodeResourcesLeastAllocated"
                    },
                    {
                        "Args": {
                            "DefaultConstraints": null,
                            "DefaultingType": "System"
                        },
                        "Name": "PodTopologySpread"
                    },
                    {
                        "Args": {
                            "BindTimeoutSeconds": 600
                        },
                        "Name": "VolumeBinding"
                    }
                ],
                "Plugins": {
                    "Bind": {
                        "Disabled": null,
                        "Enabled": [
                            {
                                "Name": "DefaultBinder",
                                "Weight": 0
                            }
                        ]
                    },
                    "Filter": {
                        "Disabled": null,
                        "Enabled": [
                            {
                                "Name": "NodeUnschedulable",
                                "Weight": 0
                            },
                            {
                                "Name": "NodeName",
                                "Weight": 0
                            },
                            {
                                "Name": "TaintToleration",
                                "Weight": 0
                            },
                            {
                                "Name": "NodeAffinity",
                                "Weight": 0
                            },
                            {
                                "Name": "NodePorts",
                                "Weight": 0
                            },
                            {
                                "Name": "NodeResourcesFit",
                                "Weight": 0
                            },
                            {
                                "Name": "VolumeRestrictions",
                                "Weight": 0
                            },
                            {
                                "Name": "EBSLimits",
                                "Weight": 0
                            },
                            {
                                "Name": "GCEPDLimits",
                                "Weight": 0
                            },
                            {
                                "Name": "NodeVolumeLimits",
                                "Weight": 0
                            },
                            {
                                "Name": "AzureDiskLimits",
                                "Weight": 0
                            },
                            {
                                "Name": "VolumeBinding",
                                "Weight": 0
                            },
                            {
                                "Name": "VolumeZone",
                                "Weight": 0
                            },
                            {
                                "Name": "PodTopologySpread",
                                "Weight": 0
                            },
                            {
                                "Name": "InterPodAffinity",
                                "Weight": 0
                            }
                        ]
                    },
                    "Permit": {
                        "Disabled": null,
                        "Enabled": null
                    },
                    "PostBind": {
                        "Disabled": null,
                        "Enabled": null
                    },
                    "PostFilter": {
                        "Disabled": null,
                        "Enabled": [
                            {
                                "Name": "DefaultPreemption",
                                "Weight": 0
                            }
                        ]
                    },
                    "PreBind": {
                        "Disabled": null,
                        "Enabled": [
                            {
                                "Name": "VolumeBinding",
                                "Weight": 0
                            }
                        ]
                    },
                    "PreFilter": {
                        "Disabled": null,
                        "Enabled": [
                            {
                                "Name": "NodeResourcesFit",
                                "Weight": 0
                            },
                            {
                                "Name": "NodePorts",
                                "Weight": 0
                            },
                            {
                                "Name": "PodTopologySpread",
                                "Weight": 0
                            },
                            {
                                "Name": "InterPodAffinity",
                                "Weight": 0
                            },
                            {
                                "Name": "VolumeBinding",
                                "Weight": 0
                            }
                        ]
                    },
                    "PreScore": {
                        "Disabled": null,
                        "Enabled": [
                            {
                                "Name": "InterPodAffinity",
                                "Weight": 0
                            },
                            {
                                "Name": "PodTopologySpread",
                                "Weight": 0
                            },
                            {
                                "Name": "TaintToleration",
                                "Weight": 0
                            }
                        ]
                    },
                    "QueueSort": {
                        "Disabled": null,
                        "Enabled": [
                            {
                                "Name": "PrioritySort",
                                "Weight": 0
                            }
                        ]
                    },
                    "Reserve": {
                        "Disabled": null,
                        "Enabled": [
                            {
                                "Name": "VolumeBinding",
                                "Weight": 0
                            }
                        ]
                    },
                    "Score": {
                        "Disabled": null,
                        "Enabled": [
                            {
                                "Name": "NodeResourcesBalancedAllocation",
                                "Weight": 1
                            },
                            {
                                "Name": "ImageLocality",
                                "Weight": 1
                            },
                            {
                                "Name": "InterPodAffinity",
                                "Weight": 1
                            },
                            {
                                "Name": "NodeResourcesLeastAllocated",
                                "Weight": 1
                            },
                            {
                                "Name": "NodeAffinity",
                                "Weight": 1
                            },
                            {
                                "Name": "NodePreferAvoidPods",
                                "Weight": 10000
                            },
                            {
                                "Name": "PodTopologySpread",
                                "Weight": 2
                            },
                            {
                                "Name": "TaintToleration",
                                "Weight": 1
                            }
                        ]
                    }
                },
                "SchedulerName": "default-scheduler"
            }
        ]
    }
}

    前置知识 k8s的调度框架 scheduler framework

    我们发现在 componentconfig.Profiles其实主要分两块：

    • Plugins段代表调度框架每个扩展点的插件开启和禁用的情况：

      • 可以理解为在各个阶段：需要依次执行开启列表中的插件，不执行禁用列表中的插件
    • 比如在Filter阶段(我节选了一部分) 都是咱们经常能看到的插件：去掉NodeUnschedulable 、过滤NodeName、过滤TaintToleration 还有根据request资源情况的NodeResourcesFit
     "Filter": {
                        "Disabled": null,
                        "Enabled": [
                            {
                                "Name": "NodeUnschedulable",
                                "Weight": 0
                            },
                            {
                                "Name": "NodeName",
                                "Weight": 0
                            },
                            {
                                "Name": "TaintToleration",
                                "Weight": 0
                            },
                            {
                                "Name": "NodeAffinity",
                                "Weight": 0
                            },
                            {
                                "Name": "NodePorts",
                                "Weight": 0
                            },
                            {
                                "Name": "NodeResourcesFit",
                                "Weight": 0
                            },
                        ]
                    },
    • PluginConfig段代表各个插件的配置字段：这里是个map 的k-v形式：很好理解就是每个插件的配置字段不一样，直接用map最方便
    • 比如 NodeResourcesLeastAllocated插件
                        {
                        "Args": {
                            "Resources": [
                                {
                                    "Name": "cpu",
                                    "Weight": 1
                                },
                                {
                                    "Name": "memory",
                                    "Weight": 1
                                }
                            ]
                        },
                        "Name": "NodeResourcesLeastAllocated"
                    },

    要点6：关于插件权重的讨论

    • 在上面的配置中可以看到很多 阶段如Filter和Score 的插件中都有权重的配置
                                {
                                "Name": "NodePreferAvoidPods",
                                "Weight": 10000
                            },
    • 但是只有在Score中的插件的Weight值才大于0
    • 其余阶段的插件的Weight值都是0

    结合代码看看 ：找到 Plugin.Weight (1.22分支)

    • 位置 D:\go_path\src\github.com\kubernetes\kubernetes\pkg\scheduler\apis\config\types.go
    
// Plugin specifies a plugin name and its weight when applicable. Weight is used only for Score plugins.
type Plugin struct {
    // Name defines the name of plugin
    Name string
    // Weight defines the weight of plugin, only used for Score plugins.
    Weight int32
}
    • 我们发现Plugin结构体就是一个名字和权重
    • 并且从注释中明确的看到了权重只有在 Score阶段的插件中才有作用

    小乙老师关于k8s调度器源码文章推荐

    那么基于真实负载调度的调度器该怎么编写呢 ?

    kubernetesgo运维源码分析
    ning1875
    167 声望67 粉丝

    k8s/prometheus/cicd运维开发专家，想进阶的dy搜 小乙运维杂货铺

    « 上一篇
    k8s默认调度器关于pod申请资源过滤的源码细节
    下一篇 »
    从运维小白成长为运维开发专家的修炼之路

    引用和评论

    推荐阅读
    头像
    k8s中的11运维开发方向是哪些 如果快速提升k8s开发能力

    ning1875赞 1阅读 421

    头像
    Go 语言-计算密集型服务 性能优化

    Jesse赞 4阅读 5k

    头像
    IO 密集型服务 耗时优化

    Jesse赞 4阅读 4.2k

    头像
    Go 语言 JSON 与 Cache 库 调研与选型

    Jesse赞 2阅读 3.9k

    头像
    高质量代码的原则

    寒青赞 4阅读 5.3k

    头像
    CentOS7 Docker安装Gitlab

    YYGP阅读 6.6k

    头像
    Go 语言-内存泄漏排查两例

    Jesse赞 1阅读 1.9k

    0 条评论
    得票最新