kubernetes源码调试：debug apiserver

在远程机器上启动集群，然后在本地机器上，通过IDE远程调试kubernetes的代码。

以apiserver为例。

一.环境准备(远程机器)

1. 环境

• OS: centos7.6
• kubernetes源码：v1.20
• golang: go1.17.12
• dlv: 1.9.1

2. 安装golang

参考：https://docs.studygolang.com/doc/install

3. 安装dlv

参考：https://github.com/go-delve/delve/tree/master/Documentation/installation

# go install github.com/go-delve/delve/cmd/dlv@latest

4. 下载kubernetes源码

git clone -b release-1.20 [e-mail]:kubernetes/kubernetes.git

二.修改编译参数（远程机器）

进入到kubernetes的源码，修改hack/lib/golang.sh文件，从而让编译器不去优化掉debug的信息，主要包括：

• 禁止-w -s，保留文件名、行号；
• 加上-gcflags="all=-N -l"，禁止优化和内联；

修改点：

kube::golang::build_binaries() {
  # Create a sub-shell so that we don't pollute the outer environment
  (
    # Check for `go` binary and set ${GOPATH}.
    kube::golang::setup_env
    V=2 kube::log::info "Go version: $(GOFLAGS='' go version)"

    local host_platform
    host_platform=$(kube::golang::host_platform)

    local goflags goldflags goasmflags gogcflags gotags
    # If GOLDFLAGS is unset, then set it to the a default of "-s -w".
    # Disable SC2153 for this, as it will throw a warning that the local
    # variable goldflags will exist, and it suggest changing it to this.
    # shellcheck disable=SC2153
    # goldflags="${GOLDFLAGS=-s -w -buildid=} $(kube::version::ldflags)"
    goldflags="${GOLDFLAGS=-buildid=} $(kube::version::ldflags)"        // 去掉-s -w
    goasmflags="-trimpath=${KUBE_ROOT}"
    gogcflags="${GOGCFLAGS:-} -trimpath=${KUBE_ROOT}"
    gogcflags="${gogcflags} -N -l"                                    // 增加-N -l

    # extract tags if any specified in GOFLAGS
    # shellcheck disable=SC2001
    gotags="selinux,notest,$(echo "${GOFLAGS:-}" | sed -e 's|.*-tags=\([^-]*\).*|\1|')"

    local -a targets=()
    local arg
    ...
}

三. 安装etcd（远程机器）

进入kubernetes的源码，执行以下脚本，安装etcd:

# ./hack/install-etcd.sh
Downloading https://github.com/coreos/etcd/releases/download/v3.4.13/etcd-v3.4.13-linux-amd64.tar.gz succeed
etcd v3.4.13 installed. To use:
export PATH="/root/go/src/github.com/kubernetes/kubernetes/third_party/etcd:${PATH}"

四. 启动本地集群（远程机器）

准备工作完成后，执行下面的脚本，启动本地集群：

# ./hack/local-up-cluster.sh

检查kube进程是否启动：

# ps -a|grep kube
18999 pts/0    00:26:51 kube-controller
19001 pts/0    00:02:55 kube-scheduler
19162 pts/0    00:26:35 kubelet
19929 pts/0    00:00:42 kube-proxy
89307 pts/2    00:00:07 kube-apiserver

kubectl访问本地集群：

# export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
# cluster/kubectl.sh get ns 
NAME              STATUS   AGE
default           Active   45m
kube-node-lease   Active   45m
kube-public       Active   45m
kube-system       Active   45m

五. dlv启动kube-apiserver（远程机器）

查看kube-apiserver的启动参数：

# ps -ef|grep kube-apiserver
root      89307  30528 12 09:37 pts/2    00:00:17 /root/go/src/github.com/kubernetes/kubernetes/_output/local/bin/linux/amd64/kube-apiserver --authorization-mode=Node,RBAC --cloud-provider= --cloud-config= --v=3 --vmodule= --audit-policy-file=/tmp/kube-audit-policy-file --audit-log-path=/tmp/kube-apiserver-audit.log --authorization-webhook-config-file= --authentication-token-webhook-config-file= --cert-dir=/var/run/kubernetes --client-ca-file=/var/run/kubernetes/client-ca.crt --kubelet-client-certificate=/var/run/kubernetes/client-kube-apiserver.crt --kubelet-client-key=/var/run/kubernetes/client-kube-apiserver.key --service-account-key-file=/tmp/kube-serviceaccount.key --service-account-lookup=true --service-account-issuer=https://kubernetes.default.svc --service-account-signing-key-file=/tmp/kube-serviceaccount.key --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,Priority,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --disable-admission-plugins= --admission-control-config-file= --bind-address=0.0.0.0 --secure-port=6443 --tls-cert-file=/var/run/kubernetes/serving kube-apiserver.crt --tls-private-key-file=/var/run/kubernetes/serving-kube-apiserver.key --storage-backend=etcd3 --storage-media-type=application/vnd.kubernetes.protobuf --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.0.0.0/24 --feature-gates=AllAlpha=false --external-hostname=localhost --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-client-ca-file=/var/run/kubernetes/request-header-ca.crt --requestheader-allowed-names=system:auth-proxy --proxy-client-cert-file=/var/run/kubernetes/client-auth-proxy.crt --proxy-client-key-file=/var/run/kubernetes/client-auth-proxy.key --cors-allowed-origins=/127.0.0.1(:[0-9]+)?$,/localhost(:[0-9]+)?$

杀掉kube-apiserver进程，使用dlv启动：

• dlv监听时，使用12345端口；
• 可以让Goland远程连接该端口，进行远程debug；

# kill -9 89307

## 1. dlv --headless exec启动apiserver进程
## 2. 进程参数中，增加--listen=0.0.0.0:12345参数
# dlv --headless exec  /root/go/src/github.com/kubernetes/kubernetes/_output/local/bin/linux/amd64/kube-apiserver --listen=0.0.0.0:12345 --api-version=2 --log  --  --authorization-mode=Node,RBAC  --cloud-provider= --cloud-config=   --v=3 --vmodule= --audit-policy-file=/tmp/kube-audit-policy-file --audit-log-path=/tmp/kube-apiserver-audit.log --authorization-webhook-config-file= --authentication-token-webhook-config-file= --cert-dir=/var/run/kubernetes --client-ca-file=/var/run/kubernetes/client-ca.crt --kubelet-client-certificate=/var/run/kubernetes/client-kube-apiserver.crt --kubelet-client-key=/var/run/kubernetes/client-kube-apiserver.key --service-account-key-file=/tmp/kube-serviceaccount.key --service-account-lookup=true --service-account-issuer=https://kubernetes.default.svc --service-account-signing-key-file=/tmp/kube-serviceaccount.key --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,Priority,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --disable-admission-plugins= --admission-control-config-file= --bind-address=0.0.0.0 --secure-port=6443 --tls-cert-file=/var/run/kubernetes/serving-kube-apiserver.crt --tls-private-key-file=/var/run/kubernetes/serving-kube-apiserver.key --storage-backend=etcd3 --storage-media-type=application/vnd.kubernetes.protobuf --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.0.0.0/24 --feature-gates=AllAlpha=false --external-hostname=localhost --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-client-ca-file=/var/run/kubernetes/request-header-ca.crt --requestheader-allowed-names=system:auth-proxy --proxy-client-cert-file=/var/run/kubernetes/client-auth-proxy.crt --proxy-client-key-file=/var/run/kubernetes/client-auth-proxy.key --cors-allowed-origins=/127.0.0.1\(:[0-9]+\)?$,/localhost\(:[0-9]+\)?$

六. Goland远程调试（本地机器）

Goland上配置远程Debug的ip和端口:

Goland使用上面的配置，启动Debug: