KubeCube user management and identity authentication

Preface
KubeCube ( https://kubecube.io) is a lightweight enterprise-level container platform recently open sourced by NetEase Shufan, which provides enterprises with visualized management of kubernetes resources and unified multi-cluster and multi-tenant management functions. The KubeCube community will interpret the design features and technical implementation of KubeCube through a series of technical articles to help developers and users understand and use KubeCube more quickly. This article is the third one, focusing on the implementation of user management and identity authentication in KubeCube.

User Management

所有 Kubernetes 集群都有两类用户：由 Kubernetes 管理的服务账号和普通用户。

Kubernetes 假定普通用户是由一个与集群无关的服务通过以下方式之一进行管理的：


* 负责分发私钥的管理员

* 类似 Keystone 或者 Google Accounts 这类用户数据库

* 包含用户名和密码列表的文件

有鉴于此，Kubernetes 并不包含用来代表普通用户账号的对象。 普通用户的信息无法通过 API 调用添加到集群中。

According to the Kubernetes official statement, Kubernetes itself does not directly provide user management features, does not support ordinary user objects, and does not store any information about ordinary users. If you need to create a user, you need to create a private key and certificate for the user, and use the certificate for identity authentication. In addition, because user information is not stored, the cluster administrator cannot centrally manage users and has no awareness of other users. Therefore, KubeCube first redefines the concept of user, that is, it provides the resource type User, stores user information, performs user management, and facilitates subsequent identity authentication and permission verification.

apiVersion: user.kubecube.io/v1
kind: User
metadata:
    name: 登录账号，用户唯一标识，用户自定义，不可重复，不可修改
spec:
    password: 密码，必填，系统会将密码进行md5加盐加密后保存
    displayName: 用户名
    email: 邮箱
    phone: 电话
    language: 语言：en/ch
    loginType: 用户登录方式：normal/ldap/github/...
    state: 用户状态：normal/forbidden
status:
    lastLoginTime: 上次登录时间
    lastLoginIp: 上次登录IP

Users can be created manually by the administrator on the front-end page, or automatically created by the system when they log in for the first time using external authentication. Therefore, users can be divided into ordinary registered users of the system and third-party authorized login users in terms of registration methods. However, for these two creation methods, the corresponding User cr is created in the control cluster. Then Warden's resource synchronization manager will synchronize the cr from the control cluster to the computing cluster to facilitate subsequent multi-cluster unified authentication.

In this way, on the user management page, you only need to query the User resources in the control cluster to achieve centralized user management. And, you can easily add users, query users, and modify user meta-information.

Identity authentication

In KubeCube, local authentication and external authentication are supported. Local authentication means that a normal user is created in KubeCube, and the user uses the username and password registered when the user was created to log in and authenticate. External authentication means that there is no need to create a user, and the user's identity is authenticated through a third-party authentication platform to access KubeCube. The implementation of these two authentication methods will be introduced separately below.

Local authentication

In KubeCube, user identity authentication is mainly performed through JWT (JSON Web Token).

When using local authentication to log in, the user needs to enter a user name and password. KubeCube will query User cr in the cluster based on the user name and compare the passwords. If the User is queried and the passwords are consistent, the login is considered successful. After KubeCube updates the user's login status, it will generate a JWT based on the user name and stitch it into a Bearer Token, which will be stored in a cookie and returned.

The code after successfully verifying the user name and password when the user logs in is as follows:

  // generate token and return
    authJwtImpl := jwt.GetAuthJwtImpl()
    token, err := authJwtImpl.GenerateToken(&v1beta1.UserInfo{Username: name})
    if err != nil {
        response.FailReturn(c, errcode.AuthenticateError)
        return
    }
    bearerToken := jwt.BearerTokenPrefix + " " + token
    c.SetCookie(constants.AuthorizationHeader, bearerToken, int(authJwtImpl.TokenExpireDuration), "/", "", false, true)

    user.Spec.Password = ""
    response.SuccessReturn(c, user)
    return

After the user successfully logs in, for each subsequent request, the front end will bring the JWT request through the cookie, and the back-end authentication middleware will then verify the JWT. If it is valid, a new token will be generated and returned, and the above process will be repeated. In this way, even if the default valid time of the JWT generated in KubeCube is 1 hour, as long as the user continues to access, the JWT will be constantly refreshed so that the user is always logged in.

Part of the code of the authentication middleware is as follows:

func Auth() gin.HandlerFunc {
    return func(c *gin.Context) {
        if !withinWhiteList(c.Request.URL, c.Request.Method, whiteList) {
            authJwtImpl := jwt.GetAuthJwtImpl()
      userToken, err := token.GetTokenFromReq(c.Request)
      if err != nil {
        response.FailReturn(c, errcode.AuthenticateError)
        return
      }

      newToken, respInfo := authJwtImpl.RefreshToken(userToken)
      if respInfo != nil {
        response.FailReturn(c, errcode.AuthenticateError)
        return
      }

      v := jwt.BearerTokenPrefix + " " + newToken

      c.Request.Header.Set(constants.AuthorizationHeader, v)
      c.SetCookie(constants.AuthorizationHeader, v, int(authJwtImpl.TokenExpireDuration), "/", "", false, true)
            c.Next()
        }
    }
}

External authentication

The realization of external authentication is currently mainly divided into three types, namely general authentication, LDAP authentication and OAuth2 authentication.

General Certification

In order to facilitate users to connect to a set of their own authentication system, KubeCube supports a general authentication method. The user can enable the universal authentication method and configure the address of the authentication system so that every time the user visits KubeCube, he will go to his own authentication system for authentication. After the authentication is passed, the user name of the user needs to be returned to KubeCube. KubeCube will still generate the corresponding Bearer Token according to the user name and put it in the header for subsequent permission verification. The main logic code is implemented as follows:

func Auth() gin.HandlerFunc {
    return func(c *gin.Context) {
        if !withinWhiteList(c.Request.URL, c.Request.Method, whiteList) {
            authJwtImpl := jwt.GetAuthJwtImpl()
            if generic.Config.GenericAuthIsEnable {
                h := generic.GetProvider()
                user, err := h.Authenticate(c.Request.Header)
                if err != nil || user == nil {
                    clog.Error("generic auth error: %v", err)
                    response.FailReturn(c, errcode.AuthenticateError)
                    return
                }
                newToken, error := authJwtImpl.GenerateToken(&v1beta1.UserInfo{Username: user.GetUserName()})
                if error != nil {
                    response.FailReturn(c, errcode.AuthenticateError)
                    return
                }
                b := jwt.BearerTokenPrefix + " " + newToken
                c.Request.Header.Set(constants.AuthorizationHeader, b)
            }
            c.Next()
        }
    }
}

LDAP authentication

1. When the user selects the LDAP login method, the user enters the user name and password. First, it checks whether the user exists in the cluster and whether the user is in the "disabled" state. If it does not exist or exists and is normal, then LDAP authentication starts
2. As an LDAP client, KubeCube obtains the user name and password of the user, and uses the administrator DN and administrator password as parameters to send an administrator bind request message to the LDAP server to obtain query permissions.
3. After receiving the administrator bind request message, the LDAP server verifies whether the administrator DN and administrator password are correct. If the administrator DN and administrator password are correct, the administrator binding response message that the binding is successful is sent to KubeCube.
4. After KubeCube receives the binding response message, it constructs the filter condition with the user name entered by the user as a parameter, and sends a user DN query request message to the LDAP server. For example: the filter condition is CN=User2.
5. After receiving the user DN query request message, the LDAP server searches the user DN according to the query starting point, query range, and filter conditions in the message. If the query is successful, a response message indicating that the query is successful is sent to KubeCube. There can be one or more user DNs obtained by the query. If the obtained user is not one, it is considered that the user name or password is wrong, and the authentication fails.
6. KubeCube sends a user binding request message to the LDAP server according to the user DN obtained by the query and the password entered by the user as parameters.
7. After receiving the user binding request message, the LDAP server checks whether the password entered by the user is correct.

• If the password entered by the user is correct, a binding response message indicating successful binding is sent to KubeCube.
• If the password entered by the user is incorrect, a response message indicating binding failure is sent to KubeCube. KubeCube takes the next user DN found as a parameter, and continues to send bind requests to the LDAP server until a DN is successfully bound. If all user DNs fail to bind, KubeCube notifies the user that authentication has failed.

After the authentication is successful, the logic of the local authentication is the same: if the user does not exist in the cluster, the User cr is created according to the user name; and the corresponding Bearer Token is generated according to the user name and stored in the Cookie, which will be carried in the next request for identification user ID.

OAuth2 authentication

In KubeCube, OAuth2 authentication adopts the authorization code mode, because this mode is the authorization mode with the most complete functions and the most rigorous process. The usual OAuth2 authentication process is:

1. The user accesses the client, and the latter directs the former to the authentication server.
2. The user chooses whether to authorize the client.
3. Assuming that the user is authorized, the authentication server directs the user to the "redirection URI" specified by the client in advance, and attaches an authorization code at the same time.
4. The client receives the authorization code, attaches the earlier "redirect URI", and applies for a token from the authentication server. This step is done on the server in the background of the client and is not visible to the user.
5. The authentication server checks the authorization code and redirection URI, and after confirming that they are correct, sends an access token and refresh token to the client.

In the implementation of KubeCube, take GitHub login as an example:

The user chooses to log in with GitHub authentication when logging in, and the front-end forwards the request to GitHub;
GitHub asks whether the user agrees to authorize KubeCube;
If the user agrees, GitHub will redirect back to KubeCube (/oauth/redirect) and send back an authorization code (code);
KubeCube uses the authorization code to request a token (access_token) from GitHub;
GitHub return token (access_token);
KubeCube uses the token (access_token) to request user information data from GitHub;
Query the cluster, if the user does not exist, create User cr based on the user information;
Generate Bearer Token to access the cluster based on the user name, and return the authentication success;
The front end stores the Bearer Token in the Cookie, and carries it in the next request.
OpenAPI certification
Based on the above design scheme, it can be easily inferred that the authentication realization of OpenAPI is also completed through JWT. User is bound to each group of AK and SK, and the corresponding User is queried through AK and SK, and then Bearer Token is generated and returned by the User.Name. In the next request, the user needs to carry the Token in the Cookie or header, and the KubeCube authentication middleware can parse out the user's identity through the Token to complete the authentication.

Cluster certification
After the middleware completes the identity authentication, it will refresh the token, but if the token is directly carried in the request header to request kube-apiserver to complete the cluster authentication, you need to modify the authentication backend of kube-apiserver when KubeCube is deployed, that is, modify kube-apiserver Configuration. This will cause intrusion to the native kubernetes cluster and greatly increase the deployment cost and operation and maintenance cost of KubeCube. Therefore, we need to build another module to help complete the cluster authentication-auth-proxy.

When users request kubernetes resources for access to KubeCube, they will go to the auth-proxy module after entering the transparent transmission interface through the authentication middleware; auth-proxy will parse the Bearer Token in the request into the corresponding User; then use the User impersonation In this way, the request proxy is sent to the kube-apiserver, that is, the "admin" user is used to pretend to be the current user to request the kube-apiserver, thereby "skip" the authentication and facilitate subsequent authentication.

Concluding remarks
KubeCube's user management system is mainly implemented based on User CRD; the authentication system supports both local and external authentication methods. Local authentication is implemented based on JWT. After the third-party authentication platform is authenticated, it is also necessary to create a User cr in the cluster. Carry out subsequent user management, permission binding, etc. For cluster authentication, the Impersonation method "skip authentication" provided by Kubernetes is mainly used. The overall design and implementation are relatively simple, adhering to the KubeCube lightweight design concept.

For more information, please refer to:

KubeCube official website: https://www.kubecube.io/

KubeCube source code: https://github.com/kubecube-io/kubecube

In-depth interpretation of KubeCube multi-cluster management

KubeCube multi-level tenant model

KubeCube open source: six features that simplify the implementation of Kubernetes

Netease Shufan more open source projects

About the author: Jiahui, senior engineer of Netease Shufan, core member of KubeCube community