The realization idea of a springboard machine

vivo Internet Operation and Maintenance Team - Yang Lei

This paper introduces the realization idea of a springboard machine, expounds the basic principle, and explains the characteristics and relative advantages.

1. Introduction to the idea of springboard machine

The jumper described in this article (hereafter referred to as "jmp") supports:

Linux server
Windows server
Other terminals (MySQL terminal, Redis terminal, network device terminal, etc.)

Different from the common jumpserver solution on the market, the jump server built in this article will not store any Linux server account, password, key and other information, preventing the possibility of information leakage. The biggest feature of this article is that with the help of the PAM mechanism of Linux, by modifying the system layer configuration of the Linux server, it partially takes over the identity authentication capability of the Linux system. This point will be described in detail below.

2. Background knowledge

2.1 PAM mechanism of Linux

PAM (Pluggable Authentication Modules) mechanism is a system-level authentication framework widely used in contemporary Unix and Linux distributions. By providing a series of dynamic link libraries and two sets of programming interfaces (Service Programming Interface and Application Programming Interface), the service provided by the system is separated from the authentication method of the service, so that different authentication can be flexibly configured for different services according to needs. way without changing the service program.

2.2 Core Competencies of PAM

2.3 PAM module types

auth

It is used to identify the user's identity, such as prompting the user to enter a password, or judging whether the user is root, etc.

account

Check various attributes of the account, such as: whether to allow login, whether the maximum number of users is reached, or whether the root user is allowed to log in at this terminal, etc.

session

This module is used to define the operations to be performed before the user logs in and after the user logs out, such as: login connection information, opening and closing of user data, mounting the file system, etc.

password

Use user information to update, such as: modify user password.

2.4 Common PAM Modules

pam_unix.so module

[auth] Prompt the user to enter a password, and compare it with the /etc/shadow file, if the match returns 0 (PAM_SUCCESS). [account] Check the user's account information (including whether it has expired, etc.), if the account is available, return 0. [password] Modify the user's password, and update the shadow file with the password entered by the user as the user's new password.

pam_cracklib.so module

This module can be inserted into a program's cipher stack to check the strength of ciphers.

pam_loginuid.so module

Used to set the uid of the authenticated process so that the program can pass the normal audit.

pam_securetty.so module

If the user wants to log in as root, the logged in tty must be in /etc/securetty before.

pam_rootok.so module

The pam\_rootok module is used to verify whether the user id is 0, and if it is 0, it returns PAM\_SUCCESS.

pam_console.so module

When the user logs into the terminal, change the permissions of the terminal files. After the user logs out, change them back.

pam_permit.so module

The module returns success any time.

pam_env.so module

pam\_env allows setting environment variables; by default, if no file is specified, environment variables will be set according to /etc/security/pam\_env.conf

pam_xauth.so module

pam_xauth is used to forward xauth-keys between users.

pam_stack.so module

pam_stack can call another service; that is, multiple services can be included in one setup, and only one file can be modified when it needs to be modified.

pam_warn.so module

pam\_warn is used to record information about services, end users, remote users and remote hosts to the system log. The module always returns PAM\_IGNORE, which means that it does not want to affect the authentication process.

Third, the springboard system architecture

3.1 Microservices and High Availability Design

3.1.1 Microservice Design

The entire springboard system can be split into 5 services and 1 component.

① jmp-api service

Monitor port 8080, provide http interface capability

Verify that an account exists and is normal
Verify whether an account has login rights to a server
Verify whether an account has sudo privileges on a server
Data pull: account, host, dangerous command library, etc.

It is the only entry for jmp to access the database

② jmp-ssh service

Monitor port 2200, provide ssh proxy capability
Direct access to Linux servers and other terminals

③ jmp-socket service

Monitor port 8080, provide websocket/socket.io connection capability
Forward socket.io traffic to jmp-ssh via ssh protocol
Support the connection and access of web terminal

④ jmp-rdp service

Monitor port 8080, provide socket.io connection capability
Implement rdp proxy for easy operation of windows server
Support for Web-based Remote Desktop Services

⑤ jmp-sftp service

Provides file upload and download capabilities, supports sftp commands in jmp, and supports any sftp client connection
Access to S3 for file access

⑥ jmp-agent component

Deployed in every Linux server
jmp-agent resident process

Regularly pull service and permission information from jmp-api and cache it to a local file
Detect file changes as needed to ensure configuration files are not maliciously modified

jmp dedicated pam module

Provide jmp.so dynamic library, pam module
The installation script releases the configuration file, modifies the /etc/pam.d/xxx file, and takes effect the pam module of jmp
Take over identity recognition and authority authentication, and call the jmp-api interface to complete the authentication

3.1.2 High Availability Design

Any service in jmp is stateless, so it supports remote multi-room deployment

HTTP protocol services (jmp-api, jmp-socket, jmp-rdp), configure routing through Nginx, and configure automatic load balancing policies.
Non-http services (jmp-ssh, jmp-sftp) are highly available through Layer 4 load balancing (lvs, vgw).

Automatic downgrade policy

The ability to identify dangerous commands may take a long time. Therefore, when an interface for identifying dangerous commands is found to be timed out, the identification of dangerous commands is automatically ignored.
If the authentication interface times out, the identity information cached locally by jmp-agent is used. If the local cache cannot be obtained, the default policy of the configuration item (pass all or reject all) is used.

High availability of jmp-agent components

Since jmp-agent is deployed on the business server, the environment may change at any time, so it must have strong adaptability (insufficient disk space, full inode, insufficient memory, unstable network, abnormal domain name resolution, etc.).
Due to insufficient disk space or inode, jmp-agent may not be able to use the local file cache, so choose to downgrade and ignore the cache at this time.
In response to the problem of network instability, jmp-agent chooses to increase the communication timeout with jmp-api and jmp-ssh, and at the same time, it can downgrade the authentication to ensure that the operation is not affected.
In response to the problem of parsing exceptions, jmp-agent cannot interact with the service through the domain name. In this case, the built-in fixed IP is used to interact with the service.

3.2 Interaction diagram of each sub-service of the springboard

As can be seen from the figure, jmp-ssh as the core service carries the proxy forwarding of ssh traffic, forwards the ssh traffic from the user's ssh client and jmp-socket service to the target server, and delivers the return result from the target server to the target server. Back to ssh client, jmp-socket service. Therefore, dangerous commands from users can be identified on the jmp-ssh service, and an alarm or direct interception can be given before reaching the target server, so as to avoid malicious operations or misoperations that affect the business.

The jmp-api in the figure, as a service that directly interacts with the database and cache, assumes the role of data interface and management end in the whole system, and accepts user identity authentication and permission verification requests from the jmp-agent component in the full server, which is the whole system. in the control center.

jmp-api also provides the ability to set permissions. By connecting with the process system, it is convenient to apply for login permissions or root permissions of machines/services/projects for personnel/departments. In addition, jmp-api also provides access to login permissions and root permissions Applicants can make restrictions, limit the effective time of permissions for different projects/services, and strictly control the granularity of permissions.

Since the same project/service is often maintained by the same group of people, jmp-api has a built-in default permission policy that allows the person in charge of the project/service to directly log in to the project/service without applying for it; only supports corresponding The person in charge of operation and maintenance of a project/service has root privileges by default. If everyone else wants to obtain root privileges, they must apply for and be approved by the person in charge of operation and maintenance of the corresponding service.

The jmp-agent in the figure is deployed on each Linux server, by modifying /etc/pam.d/sshd, /etc/ on Linux

pam.d/remote, /etc/pam.d/sudo, etc. files, let jmp.so (part of jmp-agent.rpm or jmp-agent.deb) take over the identity of key system programs such as ssh service and sudo program Identification and authorization authentication. Therefore, the ability to identify all personnel on any server is realized without increasing the contents of /etc/passwd and /etc/shadow.

The jmp-rdp in the figure only serves as the rdp proxy service of the Windows server and provides web-based remote desktop capabilities.

The jmp-socket in the figure provides a web-based Linux server operation terminal, so that users can easily log in to the server without using the ssh client.

4. Core Design Ideas

4.1 Log in to the springboard

The user uses the ssh client to log in to the jmp-ssh service and interact with the jmp-ssh service.
The jmp-ssh service obtains the account, encrypted password, and secondary authentication information during the establishment of the ssh session.
The jmp-ssh service accesses the jmp-api service, and submits the account number, encrypted password, and secondary authentication information, so as to know whether the user has permission to log in to jmp.

4.2 Log in to the target server

The user can log in to the target server only if the user has logged in to jmp-ssh or has passed the front-end authentication of jmp-socket.
The user enters ssh xxxx (xxxx is the host name or IP address of the target server) under the pseudo terminal provided by jmp-ssh.
jmp-ssh connects to the target server through ssh, automatically carries the user name information, and tries to establish a session.
Since the jmp-agent on the target server takes over the sshd identification and authority authentication, jmp.so obtains the user name during the establishment of the ssh session, encrypts the user name and the local IP address information, and calls the jmp-api interface for authority authentication .
jmp-api determines whether the user has the right to log in to the machine according to the built-in policy and querying the authorization table.
jmp-agent obtains the authentication result. For those with authority, the ssh session is established successfully, otherwise the session establishment fails.
jmp-ssh obtains the session establishment result and reason, and returns it to the user's ssh terminal.

4.3 Command interaction

Command interaction is only possible when the user is already logged on to a machine.
When the user types a character on the ssh client, it is passed to jmp-ssh, and jmp-ssh judges whether the statement ends.
When the statement ends, jmp-ssh matches the statement entered by the user according to the dangerous command rules of the machine, and decides to alert, block, and pass.
jmp-ssh passes the passed statement or the statement that needs to be alerted to the target server, and the target server executes and returns the result.

4.4 Switch user/privileged account

Switching users is only possible when the user is already logged into a machine.
When the user executes commands such as sudo xxxx, su, id, etc. on the ssh client, jmp-ssh transparently transmits the commands to the target server.
The sshd process on the target server executes commands such as sudo xxxx, su, id, etc. Since the identity failure and authority authentication have been taken over by jmp-agent on the target server, jmp.so obtains the login user name, current user name, local machine Address information, target user name information, and call the jmp-api interface for sudo permission authentication.
jmp-api determines whether the user has the permission to switch to the xx account on the machine (eg, whether it has root permission).
Processes such as sudo, su, and id obtain the authentication result through jmp.so and decide whether to switch users.

4.5 Interacting with web pages

Only for cases where the user has already logged in via a web page (eg sso).
The user accesses the jmp-socket service through the web page.
The jmp-socket service obtains user name information and web page login sso information, submits it to jmp-api, and generates a temporary login credential.
jmp-socket accesses jmp-ssh and submits temporary login credentials.
jmp-ssh initiates the secondary authentication for login and waits for the user to complete the secondary authentication.
After the user completes the secondary authentication, jmp-socket assumes the role of the ssh client and interacts with jmp-ssh.

4.6 Dangerous command interception

After the user has logged in to the target server, jmp-ssh loads the dangerous command rules of the corresponding service of the target machine in the session, and initializes the regular matching logic.
After the end of the user input statement, jmp-ssh matches the statement entered by the user according to the dangerous command rules of the machine.
jmp-ssh decides to process the input as follows according to the strategy after the dangerous command rules are matched: alarm, intercept, and pass.
For passed, jmp-ssh passes the command to the target server.
For alarms, jmp-ssh transmits the command to the target server, but sends a dangerous command alarm to the user, the user's immediate leader, and the jmp system administrator.
For intercepted ones, jmp-ssh refuses to transmit the command, and at the same time sends a dangerous command warning to the user, the user's immediate leader, and the jmp system administrator.

4.7 Springboard for non-Linux servers

Windows server

For Windows servers, use the jmp-rdp service to convert rdp protocol data into application data carried by socket.io (depending on Apache Guacamole), and display real-time images through the Canvas of the web page and accept keyboard and mouse events.

MySQL Terminal and Redis Terminal

Only MySQL and Redis deployed on Linux servers are supported.
On the server through mysql.sock, make jmp-agent connect to the local MySQL service, jmp-agent forwards standard input and standard output to jmp-ssh.
On the server through redis.sock, make jmp-agent connect to the local Redis service, jmp-agent forwards standard input and standard output jmp-ssh.
This method theoretically supports any service that can be connected via unixsocket.

Network equipment management terminal

For network terminals, jmp-ssh reads the jmp-api interface, obtains the connection information (protocol type, account information, etc.) of the corresponding network device, and realizes connection and operation.

5. Permission rules and approval link design

5.1 Default permissions

Permissions you can have without applying.

5.2 Approval link for permission application

If there is no default permission, but you need to log in to the machine, or you need to use ROOT permission, you need to apply.
If you apply for permission for an organization, all members of the organization (department) have the permission to lock the application.

The approval link of the application process is clarified here:

Six, the advantages of this realization idea

6.1 Easy operation and better experience

The springboard system built through this idea is more convenient to operate, not only supports ssh, but also is compatible with rdp, and provides a web-side operation entry, and the experience is better. At the same time, due to the micro-service architecture, the coupling between services is small, and it is easier to achieve high availability, so there are few phenomena such as stalls and delays, the overall stability is reliable, and the experience is guaranteed.

6.2 Safe and reliable, easy to audit

The biggest feature of this article is that the pam mechanism is used on the target server, and jmp.so takes over the identification and authority authentication of multiple services, so that the authority can be taken over and managed in a unified manner without modifying the standard commands. And after logging in to the target machine, you can further ssh to other servers, all interactive processes are recorded throughout, and all operation commands will be recorded.

Since the springboard machine realized by this idea directly uses the username as the login name of the target server ssh session, the log recorded inside the system is also the direct username, not the unified account of programs such as jumpserver. This method It is easier to locate the real executor of the operation trajectory, which is clear at a glance.

The dangerous command interception function can largely avoid malicious operations or destructive misoperations, adding a layer of protection to business stability.

6.3 Clear responsibilities between services

Due to the adoption of the micro-service architecture, the horizontal expansion of each service can be achieved, so that more machines can be managed and controlled by expanding the service. The responsibilities between services are clear, and jmp-rdp, jmp-socket, and jmp-sftp can be reduced as needed, and new services can be added as needed, with better adaptability.

7. Summary and Outlook

As servers grow in size, how to manage these servers becomes an increasingly important issue. For the login access of the server, this paper introduces a realization idea of the springboard machine, and describes the advantages and uniqueness of this idea. Through this idea, a simple, easy-to-use and highly available springboard can be built to a certain extent, so as to solve the problem of server login. If readers are interested in this implementation idea, or have any questions, welcome to communicate with us. We are also very willing to work with you to learn and research technology.