Linux cloud host security intrusion troubleshooting steps

Introduction

Malicious programs are often implanted in the user's fault reporting system, such as mining software, ddos attack virus, syn mapping attack virus, etc. You can check the types of intrusion viruses for users according to the following process:

First, locate the virus process

For user feedback cloud host performance is stuck, CPU and memory usage is high:

Execute the TOP command to check the abnormal processes with high CPU usage. Generally, it is more than 80%. Some viruses may not occupy high CPU, but the command name is obviously not a system process or a user process, such as TSM, Lixsyn, ynlyvtpxia, etc.

as the picture shows:

Note that most users running abnormal processes are root, and they may also be other application users, generally users who can log in to the system. After the virus cracks the user's login password, use the user to log in to the system, and upload the virus program according to the permissions of the intruding user. To different directories, such viruses generally have the function of preventing deletion and automatic start of the process.

That is, it cannot be deleted from the directory, and it will be automatically restored after deletion. The process is automatically started again a few moments after the process is cleaned up with the Kill -9 command.

The red box in the above picture is the virus program. Although it does not occupy much CPU and network bandwidth, it still has the characteristics of preventing deletion and automatic startup.

After the process is found, record the PID process number, which will be used later to locate the unknown virus program.

For users who report that the connection to the cloud host is stuck, but the CPU and memory usage is not high, it may be caused by the full outgoing bandwidth of the cloud host. At this time, a large amount of packet loss occurs when pinging the public IP address of the cloud host.

Connect to the cloud host through VNC, and use the iftop command to locate the process with high bandwidth (iftop command usage tutorial can refer to the document "Linux Traffic Monitoring Tool - iftop (the most comprehensive iftop tutorial)":
https://developer.jdcloud.com/article/1729 ), generally there are a large number of external public IP data in and out of the same port, and this phenomenon is mostly caused by viruses or malicious attacks.

Use iftop -P to locate the port with high traffic, and then use the lsof -i: port number command to view and record the process number that occupies the port.

Second, locate the location of the virus program

Locate the running virus program path according to the virus process number located in the first step, and the command is as follows:

ll /proc/process number

For each running process, the Linux system will create a temporary directory named after the process number in the /proc directory. This directory records the relevant information of the process. The exe project is the program path corresponding to the process, as shown in the figure:

The virus file was uploaded to the /usr/bin directory, apparently the virus has gained root privileges. Sometimes it is also placed in the /tmp directory, depending on the different user permissions the virus has acquired. Enter the path where the file is located, most virus files will appear again after deletion, and cannot be completely removed.

This is why we generally recommend that users reset the system after being poisoned. Generate a tar package for the virus file or the directory created by the virus where the virus file is located, and then use the lrzsz tool to download it locally.

The reason why a single file is packaged in a tarball is to prevent it from being deleted by anti-virus software after being downloaded locally. After uploading it to the virus analysis website, if it is not in the tar package format, sometimes the result cannot be analyzed.

There are many virus analysis websites, the commonly used ones are:
https://x.threatbook.cn/
http://www.virustotal.com
http://r.virscan.org

Take https://x.threatbook.cn/ as an example, after opening the webpage, click the upload file in the red box in the figure below, and select the virus tar package to upload.

After the file is uploaded, click the scanned file in the red box below to enter the analysis page. The website will use different security software to analyze the virus and give the analysis result.

Some of them can be detected as viruses, and some of them are not detected as viruses. If more than one file reports a virus, it can be confirmed that it is a virus.

As shown in the figure, the scanned virus is a DDoS attack virus.

After confirming the virus type, you can try to remove it by referring to the document "Checking and Removing DDoS Trojans in Linux System" (with product).

If it cannot be completely cleared, it is recommended to use a private image or an official image to reset the system after backing up the personal data in the system.

Usually, the vast majority of security intrusions are achieved through fully open port policies and application vulnerabilities, so using the default security group to open all ports is extremely risky.

Special Note:

For linux cloud hosts, it is recommended that you bind the linux open 22-port security group, remove the default security group, and open the required ports separately. For the operation method, please refer to the following documents:

Security Group Overview ( https://docs.jdcloud.com/cn/virtual-machines/security-group-overview )

Bind the security group to the cloud host ( https://docs.jdcloud.com/cn/virtual-machines/associate-security-group )

Disassociate a cloud host from a security group ( https://docs.jdcloud.com/cn/virtual-machines/disassociate-security-group )

Configure security group inbound rules ( https://docs.jdcloud.com/cn/virtual-machines/configurate-inbound-rules )

Configure security group outbound rules ( https://docs.jdcloud.com/cn/virtual-machines/configurate-outbound-rules )

Modify the ssh protocol port ( https://docs.jdcloud.com/cn/virtual-machines/ssh-service-change-port )

Use complex passwords and change them regularly

At the same time, make a private image ( https://docs.jdcloud.com/cn/virtual-machines/create-private-image ) and cloud disk snapshots ( https://docs.jdcloud.com/cn/ cloud-disk-service/create-clouddisk-snapshot ), back up your system configuration and data so that in the event of an accident you can reset your system with a private image.