Semgrep combined with GitLab to implement code audit practice-server

1. Background

I was doing a code audit some time ago and found that many projects have hidden safety hazards, most of which are caused by unfiltered parameters; in order to solve this problem, I Web Security Development Specification Manual V1.0 , but the effect It is not too ideal, because after training, developers’ focus is mainly on the completion of functions, and secure coding is not a core indicator for them;

In order to allow developers to pay attention to security issues at all times, I put a hook on the gitlab server. This hook mainly detects the code files submitted this time and outputs them when they encounter problems that may have security risks. In this way, developers can have a deeper feeling about the content of the training and pay more attention to security issues during coding.

2. Operation steps

1. Set up the environment
2. Create project
3. Create hook
4. Hook experiment

Three, set up the environment

3.1 Install gitlab

Before officially deploying to the server, I need to build a gitlab service locally for the development and testing of hooks. Here I use docker to build faster. The commands to execute are as follows

docker run --detach  --publish 443:443 --publish 80:80  --name gitlab --restart always  gitlab/gitlab-ce

After the command is executed, the returned information is as follows

In the above figure, you can see that the container has run successfully, use the browser to access the address of gitlab

http://127.0.0.1

After accessing, you need to set an administrator's password, as shown in the figure below

After filling in the password, confirm the modification of the password, and it will jump to the homepage of gitlab, as shown in the figure below

Create a project in gitlab for hook testing, as shown in the figure below

After successfully creating the project, pay attention to the Project ID:2 in the 2 , which will be used later; next, you need to start the development and deployment of the hook. The hook can be developed in various languages. Here I am more familiar with php, so I use php Development.

3.2 Installation dependencies

The gitlab container does not support php language by default, you need to install php first, the installation command is as follows

apt update -y && apt install php -y

After the command is executed, the returned information is as follows

In the above figure, you can see that php has been installed successfully, in order to verify whether the php command can run, here I use the following command to verify

php -v

After the command is executed, the returned information is as follows

In the figure above, you can see that the version of php is 7.4.3, indicating that php has been installed successfully.

3.3 install semgrep

Semgrep needs to be called in the hook program. This program is also not installed in gitlab. It needs to be installed. Here is installed using pip, but you need to upgrade the version of pip first. The upgrade command is as follows

pip3 install --upgrade pip

After the command is executed, the returned information is as follows

In the above figure, you can see that the version of pip has been upgraded to 21.1.2, indicating that the upgrade was successful

semgrep also relies on the setuptools module, you need to use pip to upgrade first, the upgrade command is as follows

pip3 install --upgrade setuptools

After the command is executed, the returned information is as follows

In the above picture, you can see that the setuptools module has been successfully upgraded

Then you can formally install semgrep, the installation command is as follows

cd /usr/local/bin/ && python3 -m pip install semgrep

After the command is executed, the returned information is as follows

In the above figure, you can see that semgrep has been installed. Here I need to use the semgrep command again to verify it. The executed command is as follows

semgrep --version

After the command is executed, the returned information is as follows

In the above figure, you can see that the version information of semgrep is 0.52.0, confirming that the installation is successful.

3.4 View hash

Now we need to add a hook to the project we just created, here we need to find the storage path of the project, in the project page

echo -n 2 | sha256sum

After the command is executed, the returned information is as follows

find / -iname d4

After the command is executed, the returned information is as follows

In the picture above, you can see where the project is stored. Two paths are returned. One of the two paths is a soft connection. Enter the storage location of the project cd

cd /var/opt/gitlab/git-data/repositories/@hashed/d4

After the command is executed, execute the ls command again, and the information obtained is as follows

In the above figure, you can see that there is a 73 folder, this is the naming rule of gitlab, enter this folder, the command is as follows

cd 73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git/

After the command is executed, the returned information is as follows

In the picture above, you can see all the files of this project, I need to develop hook files in this location

Five, create a hook

Custom hooks need to be stored in the custom_hooks directory, there is no such folder by default, so you need to create this folder, execute the command as shown below

mkdir custom_hooks  && cd custom_hooks

5.1 New hook

After creating the custom_hooks folder and entering it, use vim to create a hook file, the command is as follows

vim  pre-receive

After entering the vim editor interface, add the following hook code, the code is as follows

#!/usr/bin/php
<?php


fwrite(STDOUT, 'please input:');
list($oldVer, $newVer, $ref_name) = explode(" ", fgets(STDIN));

//ob_start();
$cmd = "git diff --name-only  {$oldVer}..{$newVer}";
echo $cmd . PHP_EOL;
exec($cmd, $result);

$rand = date("Y-m-d-H-i-s");
$baseDir = "/tmp/11/$rand/";
$ruleFile = "/semgrep-rule.yaml";




foreach ($result as $value) {
    if (strstr($value, ".php") !== false) {
        $randName = $baseDir . $value;
        if (!is_dir(dirname($randName))) {
#        if (file_exists($randName) == false) {
            mkdir(dirname($randName), 0777, true);
        }
        $cmd = "git show {$newVer}:$value > $randName";

#        echo $cmd . PHP_EOL;


        exec($cmd, $result);
    }
}


$cmd = "/opt/gitlab/embedded/bin/semgrep  -f '$ruleFile' $baseDir  -o /tmp/11.txt";

exec($cmd, $result);


//ob_clean();


$notice = file_get_contents("/tmp/11.txt");
echo $notice . PHP_EOL;


file_put_contents("/tmp/11.txt", "");
exec("rm -rf $baseDir");

echo 0;

Save and launch this hook file, and then you need to set permissions for the custom hook directory. Here I simply and rudely set the permissions to 777. The command is as follows

chmod -R 777 ../

After the permissions are set, I also need to create a semgrep to determine whether the code is correct.

The command executed is as follows

vim /semgrep-rule.yaml 

After entering the vim editor, you need to copy the following rules into it

rules:
  - id: assert-use
    patterns:
      - pattern: assert($ASSERT, ...);
      # - pattern-not: assert(<... $ASSERT ...>, ...); - https://github.com/returntocorp/semgrep/issues/2035
      - pattern-not: assert("...", ...);
    message: |
      使用用户输入调用assert等价于eval'。
    metadata:
      references:
        - https://www.php.net/manual/en/function.assert
        - https://github.com/FloeDesignTechnologies/phpcs-security-audit/blob/master/Security/Sniffs/BadFunctions/AssertsSniff.php
    languages: [ php ]
    severity: ERROR

  - id: backticks-use
    pattern: '`...`;'
    message: |
      使用反勾号可能导致命令注入漏洞。
    metadata:
      references:
        - https://www.php.net/manual/en/language.operators.execution.php
        - https://github.com/FloeDesignTechnologies/phpcs-security-audit/blob/master/Security/Sniffs/BadFunctions/BackticksSniff.php
    languages: [ php ]
    severity: ERROR

After saving and launching the rule file, you need to modify the permissions of this rule file. Here I set the permission distance of 777. The command is as follows

chmod 777 /semgrep-rule.yaml

After setting the rule file permissions, there are two cache places that need to set permissions, otherwise an error will be reported during the running process. The first is the cache file of semgrep. The command for setting permissions is as follows

mkdir -p /var/opt/gitlab/.cache  && chmod -R 777 /var/opt/gitlab/.cache

The other is the cache file of the hook itself, which also needs to set permissions. The commands to be executed are as follows

echo '' > /tmp/11.txt  && chmod 777 /tmp/11.txt

5.2 Test hook

Now you can formally test the usability of the hook. First, you need to pull the project code you just created. The command is as follows

git clone http://127.0.0.1/root/test.git

After executing the command, the returned information is as follows

In the above picture, you can see that the project has been pulled down, and then I need to edit a php file, the command is as follows

vim index.php

After the command is executed, store the test code in

<?php

phpinfo();


$cmd = "ls {$_GET['x']}";

exec($cmd);

After saving and exiting, submit the code to gitlab, the command is as follows

echo ' ' >> index.php && git add . && git commit . -m 'init' && git push

But after git pushes to the gitlab server, gitlab will call the hook and output the information returned by the hook, as shown in the figure below

In the figure above, you can see that the hook prompts that index.php file is not safe. Sincerely, the entire deployment is complete.

---

Author: Tang Qingsong

Date: 2021-06-03

WeChat: songboy8888