How to use gitlab's Api in spring boot

foreword

Recently, there is a project requirement to obtain the data through gitlab's api. To record my process of stepping on the pit.
First, I tried to get the users data, that is, the data of all users.

Here is the gitlab official API documentation https://docs.gitlab.com/ee/api/users.html

Here's what I did at the beginning and what my teacher suggested.

initial approach

1. Get Admin access token

First, we need to obtain gitlab's Admin access token, and attach this token to the request, so that gitlab can authenticate our identity and return data.

1. Log in to GitLab. 2. After logging in, click the avatar icon in the upper right corner and select Preferences.

3. You can create a new token in the access token interface.

When you are the administrator of the group, you can get users through the token.

2. Use the spring boot unit test to initiate a request

Here I use apache to initiate an http request, and use fastjson to process the json data returned after the request.

 <dependency>
  <groupId>org.apache.httpcomponents</groupId>
  <artifactId>httpclient</artifactId>
  <version>4.5.9</version>
</dependency>

 <dependency>
    <groupId>com.alibaba</groupId>
    <artifactId>fastjson</artifactId>
    <version>1.2.76</version>
</dependency>

Below is the code I tested. To explain:

1. First define the url you want to request, the format is code. You can also see the official documentation.
2. Build the Http request class and set the parameters. The parameter must have private_token. Here I added without_project_bots to filter robot users.
3. Use the execute function to initiate a get request and receive the returned json data.
4. Convert from json array to entity array

This gives us the data we need.

 @Test
  void getUsers() throws IOException, URISyntaxException {

    String url = "https://your gitlab address/api/v4/users";


    CloseableHttpClient httpclients = HttpClients.createDefault();
    HttpGet httpGet = new HttpGet(url);
    // 设置参数信息
    URI uri = new URIBuilder(httpGet.getURI())
            .addParameter("private_token", "your amind token")
            .addParameter("without_project_bots", "true")
            .build();
    ((HttpRequestBase) httpGet).setURI(uri);

    // 发起请求
    CloseableHttpResponse response = httpclients.execute(httpGet);

    // 处理数据
    HttpEntity httpEntity = (HttpEntity) response.getEntity();
    String result =  EntityUtils.toString(httpEntity,"UTF-8");

    // 从json转换成实体
    final ObjectMapper objectMapper = new ObjectMapper();
    User[] langList = objectMapper.readValue(result, User[].class);

    logger.debug(String.valueOf(langList));

    response.close();
  }

Problems I encountered during

1. java needs to add a certificate

Since the request address we initiated is HTTPS, and HTTPS adds a security certificate SSL to the link.
The peer's interface requires a specific certificate to request. If there is no SSL certificate for the relevant link in the server, a sun.security.validator.ValidatorException error will be reported.

 javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

At the beginning, I added the certificate to the local, and the method of adding probably refers to the following blog
https://www.cnblogs.com/zoro-zero/p/11607674.html

This adds the certificate to the local java keystore.

But this way only the local can use the project normally. Because the certificate is not added to the project. So this also shows that this method is not universal.

So we need to put the certificate in src/main/resourse and deploy https . Below I will explain how to add it.

2. No data source configured

 Error creating bean with name 'dataSourceScriptDatabaseInitializer' defined in class path resource [org/springframework/boot/autoconfigure/sql/init/DataSourceInitializationConfiguration.class]

Since I am only testing in unit tests, the project has just been initialized, and the database is not connected, that is, the data source is not configured.

So the error occurred because not enough data was passed to Spring Boot to configure the data source.

So, if we don't need the data source temporarily, we can use exclude={DataSourceAutoConfiguration.class} for automatic configuration. This will solve the error

 @SpringBootApplication(exclude={DataSourceAutoConfiguration.class})
public class NonlycodeApplication {
}

3. JSON data has more properties than entity properties

After I request the data, I can see that there is a lot of data returned by gitlab. There are theme_id, note, is_admin and other information.

---

However, only a few pieces of required information are currently defined in the entity I define, such as id, name, username and other information.

 public class User implements Serializable {

  private static final long serialVersionUID = 1L;

  @Id
  private Long id;

  private String email = "";

  private Integer following = 0;

  private Integer followers = 0;

  private String name = "";

  private  String password = "yunzhi";

  @Column(nullable = false, unique = true)
  private String username;

When json converts data, an error will be reported in this case. And the error says: not marked as ignorable.

So we need to add comments to ignore unnecessary data.

My workaround was to add @JsonIgnoreProperties(ignoreUnknown = true) .
Finally the error disappears.

 @JsonIgnoreProperties(ignoreUnknown = true)
@Entity
public class User implements Serializable {

Access https using RestTemplate

Access to https requires authentication. It is divided into one-way authentication and two-way authentication.

It is enough to access the gitlab API one-way authentication, that is, as long as the CA certificate of gitlab is authenticated.

And as long as the HTTP request library we use supports HTTPS, one-way authentication can be done automatically.

Traditionally, to access restful services in java code, Apache's HttpClient is generally used. Like my code above. However, this method is too cumbersome to use. Spring provides a simple and convenient template class to operate, which is RestTemplate.

Since RestTemplate does not support HTTPS, we need to encapsulate it.

Refer to this article https://blog.csdn.net/MyronCham/article/details/103481046

1. Create a new factory class

2. Configure the factory class

 public class HttpsClientRequestFactory extends SimpleClientHttpRequestFactory {
  @Override
  protected void prepareConnection(HttpURLConnection connection, String httpMethod) {
    try {
      if (!(connection instanceof HttpsURLConnection)) {
        throw new RuntimeException("An instance of HttpsURLConnection is expected");
      }

      HttpsURLConnection httpsConnection = (HttpsURLConnection) connection;

      TrustManager[] trustAllCerts = new TrustManager[]{
              new X509TrustManager() {
                @Override
                public java.security.cert.X509Certificate[] getAcceptedIssuers() {
                  return null;
                }
                @Override
                public void checkClientTrusted(X509Certificate[] certs, String authType) {
                }
                @Override
                public void checkServerTrusted(X509Certificate[] certs, String authType) {
                }

              }
      };
      SSLContext sslContext = SSLContext.getInstance("TLS");
      sslContext.init(null, trustAllCerts, new java.security.SecureRandom());
      httpsConnection.setSSLSocketFactory(new MyCustomSSLSocketFactory(sslContext.getSocketFactory()));

      httpsConnection.setHostnameVerifier(new HostnameVerifier() {
        @Override
        public boolean verify(String s, SSLSession sslSession) {
          return true;
        }
      });

      super.prepareConnection(httpsConnection, httpMethod);
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
  private static class MyCustomSSLSocketFactory extends SSLSocketFactory {

    private final SSLSocketFactory delegate;

    public MyCustomSSLSocketFactory(SSLSocketFactory delegate) {
      this.delegate = delegate;
    }

    // 返回默认启用的密码套件。除非一个列表启用，对SSL连接的握手会使用这些密码套件。
    // 这些默认的服务的最低质量要求保密保护和服务器身份验证
    @Override
    public String[] getDefaultCipherSuites() {
      return delegate.getDefaultCipherSuites();
    }

    // 返回的密码套件可用于SSL连接启用的名字
    @Override
    public String[] getSupportedCipherSuites() {
      return delegate.getSupportedCipherSuites();
    }


    @Override
    public Socket createSocket(final Socket socket, final String host, final int port,
                               final boolean autoClose) throws IOException {
      final Socket underlyingSocket = delegate.createSocket(socket, host, port, autoClose);
      return overrideProtocol(underlyingSocket);
    }


    @Override
    public Socket createSocket(final String host, final int port) throws IOException {
      final Socket underlyingSocket = delegate.createSocket(host, port);
      return overrideProtocol(underlyingSocket);
    }

    @Override
    public Socket createSocket(final String host, final int port, final InetAddress localAddress,
                               final int localPort) throws
            IOException {
      final Socket underlyingSocket = delegate.createSocket(host, port, localAddress, localPort);
      return overrideProtocol(underlyingSocket);
    }

    @Override
    public Socket createSocket(final InetAddress host, final int port) throws IOException {
      final Socket underlyingSocket = delegate.createSocket(host, port);
      return overrideProtocol(underlyingSocket);
    }

    @Override
    public Socket createSocket(final InetAddress host, final int port, final InetAddress localAddress,
                               final int localPort) throws
            IOException {
      final Socket underlyingSocket = delegate.createSocket(host, port, localAddress, localPort);
      return overrideProtocol(underlyingSocket);
    }

    private Socket overrideProtocol(final Socket socket) {
      if (!(socket instanceof SSLSocket)) {
        throw new RuntimeException("An instance of SSLSocket is expected");
      }
      //((SSLSocket) socket).setEnabledProtocols(new String[]{"TLSv1.2"});
      ((SSLSocket) socket).setEnabledProtocols(new String[]{"TLSv1", "TLSv1.1", "TLSv1.2"});
      return socket;
    }
  }

3. Write the code

User entity:

 @JsonIgnoreProperties(ignoreUnknown = true)
@Entity
public class User implements Serializable {

  private static final long serialVersionUID = 1L;

  @Id
  @JsonView
  private Long id;

  @JsonView
  private String email = "";

  @JsonView
  private Integer following = 0;

  @JsonView
  private Integer followers = 0;

  @JsonView
  private String name = "";

  private  String password = "yunzhi";

  @JsonView
  private String username;


  public void setId(Long id) {this.id = id;}

  public Long getId() {return this.id;}

  public void setEmail(String email) {this.email = email;}

  public String getEmail() {return this.email;}

  public void setName(String name) {this.name = name;}

  public String getName() {return this.name;}

  public void setUsername(String username) {this.username = username;}

  public String getUsername() {return this.username;}

  public void setFollowing(Integer following) {this.following = following;}

  public Integer getFollowing() {return this.following;}

  public void setFollowers(Integer followers) {this.followers = followers;}

  public Integer getFollowers() {return this.followers;}



}

controller layer:

 @RestController
@RequestMapping("user")
public class UserController {

  @Autowired
  UserService userService;


  @GetMapping("/getUsers")
  @JsonView(GetUsersJsonView.class)
  public User[] getUsers() throws JsonProcessingException {
    return  this.userService.getUsers();
  }

  public class GetUsersJsonView {
  }
}

service layer:
Pay attention to fill in your own url and token. The logic is similar to the previous code.

 @Override
  public User[] getUsers() throws JsonProcessingException {
    String url = "https://your gitlab address /api/v4/users/" +
            "?private_token={private_token}&without_project_bots={without_project_bots}";

    final Map<String, String> variables = new HashMap<>();

    variables.put("private_token", "your token");
    variables.put("without_project_bots", "true");

    RestTemplate restTemplate = new RestTemplate(new HttpsClientRequestFactory());
    String response = restTemplate.getForObject(url, String.class, variables);


    // 从json转换成实体
    final ObjectMapper objectMapper = new ObjectMapper();
    User[] userList = objectMapper.readValue(response, User[].class);

    return userList;
  }

4. url access

5. Results

Information was successfully obtained.

Summarize

After this request for the gitlab API, I feel that I have a deeper understanding of how to make a request and how to process the returned json data. At the same time, I also feel that my understanding of https and SSL certificates is not enough, and I need to supplement relevant knowledge later.