Go, a library of resty every day

Introduction

resty is an HTTP client library of Go language. resty is powerful and feature-rich. It supports almost all HTTP methods (GET/POST/PUT/DELETE/OPTION/HEAD/PATCH, etc.) and provides a simple and easy-to-use API.

Quick to use

The code in this article uses Go Modules.

Create a directory and initialize:

$ mkdir resty && cd resty
$ go mod init github.com/darjun/go-daily-lib/resty

Install the resty library:

$ go get -u github.com/go-resty/resty/v2

Let's get Baidu homepage information:

package main

import (
  "fmt"
  "log"

  "github.com/go-resty/resty/v2"
)

func main() {
  client := resty.New()

  resp, err := client.R().Get("https://baidu.com")

  if err != nil {
    log.Fatal(err)
  }

  fmt.Println("Response Info:")
  fmt.Println("Status Code:", resp.StatusCode())
  fmt.Println("Status:", resp.Status())
  fmt.Println("Proto:", resp.Proto())
  fmt.Println("Time:", resp.Time())
  fmt.Println("Received At:", resp.ReceivedAt())
  fmt.Println("Size:", resp.Size())
  fmt.Println("Headers:")
  for key, value := range resp.Header() {
    fmt.Println(key, "=", value)
  }
  fmt.Println("Cookies:")
  for i, cookie := range resp.Cookies() {
    fmt.Printf("cookie%d: name:%s value:%s\n", i, cookie.Name, cookie.Value)
  }
}

resty relatively simple to use.

• First, call a resty.New() create a client object;
• Call R() client object to create a request object;
• Get()/Post() request object, pass in the parameter URL, and then send an HTTP request to the corresponding URL. Return a response object;
• The response object provides many methods to check the status of the response, headers, cookies and other information.

In the above program we obtained:

• StatusCode() : status code, such as 200;
• Status() : Status code and status information, such as 200 OK;
• Proto() : Protocol, such as HTTP/1.1;
• Time() : the time from sending the request to receiving the response;
• ReceivedAt() : the moment when the response is received;
• Size() : response size;
• Header() : header information in response to http.Header return type, i.e. map[string][]string ;
• Cookies() : The cookie information set by the server through the Set-Cookie

Basic information of the response output from the running program:

Response Info:
Status Code: 200
Status: 200 OK
Proto: HTTP/1.1
Time: 415.774352ms
Received At: 2021-06-26 11:42:45.307157 +0800 CST m=+0.416547795
Size: 302456

Header information:

Headers:
Server = [BWS/1.1]
Date = [Sat, 26 Jun 2021 03:42:45 GMT]
Connection = [keep-alive]
Bdpagetype = [1]
Bdqid = [0xf5a61d240003b218]
Vary = [Accept-Encoding Accept-Encoding]
Content-Type = [text/html;charset=utf-8]
Set-Cookie = [BAIDUID=BF2EE47AAAF7A20C6971F1E897ABDD43:FG=1; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com BIDUPSID=BF2EE47AAAF7A20C6971F1E897ABDD43; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com PSTM=1624678965; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com BAIDUID=BF2EE47AAAF7A20C716E90B86906D6B0:FG=1; max-age=31536000; expires=Sun, 26-Jun-22 03:42:45 GMT; domain=.baidu.com; path=/; version=1; comment=bd BDSVRTM=0; path=/ BD_HOME=1; path=/ H_PS_PSSID=34099_31253_34133_34072_33607_34135_26350; path=/; domain=.baidu.com]
Traceid = [1624678965045126810617700867425882583576]
P3p = [CP=" OTI DSP COR IVA OUR IND COM " CP=" OTI DSP COR IVA OUR IND COM "]
X-Ua-Compatible = [IE=Edge,chrome=1]

Note that there is a Set-Cookie header, which will appear in the Cookie section:

Cookies:
cookie0: name:BAIDUID value:BF2EE47AAAF7A20C6971F1E897ABDD43:FG=1
cookie1: name:BIDUPSID value:BF2EE47AAAF7A20C6971F1E897ABDD43
cookie2: name:PSTM value:1624678965
cookie3: name:BAIDUID value:BF2EE47AAAF7A20C716E90B86906D6B0:FG=1
cookie4: name:BDSVRTM value:0
cookie5: name:BD_HOME value:1
cookie6: name:H_PS_PSSID value:34099_31253_34133_34072_33607_34135_26350

Automatic Unmarshal

Many websites now provide API interfaces to return structured data, such as JSON/XML format. resty can automatically Unmarshal the response data to the corresponding structure object. Let's look at an example. We know that many js files are hosted on api.cdnjs.com/libraries . We can get the basic information of these libraries through 060de570a7dbf4 and return a JSON data in the following format:

Next, we define the structure, and then use resty pull information, automatically Unmarshal:

type Library struct {
  Name   string
  Latest string
}

type Libraries struct {
  Results []*Library
}

func main() {
  client := resty.New()

  libraries := &Libraries{}
  client.R().SetResult(libraries).Get("https://api.cdnjs.com/libraries")
  fmt.Printf("%d libraries\n", len(libraries.Results))

  for _, lib := range libraries.Results {
    fmt.Println("first library:")
    fmt.Printf("name:%s latest:%s\n", lib.Name, lib.Latest)
    break
  }
}

As you can see, we only need to create an object of the result type, and then call the SetResult() method of the resty will automatically unmarshal the response data to the incoming object. Here, the chain call method is used when setting the request information, that is, multiple settings are completed in one line.

run:

$ go run main.go
4040 libraries
first library:
name:vue latest:https://cdnjs.cloudflare.com/ajax/libs/vue/3.1.2/vue.min.js

There are a total of 4040 libraries, the first one is Vue✌️. We request https://api.cdnjs.com/libraries/vue to get the detailed information of Vue:

If you are interested, you can use resty to pull this information.

Under general request, resty will infer the data format Content-Type in the response. But sometimes there is no Content-Type header in the response or inconsistent with the content format, we can force resty to parse the response in a specific format ForceContentType()

client.R().
  SetResult(result).
  ForceContentType("application/json")

Request information

resty provides a wealth of methods for setting request information. We can set the query string in two ways. One is to call SetQueryString() request object to set the query string we have spliced:

client.R().
  SetQueryString("name=dj&age=18").
  Get(...)

The other is to call SetQueryParams() request object, pass in map[string]string , and resty will help us stitch together. Obviously this is more convenient:

client.R().
  SetQueryParams(map[string]string{
    "name": "dj",
    "age": "18",
  }).
  Get(...)

resty also provides a very practical interface for setting path parameters. We call SetPathParams() pass in the map[string]string parameter, and then the key in map can be used in the URL path later:

client.R().
  SetPathParams(map[string]string{
    "user": "dj",
  }).
  Get("/v1/users/{user}/details")

Note that the keys in the path need to be wrapped {}

Set the header:

client.R().
  SetHeader("Content-Type", "application/json").
  Get(...)

Set the request message body:

client.R().
  SetHeader("Content-Type", "application/json").
  SetBody(`{"name": "dj", "age":18}`).
  Get(...)

The message body can be of multiple types: string, []byte , object, map[string]interface{} etc.

Set to carry the Content-Length header, resty automatically calculated:

client.R().
  SetBody(User{Name:"dj", Age:18}).
  SetContentLength(true).
  Get(...)

Some websites need to obtain the token before they can access its API. Set the token:

client.R().
  SetAuthToken("youdontknow").
  Get(...)

Case study

Finally, we use a case to string together the above introductions. Now we want to get the organization's warehouse information through the API provided by GitHub, API document see the link after the article. The GitHub API request address is https://api.github.com , and the request format for obtaining warehouse information is as follows:

GET /orgs/{org}/repos

We can also set the following parameters:

• accept : header , this is required, it needs to be set to application/vnd.github.v3+json ;
• org : organization name, path parameter ;
• type : warehouse type, query parameter , such as public/private/forks (fork warehouse), etc.;
• sort : the sorting rule of the warehouse, query parameter , such as created/updated/pushed/full_name and so on. Sort by creation time by default;
• direction : ascending asc or descending dsc , query parameter ;
• per_page : How many entries per page, the maximum is 100, the default is 30, query parameter ;
• page : The current page number of the request, which is per_page for paging management together with 060de570a7ea48. The default is 1, query parameter .

To access the GitHub API, a token must be set. Log in to your GitHub account, click on the avatar in the upper right corner, and select Settings :

Then, select Developer settings :

Select Personal access tokens , and then click Generate new token upper right corner:

Fill in the Note to indicate the purpose of the token. Just fill in this according to your own situation. The following checkboxes are used to select which permissions the token has, there is no need to check them here:

Click the Generate token button below to generate a token:

Note that this token can only be seen now, and you will not be able to see it next time you close the page. So save it, and don’t use my token. I will delete the token after testing the program😭.

The JSON format data in the response is as follows:

There are many fields. For convenience, I will deal with a few fields here:

type Repository struct {
  ID              int        `json:"id"`
  NodeID          string     `json:"node_id"`
  Name            string     `json:"name"`
  FullName        string     `json:"full_name"`
  Owner           *Developer `json:"owner"`
  Private         bool       `json:"private"`
  Description     string     `json:"description"`
  Fork            bool       `json:"fork"`
  Language        string     `json:"language"`
  ForksCount      int        `json:"forks_count"`
  StargazersCount int        `json:"stargazers_count"`
  WatchersCount   int        `json:"watchers_count"`
  OpenIssuesCount int        `json:"open_issues_count"`
}

type Developer struct {
  Login      string `json:"login"`
  ID         int    `json:"id"`
  NodeID     string `json:"node_id"`
  AvatarURL  string `json:"avatar_url"`
  GravatarID string `json:"gravatar_id"`
  Type       string `json:"type"`
  SiteAdmin  bool   `json:"site_admin"`
}

Then use resty set path parameters, query parameters, headers, Token and other information, and then initiate a request:

func main() {
  client := resty.New()

  var result []*Repository
  client.R().
    SetAuthToken("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").
    SetHeader("Accept", "application/vnd.github.v3+json").
    SetQueryParams(map[string]string{
      "per_page":  "3",
      "page":      "1",
      "sort":      "created",
      "direction": "asc",
    }).
    SetPathParams(map[string]string{
      "org": "golang",
    }).
    SetResult(&result).
    Get("https://api.github.com/orgs/{org}/repos")

  for i, repo := range result {
    fmt.Printf("repo%d: name:%s stars:%d forks:%d\n", i+1, repo.Name, repo.StargazersCount, repo.ForksCount)
  }
}

The above program pulls 3 warehouses in ascending order of creation time:

$ go run main.go
repo1: name:gddo stars:1097 forks:289
repo2: name:lint stars:3892 forks:518
repo3: name:glog stars:2738 forks:775

Trace

After introducing resty , let's take a look at an auxiliary function provided by resty EnableTrace() method on the request object to enable trace. Enabling trace can record the time-consuming and other information of each step of the request. resty supports chained calls, which means that we can complete the creation request in one line, enable trace, and initiate a request :

client.R().EnableTrace().Get("https://baidu.com")

After completing the request, we get the information TraceInfo()

ti := resp.Request.TraceInfo()
fmt.Println("Request Trace Info:")
fmt.Println("DNSLookup:", ti.DNSLookup)
fmt.Println("ConnTime:", ti.ConnTime)
fmt.Println("TCPConnTime:", ti.TCPConnTime)
fmt.Println("TLSHandshake:", ti.TLSHandshake)
fmt.Println("ServerTime:", ti.ServerTime)
fmt.Println("ResponseTime:", ti.ResponseTime)
fmt.Println("TotalTime:", ti.TotalTime)
fmt.Println("IsConnReused:", ti.IsConnReused)
fmt.Println("IsConnWasIdle:", ti.IsConnWasIdle)
fmt.Println("ConnIdleTime:", ti.ConnIdleTime)
fmt.Println("RequestAttempt:", ti.RequestAttempt)
fmt.Println("RemoteAddr:", ti.RemoteAddr.String())

We can obtain the following information:

• DNSLookup : DNS query time. If you provide a domain name instead of an IP, you need to query the DNS system for the corresponding IP before proceeding;
• ConnTime consuming to obtain a connection, it may be obtained from the connection pool, or it may be created;
• TCPConnTime : TCP connection is time-consuming, from the end of the DNS query to the establishment of the TCP connection;
• TLSHandshake : TLS handshake takes time;
• ServerTime : server processing time-consuming, calculating the time interval from the connection establishment to the client receiving the first byte;
• ResponseTime : The response is time-consuming, the time interval from receiving the first response byte to receiving the complete response;
• TotalTime : the time consuming of the whole process;
• IsConnReused : Whether the TCP connection is reused;
• IsConnWasIdle : Whether the connection is obtained from an idle connection pool;
• ConnIdleTime : connection idle time;
• RequestAttempt : The number of requests in the request execution process, including the number of retries;
• RemoteAddr : The remote service address, in IP:PORT format.

resty distinguishes these very finely. In fact, resty also uses the functions provided by the net/http/httptrace httptrace provides a structure where we can set callback functions at various stages:

// src/net/http/httptrace.go
type ClientTrace struct {
  GetConn func(hostPort string)
  GotConn func(GotConnInfo)
  PutIdleConn func(err error)
  GotFirstResponseByte func()
  Got100Continue func()
  Got1xxResponse func(code int, header textproto.MIMEHeader) error // Go 1.11
  DNSStart func(DNSStartInfo)
  DNSDone func(DNSDoneInfo)
  ConnectStart func(network, addr string)
  ConnectDone func(network, addr string, err error)
  TLSHandshakeStart func() // Go 1.8
  TLSHandshakeDone func(tls.ConnectionState, error) // Go 1.8
  WroteHeaderField func(key string, value []string) // Go 1.11
  WroteHeaders func()
  Wait100Continue func()
  WroteRequest func(WroteRequestInfo)
}

You can simply understand the meaning of the callback from the field name. resty sets the following callback after enabling trace:

// src/github.com/go-resty/resty/trace.go
func (t *clientTrace) createContext(ctx context.Context) context.Context {
  return httptrace.WithClientTrace(
    ctx,
    &httptrace.ClientTrace{
      DNSStart: func(_ httptrace.DNSStartInfo) {
        t.dnsStart = time.Now()
      },
      DNSDone: func(_ httptrace.DNSDoneInfo) {
        t.dnsDone = time.Now()
      },
      ConnectStart: func(_, _ string) {
        if t.dnsDone.IsZero() {
          t.dnsDone = time.Now()
        }
        if t.dnsStart.IsZero() {
          t.dnsStart = t.dnsDone
        }
      },
      ConnectDone: func(net, addr string, err error) {
        t.connectDone = time.Now()
      },
      GetConn: func(_ string) {
        t.getConn = time.Now()
      },
      GotConn: func(ci httptrace.GotConnInfo) {
        t.gotConn = time.Now()
        t.gotConnInfo = ci
      },
      GotFirstResponseByte: func() {
        t.gotFirstResponseByte = time.Now()
      },
      TLSHandshakeStart: func() {
        t.tlsHandshakeStart = time.Now()
      },
      TLSHandshakeDone: func(_ tls.ConnectionState, _ error) {
        t.tlsHandshakeDone = time.Now()
      },
    },
  )
}

Then when obtaining TraceInfo , calculate the time-consuming according to each time point:

// src/github.com/go-resty/resty/request.go
func (r *Request) TraceInfo() TraceInfo {
  ct := r.clientTrace

  if ct == nil {
    return TraceInfo{}
  }

  ti := TraceInfo{
    DNSLookup:      ct.dnsDone.Sub(ct.dnsStart),
    TLSHandshake:   ct.tlsHandshakeDone.Sub(ct.tlsHandshakeStart),
    ServerTime:     ct.gotFirstResponseByte.Sub(ct.gotConn),
    IsConnReused:   ct.gotConnInfo.Reused,
    IsConnWasIdle:  ct.gotConnInfo.WasIdle,
    ConnIdleTime:   ct.gotConnInfo.IdleTime,
    RequestAttempt: r.Attempt,
  }

  if ct.gotConnInfo.Reused {
    ti.TotalTime = ct.endTime.Sub(ct.getConn)
  } else {
    ti.TotalTime = ct.endTime.Sub(ct.dnsStart)
  }

  if !ct.connectDone.IsZero() {
    ti.TCPConnTime = ct.connectDone.Sub(ct.dnsDone)
  }

  if !ct.gotConn.IsZero() {
    ti.ConnTime = ct.gotConn.Sub(ct.getConn)
  }

  if !ct.gotFirstResponseByte.IsZero() {
    ti.ResponseTime = ct.endTime.Sub(ct.gotFirstResponseByte)
  }

  if ct.gotConnInfo.Conn != nil {
    ti.RemoteAddr = ct.gotConnInfo.Conn.RemoteAddr()
  }

  return ti
}

Run output:

$ go run main.go
Request Trace Info:
DNSLookup: 2.815171ms
ConnTime: 941.635171ms
TCPConnTime: 269.069692ms
TLSHandshake: 669.276011ms
ServerTime: 274.623991ms
ResponseTime: 112.216µs
TotalTime: 1.216276906s
IsConnReused: false
IsConnWasIdle: false
ConnIdleTime: 0s
RequestAttempt: 1
RemoteAddr: 18.235.124.214:443

We see that TLS consumes nearly half of the time.

to sum up

In this article, I introduced a very convenient and easy-to-use HTTP Client library in the Go language. resty provides a very useful and rich API. Chain call, automatic Unmarshal, and request parameter/path setting are very convenient and easy to use, making our work more effective. Due to space limitations, many advanced features have not been introduced one by one, such as submitting forms, uploading files, and so on. It can only be left for those who are interested to explore.

If you find a fun and useful Go language library, welcome to submit an issue on the Go Daily Library GitHub😄

reference

1. Go a library GitHub every day: https://github.com/darjun/go-daily-lib
2. resty GitHub：github.com/go-resty/resty
3. GitHub API：https://docs.github.com/en/rest/overview/resources-in-the-rest-api

I

My blog: https://darjun.github.io

Welcome to follow my WeChat public account [GoUpUp], learn together and make progress together~