Release limited resources to avoid leakage

This article is a translation of the book "100 Go Mistackes: How to Avoid Them". Due to the limited level of translation, it is inevitable that there will be translation accuracy problems, please understand. Follow the public account "Go School" for more series of articles

As we all know, computer resources (memory, disk) are limited. When programming, these resources must be closed and released somewhere in the code to avoid leakage due to insufficient resources. However, developers often neglect to close the opened resources when writing code, which leads to abnormalities in the program due to insufficient resources.

This article mainly introduces that in Go, all structures that implement the io.Closer interface must eventually be closed to release resources.

The following example is a getBody function that constructs an HTTP GET request and processes the resulting HTTP response.

The following is the implementation of the first version:

func getBody(url string) (string, error) {
    resp, err := http.Get(url)
    if err != nil {
        return "", err
    }
    body, err := ioutil.ReadAll(resp.Body) ①
    if err != nil {
        return "", err
    }
    return string(body), nil
}

① Read resp.Body and convert it into a byte array []byte

We used the http.Get method, and then we used ioutil.ReadAll to parse the response value. The function of this function looks normal. At least, it returned the HTTP response correctly.
However, there is a problem of resource leakage. Let us see where .

resp is a *http.Response pointer type. It contains an io.ReaderCloser field (io.ReadCloser contains both io.Reader interface and io.Closer interface). If http.Get does not return an error, the field must be closed. Otherwise, it will cause resource leakage. It takes up some memory, which is no longer needed after the function is executed, but it cannot be reclaimed by the GC because it has not actively released resources. At the same time, the client cannot reuse the TCP connection when resources are scarce.

The most convenient way to deal with the closure of the subject is to use the defer statement:

func getBody(url string) (string, error) {
    resp, err := http.Get(url)
    if err != nil {
        return "", err
    }
    defer resp.Body.Close() ①
    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        return "", err
    }
    return string(body), nil
}

① If http.Get does not return an error, we will use defer to close the response value.

In this implementation, we use a deferred function (defer) to correctly handle the closed return resource, so that the delayed close statement will be executed once the getBody function returns.

Note: We should message resp.Body.Close() to return an error. We will see how to handle errors in delayed functions in the chapter on error management. In this example and subsequent examples, we will ignore the error for now.

What we should pay attention to is that no matter whether we read the content from response.Body, we need to close the response resource .

For example, in the function below we only return the HTTP status code. However, the response body must also be closed:

func getStatusCode(url string) (int, error) {
    resp, err := http.Get(url)
    if err != nil {
        return 0, err
    }
    defer resp.Body.Close() ①
    return resp.StatusCode, nil
}

① Even if the content is not read, the response body needs to be closed.

We should ensure that resources are released at the right moment. For example, if you delay calling resp.Body.Close() regardless of the type of error:

func getStatusCode(url string) (int, error) {
    resp, err := http.Get(url)
    defer resp.Body.Close() ①
    if err != nil {
        return 0, err
    }
    
    return resp.StatusCode, nil
}

① At this stage, resp may be nil

Because resp may be nil, this code may cause the program to panic:

panic: runtime error: invalid memory address or nil pointer dereference

One last thing to note about the closing of the HTTP request body. A very rare situation is to close the response if the response is empty instead of nil:

resp, err := http.Get(url)
if resp != nil { ①
    defer resp.Body.Close() ②
}
if err != nil {
    return "", err
}

① If response is not nil

② Close the response body as a delay function

The implementation makes it wrong. The implementation relies on some conditions (for example, redirection failure), and neither resp nor err is nil.

However, according to the official Go document: When an error occurs, anything can be ignored . A non-nil response that returns a non-nil error will only appear when CheckRedirect fails, however, the Response.Body returned at this time has been closed.

Therefore, the check statement of if resp != nil {} is unnecessary. We should stick to the original solution and only close the body in the deferred function if there are no errors.

Note: On the server side, when implementing an HTTP handler, there is no need to close the request, because it will be automatically closed by the server.

Closing resources to avoid leakage is not only related to the HTTP response body. Generally speaking, all structures that implement the io.Closer interface need to be closed at some point . This interface contains the only Close method:

type Closer interface {
    Close() error
}

Let's look at some other examples of resources that need to be closed to avoid leaks:

2.9.1 sql.Rows

sql.Rows is a structure used for sql query results. Because this structure implements the io.Closer interface, it must be closed. We can also use the delay function to handle the shutdown logic as follows:

db, err := sql.Open("postgres", dataSourceName) ①
if err != nil {
    return err
}
rows, err := db.Query("SELECT * FROM MYTABLE") ②
if err != nil {
    return err
}
defer rows.Close() ③

// Use rows

① Create a SQL connection

② Execute a SQL query

③ Close rows

If the Query call does not return an error, then we need to close rows in time.

2.9.2 os.File

os.File represents an open file identifier. Like sql.Rows, it should be closed eventually:

f, err := os.Open("events.log") ①
if err != nil {
    return err
}
defer f.Close() ②

// Use file descriptor

① Open the file

② Close the file identifier

When the function block returns, we use defer again to schedule the Close method.

Note: The file being closed does not guarantee that the content of the file has been written to the disk. In fact, write
The imported content may be left in the buffer of the file system and has not been flushed to the disk. if
Persistence is a key factor, we should use the Sync() method to refresh the contents of the buffer
To the disk.

2.9.3 Compression implementation

The compressed write and read implementations also need to be turned off. In fact, the internal buffer they created also needs to be released manually. For example: gzip.Writer.

var b bytes.Buffer ①
w := gzip.NewWriter(&b) ②

defer w.Close() ③

① Create a buffer

② Create a new gzip writer

③ Close gzip.Writer

gzip.Reader has the same logic:

var b bytes.Buffer ①
r, err := gzip.NewReader(&b) ②
if err != nil {
    return nil, err
}

defer r.Close() ③

① Create a buffer

② Create a new gzip writer

③ Close gzip.Writer

summary

We have seen how important it is to shut down limited resources to avoid leakage. Limited resources must be shut down at the right time and in specific scenarios. Sometimes, it is not clear whether resources are needed. We can only decide by reading the relevant API documentation or actual practice. However, we must be careful, if a structure implements the io.Closer interface, we must call the Close method at the end.