Super full summary: how the Go language manipulates files

foreword

Hello everyone, my name is asong .

We all know that in Unix everything is called a file, and file processing is a very common problem, so this article summarizes Go the common way of language manipulation files, the overall idea is as follows :

Go language version: 1.18

All the code in this article has been uploaded github : https://github.com/asong2020/Golang_Dream/tree/master/code_demo/file_operate_demo

What does the operation file include?

Operating a file is inseparable from these actions:

• Create a file
• open a file
• read file
• write to file
• close file
• pack/unpack
• compress/decompress
• change file permissions
• Delete Files
• move files
• rename file
• empty file

Therefore, this article summarizes some example methods for these operations for your reference;

Libraries that can be used to manipulate files in Go

Go language official library: os , io/ioutil , bufio covers all scenarios of file operations,

os provides a method to directly call the file IO , bufio provides a method to operate the file in the buffer, IO io/ioutil IO ，不过Go语言Go1.16版本io/ioutil库， io/ioutil包是一个定义不明确and incomprehensible collection of things. All the functions provided by this package have been moved to other packages, so the methods for manipulating files in ---077d70e78082270ae4eabe93f6ca76ad io/ioutil are all in io the library has the same meaning, you will use ioutil in the future The methods in can find corresponding methods in other packages through annotations.

Basic operations on files

Here I classify creating a file, opening a file, closing a file, and changing file permissions as the basic operations on the file, and the basic operations on the file directly use the methods in the library os , because we need to IO operation, see the following example:

 import (
    "log"
    "os"
)
func main() {
    // 创建文件
    f, err := os.Create("asong.txt")
    if err != nil{
        log.Fatalf("create file failed err=%s\n", err)
    }
    // 获取文件信息
    fileInfo, err := f.Stat()
    if err != nil{
        log.Fatalf("get file info failed err=%s\n", err)
    }

    log.Printf("File Name is %s\n", fileInfo.Name())
    log.Printf("File Permissions is %s\n", fileInfo.Mode())
    log.Printf("File ModTime is %s\n", fileInfo.ModTime())

    // 改变文件权限
    err = f.Chmod(0777)
    if err != nil{
        log.Fatalf("chmod file failed err=%s\n", err)
    }

    // 改变拥有者
    err = f.Chown(os.Getuid(), os.Getgid())
    if err != nil{
        log.Fatalf("chown file failed err=%s\n", err)
    }

    // 再次获取文件信息 验证改变是否正确
    fileInfo, err = f.Stat()
    if err != nil{
        log.Fatalf("get file info second failed err=%s\n", err)
    }
    log.Printf("File change Permissions is %s\n", fileInfo.Mode())

    // 关闭文件
    err = f.Close()
    if err != nil{
        log.Fatalf("close file failed err=%s\n", err)
    }
    
    // 删除文件
    err = os.Remove("asong.txt")
    if err != nil{
        log.Fatalf("remove file failed err=%s\n", err)
    }
}

write file

Quick write file

os / ioutil packages provide WriteFile methods can quickly handle creating/opening files/writing data/closing files, examples are as follows:

 func writeAll(filename string) error {
    err := os.WriteFile("asong.txt", []byte("Hi asong\n"), 0666)
    if err != nil {
        return err
    }
    return nil
}

write file line by line

os 、 buffo写数据都没有提供按行写入的方法，所以我们可以在调用os.WriteString 、 bufio.WriteString You can add line breaks to the data, see an example:

 import (
    "bufio"
    "log"
    "os"
)
// 直接操作IO
func writeLine(filename string) error {
    data := []string{
        "asong",
        "test",
        "123",
    }
    f, err := os.OpenFile(filename, os.O_WRONLY, 0666)
    if err != nil{
        return err
    }

    for _, line := range data{
        _,err := f.WriteString(line + "\n")
        if err != nil{
            return err
        }
    }
    f.Close()
    return nil
}
// 使用缓存区写入
func writeLine2(filename string) error {
    file, err := os.OpenFile(filename, os.O_WRONLY, 0666)
    if err != nil {
        return err
    }

    // 为这个文件创建buffered writer
    bufferedWriter := bufio.NewWriter(file)
    
    for i:=0; i < 2; i++{
        // 写字符串到buffer
        bytesWritten, err := bufferedWriter.WriteString(
            "asong真帅\n",
        )
        if err != nil {
            return err
        }
        log.Printf("Bytes written: %d\n", bytesWritten)
    }
    // 写内存buffer到硬盘
    err = bufferedWriter.Flush()
    if err != nil{
        return err
    }

    file.Close()
    return nil
}

offset write

In some scenarios, we want to write data according to a given offset, we can use the os writeAt , the example is as follows:

 import "os"

func writeAt(filename string) error {
    data := []byte{
        0x41, // A
        0x73, // s
        0x20, // space
        0x20, // space
        0x67, // g
    }
    f, err := os.OpenFile(filename, os.O_WRONLY, 0666)
    if err != nil{
        return err
    }
    _, err = f.Write(data)
    if err != nil{
        return err
    }

    replaceSplace := []byte{
        0x6F, // o
        0x6E, // n
    }
    _, err = f.WriteAt(replaceSplace, 2)
    if err != nil{
        return err
    }
    f.Close()
    return nil
}

buffer write

os库中的方法IO操作， IO操作会增加CPU ，所以We can use the memory buffer to reduce the IO operation, use the memory buffer before writing bytes to the hard disk, and write the memory data buffer to the hard disk when the capacity of the memory buffer reaches a certain value, bufio This is how a library is shown. Let's take an example to see how to use it:

 import (
    "bufio"
    "log"
    "os"
)

func writeBuffer(filename string) error {
    file, err := os.OpenFile(filename, os.O_WRONLY, 0666)
    if err != nil {
        return err
    }

    // 为这个文件创建buffered writer
    bufferedWriter := bufio.NewWriter(file)

    // 写字符串到buffer
    bytesWritten, err := bufferedWriter.WriteString(
        "asong真帅\n",
    )
    if err != nil {
        return err
    }
    log.Printf("Bytes written: %d\n", bytesWritten)

    // 检查缓存中的字节数
    unflushedBufferSize := bufferedWriter.Buffered()
    log.Printf("Bytes buffered: %d\n", unflushedBufferSize)

    // 还有多少字节可用（未使用的缓存大小）
    bytesAvailable := bufferedWriter.Available()
    if err != nil {
        return err
    }
    log.Printf("Available buffer: %d\n", bytesAvailable)
    // 写内存buffer到硬盘
    err = bufferedWriter.Flush()
    if err != nil{
        return err
    }

    file.Close()
    return nil
}

read file

read full file

There are two ways we can read the whole file:

• os , io/ioutil provide the readFile method to quickly read the full text
• io/ioutil provides the ReadAll method to read the full text after opening the file handle;

 import (
    "io/ioutil"
    "log"
    "os"
)

func readAll(filename string) error {
    data, err := os.ReadFile(filename)
    if err != nil {
        return err
    }
    log.Printf("read %s content is %s", filename, data)
    return nil
}

func ReadAll2(filename string) error {
    file, err := os.Open("asong.txt")
    if err != nil {
        return err
    }

    content, err := ioutil.ReadAll(file)
    log.Printf("read %s content is %s\n", filename, content)

    file.Close()
    return nil
}

read line by line

os The library provides Read method is to read according to the byte length. If we want to read the file by line, we need to use it together with bufio , bufio中ReadLine 、 ReadBytes("\n") 、 ReadString("\n")按行读取数据，下面我使用ReadBytes("\n") to write an example:

 func readLine(filename string) error {
    file, err := os.OpenFile(filename, os.O_RDONLY, 0666)
    if err != nil {
        return err
    }
    bufferedReader := bufio.NewReader(file)
    for {
        // ReadLine is a low-level line-reading primitive. Most callers should use
        // ReadBytes('\n') or ReadString('\n') instead or use a Scanner.
        lineBytes, err := bufferedReader.ReadBytes('\n')
        bufferedReader.ReadLine()
        line := strings.TrimSpace(string(lineBytes))
        if err != nil && err != io.EOF {
            return err
        }
        if err == io.EOF {
            break
        }
        log.Printf("readline %s every line data is %s\n", filename, line)
    }
    file.Close()
    return nil
}

Read file in chunks

In some scenarios, we want to read the file according to the byte length, then we can do the following:

• os library Read method
• os Library cooperation bufio.NewReader call Read method
• os Library with io Library ReadFull , ReadAtLeast method

 // use bufio.NewReader
func readByte(filename string) error {
    file, err := os.OpenFile(filename, os.O_RDONLY, 0666)
    if err != nil {
        return err
    }
    // 创建 Reader
    r := bufio.NewReader(file)

    // 每次读取 2 个字节
    buf := make([]byte, 2)
    for {
        n, err := r.Read(buf)
        if err != nil && err != io.EOF {
            return err
        }

        if n == 0 {
            break
        }
        log.Printf("writeByte %s every read 2 byte is %s\n", filename, string(buf[:n]))
    }
    file.Close()
    return nil
}

// use os
func readByte2(filename string) error{
    file, err := os.OpenFile(filename, os.O_RDONLY, 0666)
    if err != nil {
        return err
    }

    // 每次读取 2 个字节
    buf := make([]byte, 2)
    for {
        n, err := file.Read(buf)
        if err != nil && err != io.EOF {
            return err
        }

        if n == 0 {
            break
        }
        log.Printf("writeByte %s every read 2 byte is %s\n", filename, string(buf[:n]))
    }
    file.Close()
    return nil
}


// use os and io.ReadAtLeast
func readByte3(filename string) error{
    file, err := os.OpenFile(filename, os.O_RDONLY, 0666)
    if err != nil {
        return err
    }

    // 每次读取 2 个字节
    buf := make([]byte, 2)
    for {
        n, err := io.ReadAtLeast(file, buf, 0)
        if err != nil && err != io.EOF {
            return err
        }

        if n == 0 {
            break
        }
        log.Printf("writeByte %s every read 2 byte is %s\n", filename, string(buf[:n]))
    }
    file.Close()
    return nil
}

delimiter read

bufio The package provides Scanner scanner module, its main function is to split the data stream into tags and remove the spaces between them, he supports our customization Split function is used as a separation function. The delimiter may not be a simple byte or character. We can customize the separation function, implement the separation rules in the separation function and how much the pointer moves, and what data is returned, if not customized Split函数，那么ScanLines ，也就是使用换行作为分隔符， bufio中ScanRunes , ScanWrods , let's use the SacnWrods method to write an example to get the text separated by spaces:

 func readScanner(filename string) error {
    file, err := os.OpenFile(filename, os.O_RDONLY, 0666)
    if err != nil {
        return err
    }

    scanner := bufio.NewScanner(file)
    // 可以定制Split函数做分隔函数
    // ScanWords 是scanner自带的分隔函数用来找空格分隔的文本字
    scanner.Split(bufio.ScanWords)
    for {
        success := scanner.Scan()
        if success == false {
            // 出现错误或者EOF是返回Error
            err = scanner.Err()
            if err == nil {
                log.Println("Scan completed and reached EOF")
                break
            } else {
                return err
            }
        }
        // 得到数据，Bytes() 或者 Text()
        log.Printf("readScanner get data is %s", scanner.Text())
    }
    file.Close()
    return nil
}

pack/unpack

Go language archive package provides tar , zip two packing/unpacking methods, here are zip package as an example:

zip Unpack example:

 import (
    "archive/zip"
    "fmt"
    "io"
    "log"
    "os"
)

func main()  {
    // Open a zip archive for reading.
    r, err := zip.OpenReader("asong.zip")
    if err != nil {
        log.Fatal(err)
    }
    defer r.Close()
    // Iterate through the files in the archive,
    // printing some of their contents.
    for _, f := range r.File {
        fmt.Printf("Contents of %s:\n", f.Name)
        rc, err := f.Open()
        if err != nil {
            log.Fatal(err)
        }
        _, err = io.CopyN(os.Stdout, rc, 68)
        if err != nil {
            log.Fatal(err)
        }
        rc.Close()
    }
}

zip Package example:

 func writerZip()  {
    // Create archive
    zipPath := "out.zip"
    zipFile, err := os.Create(zipPath)
    if err != nil {
        log.Fatal(err)
    }

    // Create a new zip archive.
    w := zip.NewWriter(zipFile)
    // Add some files to the archive.
    var files = []struct {
        Name, Body string
    }{
        {"asong.txt", "This archive contains some text files."},
        {"todo.txt", "Get animal handling licence.\nWrite more examples."},
    }
    for _, file := range files {
        f, err := w.Create(file.Name)
        if err != nil {
            log.Fatal(err)
        }
        _, err = f.Write([]byte(file.Body))
        if err != nil {
            log.Fatal(err)
        }
    }
    // Make sure to check the error on Close.
    err = w.Close()
    if err != nil {
        log.Fatal(err)
    }
}

Summarize

本文归根结底是介绍os 、 io 、 bufio这些包如何操作文件，因为Go语言操作提供了太多了方法， By introducing all of this article, it can be easily used as a document query when using it. If you ask what method is the best way to operate the file, I can't answer this for you. It needs to be analyzed according to the specific scene. You know all these methods, just write a benchmark and compare them. Practice is the only criterion for testing the truth.

All the codes in this article have been uploaded github : https://github.com/asong2020/Golang_Dream/tree/master/code_demo/file_operate_demo

Well, this article ends here, I'm asong , see you next time.

Welcome to the public account: Golang Dream Factory