Created a new kv storage engine in Go language

After about 4 months of polishing, the first release version of LotusDB was finally released. I read it, and there were more than 200 commits (close to the number of commits of rosedb for more than a year).

project address: https://github.com/flower-corp/lotusdb

After the practice of rosedb on the bitcask model and my accumulation of experience in storage, at the end of last year, I suddenly thought of an idea on the way to work, which gave me the idea of making a new kv storage engine.

After I have an idea, it is verified, because I still have no idea in my heart, I looked it up on Github again, and there is no implementation of the same type. Later, I found some big brothers to communicate with them and proved that my idea is feasible.

During this period, I also found a paper on optimizing LSM on Usenix Fast. I found that the content of the paper is very similar to my idea. This is another theoretical basis, so I decided to start.

Interested can refer to the following paper, called SLM-DB, address: https://www.usenix.org/conference/fast19/presentation/kaiyrakhmet

As we all know, the two most mainstream models of data storage engines are B+ tree and LSM tree. B+ tree is widely used in relational databases such as Mysql, and rocksdb, the typical representative of LSM, is also the first choice for most distributed system data storage. .

The read performance of the B+ tree is stable, and the write throughput of the LSM is high. LotusDB has made a huge change on this basis, which is to completely abandon the SST file in the LSM and use the B+ tree to store the index, and the value storage refers to Wisckey and The design of the bitcask model is stored in a separate value log file.

LotusDB is a combination of the advantages of LSM and B+ tree. There is no implementation of the same type at present. We should be the first to eat crabs.

The architecture diagram of LotusDB is as follows:

The writing of the foreground is exactly the same as that of LSM, first write wal and then write memtable.

The reading will start from the memtable. If the memtable is found, it will return directly; if it is not found, the index will be queried from the B+ tree, and then the value will be obtained from the value log according to the index information.

You can get a rough idea first, and then I will publish a complete series of articles "LotusDB Design and Implementation" , which comprehensively analyzes the architectural details and code implementation of LotusDB. I have already written several articles to be published. Welcome to the follow-up of the official account. renew:

Let's take a look at some basic interfaces provided by LotusDB. At present, it implements the basic Put, Get, and Delete interfaces, and supports Column Family (referenced from rocksdb), as well as automatic GC recovery of value log.

The simple usage is as follows:

package main

import (
    "github.com/flower-corp/lotusdb"
    "io/ioutil"
    "time"
)

// basic operations for LotusDB:
// put
// put with options
// get
// delete
// delete with options
func main() {
    path, _ := ioutil.TempDir("", "lotusdb")
    opts := lotusdb.DefaultOptions(path)
    db, err := lotusdb.Open(opts)
    if err != nil {
        panic(err)
    }
    defer db.Close()

    // 1.----put----
    key1 := []byte("name")
    err = db.Put(key1, []byte("lotusdb"))
    if err != nil {
        // ...
    }

    key2 := []byte("feature")
    // 2.----put with options----
    writeOpts := &lotusdb.WriteOptions{
        Sync:      true,
        ExpiredAt: time.Now().Add(time.Second * 100).Unix(),
    }
    err = db.PutWithOptions(key2, []byte("store data"), writeOpts)
    if err != nil {
        // ...
    }

    // 3.----get----
    val, err := db.Get(key1)
    if err != nil {
        // ...
    }
    if len(val) > 0 {
        // ...
    }

    // 4.----delete----
    err = db.Delete(key1)
    if err != nil {
        // ...
    }

    // 5.----delete with options----
    deleteOpts := &lotusdb.WriteOptions{
        Sync:       false,
        DisableWal: true,
    }
    err = db.DeleteWithOptions([]byte("dummy key"), deleteOpts)
    if err != nil {
        // ...
    }
}

At present, I think that the code quality of LotusDB is much better than the previous RoseDB. The unit tests are more complete, the comments are clear, and the code is more concise and standardized. If you are a Go novice, or are planning to learn Go, you can also use the project as an exercise material and compare it yourself. to learn.

Of course, our vision is to build a storage engine that can actually be implemented in the production environment. The current version is just the beginning, and there will be a lot of work in the future, including but not limited to:

• batch operation to ensure atomicity
• Multiple Column Family guarantees atomicity
• SSI based transactions
• Iterator iterator
• data compression
• data backup
• split of index

If you have any questions in the process of using or learning, you can contact me on WeChat:

If you find it helpful, welcome to give the project a star and support !

Project address: https://github.com/flower-corp/lotusdb