Talking about distributed unique id, this article is very real

Introduction to Distributed Unique ID

The globally unique id of a distributed system is a scenario that all systems will encounter. It is often used in search and storage, as a unique identification or sorting, such as the globally unique order number, coupon code of coupons, etc. The appearance of two identical order numbers will undoubtedly be a huge bug for users.

In a single system, there is no challenge to generate a unique id, because there is only one machine and one application, and you can directly use a singleton plus an atomic operation to auto-increment. In a distributed system, for different applications, different computer rooms, and different machines, if the IDs generated are all unique, it really takes some effort.

One sentence summary:

distributed unique ID is to uniquely identify the data.

Characteristics of Distributed Unique ID

The core of distributed unique ID is uniqueness, and the others are additional attributes. Generally speaking, an excellent global unique ID scheme has the following characteristics, for reference only:

• The only one in the whole world: it cannot be repeated, the core feature!
• Roughly orderly or monotonically increasing: The self-increasing feature is conducive to search, sort, or range query, etc.
• High performance: Generate ID response quickly and low latency
• High availability: If you can only use a stand-alone machine and hang up, all the services that rely on the global unique ID for the entire company will be unavailable, so the service that generates ID must be highly available
• Convenient to use: friendly to the access person, it is best to be packaged out of the box
• Information security: In some scenarios, if they are continuous, it is easy to guess and attacks are possible. There is a trade-off.

Distributed unique ID generation scheme

UUID generated directly

Friends who have written Java know that sometimes we will use a UUID to write logs, and a random ID will be generated to serve as the unique identification code for the current user's request record. Just use the following code:

String uuid = UUID.randomUUID();

The usage is simple and rude. The full name of UUID is actually Universally Unique IDentifier , or GUID(Globally Unique IDentifier) , which is essentially a 128-bit binary integer, usually we will represent it as a string of 32 hexadecimal numbers, almost never repeated, 2 of 128 times Fang, that is a huge number.

The following is a description of Baidu Baike:

UUID consists of a combination of the following parts:

(1) The first part of the UUID is related to time. If you generate a UUID after a few seconds after generating a UUID, the first part is different, and the rest are the same.

(2) Clock sequence.

(3) The globally unique IEEE machine identification number. If there is a network card, it is obtained from the MAC address of the network card, and if there is no network card, it is obtained by other means.

The only drawback of UUID is that the generated result string will be relatively long. The most commonly used standard for UUID is Microsoft's GUID (Globals Unique Identifiers). In ColdFusion, you can use the CreateUUID() function to easily generate UUIDs. The format is: xxxxxxxx-xxxx- xxxx-xxxxxxxxxxxxxxxx(8-4-4-16), where each x is one in the range of 0-9 or af Hexadecimal numbers. The standard UUID format is: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx (8-4-4-4-12), you can download CreateGUID() UDF from cflib for conversion. [2]

(4) In hibernate (Java orm framework), use IP-JVM startup time-current time right shifted 32 bits-current time-internal count (8-8-4-8-4) to form UUID

If you want to repeat, if two identical virtual machines have the same boot time, the same random seed, and the uuid is generated at the same time, there is a very small probability of repeating. Therefore, we can think that it will repeat in theory, but it is impossible to repeat in practice! ! !

Advantages of uuid:

• Good performance and high efficiency
• No network request, directly generated locally
• Different machines do it individually and will not repeat

uuid is so good, is it a silver bullet? Of course the shortcomings are also very prominent:

• No way to guarantee an increasing trend, no way to sort
• The uuid is too long, and the storage takes up a lot of space, especially in the database, which is not friendly to indexing
• Without business attributes, this thing is just a string of numbers, meaningless, or regular

Of course, some people want to improve this guy, such as unreadable transformation, use uuid to int64 , and convert it to long type:

byte[] bytes = Guid.NewGuid().ToByteArray();
return BitConverter.ToInt64(bytes, 0);

For another example, transforming disorder, such as NHibernate algorithm of Comb , preserves the first 20 characters of uuid, and the following 12 characters are guid . The time is roughly in order, which is a small improvement.

Comment: UUID does not exist in the database as an index, as some logs, context recognition, it is still very fragrant, but if this thing is used as an order number, it is really crashing

Database auto-increment sequence

Stand-alone database

The primary key of the database itself has a natural feature of self-increment. As long as the ID is set as the primary key and self-incremented, we can insert a record into the database and return the self-incremented ID, such as the following table creation statement:

CREATE DATABASE `test`;
use test;
CREATE TABLE id_table (
    id bigint(20) unsigned NOT NULL auto_increment, 
    value char(10) NOT NULL default '',
    PRIMARY KEY (id),
) ENGINE=MyISAM;

Insert statement:

insert into id_table(value)  VALUES ('v1');

advantage:

• Stand-alone, simple and fast
• Natural self-increasing, atomicity
• Digital ID sorting, search, and paging are all more advantageous

The disadvantages are also obvious:

• Stand-alone, hang up the bucket and run away
• One machine, high concurrency is impossible

Clustered database

Since the high concurrency and high availability of a single machine cannot be solved, add machines and engage in a cluster mode database. Since the cluster mode, if there are multiple masters, then each machine must not generate its own ID, which will lead to duplicate IDs.

At this time, it is particularly important for starting value and step size For example, three machines V1, V2, V3:

统一步长：3
V1起始值：1
V2起始值：2
V3起始值：3

Generated ID:

V1：1, 4, 7, 10...
V2：2, 5, 8, 11...
V3：3, 6, 9, 12...

To set the command line, you can use:

set @@auto_increment_offset = 1;     // 起始值
set @@auto_increment_increment = 3;  // 步长

In this way, when there are enough masters, high performance is guaranteed. Even if some machines are down, slaves can be supplemented. Based on master-slave replication, it can greatly reduce the pressure on a single machine. But there are still disadvantages to this:

• The master-slave replication is delayed and the master is down. After the slave node is switched to the master node, the number may be repeated.
• After the initial value and step length are set, if you need to add machines later (horizontal expansion), it is very troublesome to adjust, and you may need to shut down and update in many cases.

Batch number segment database

The above access to the database is too frequent, and when the amount of concurrency rises, a lot of small probability problems may occur, so why don't we just take out a piece of id at once? Put it directly in the memory for use, and apply for another section when it is used up. You can also retain the advantages of the cluster mode, each time a range of IDs are retrieved from the database, such as 3 machines, and the number is issued:

每次取1000，每台步长3000
V1：1-1000,3001-4000,
V2：1001-2000,4001-5000
V3：2001-3000,5001-6000

Of course, if you don’t build multiple machines, you can apply for 10,000 numbers at a time, use optimistic lock to achieve, add a version number,

CREATE TABLE id_table (
  id int(10) NOT NULL,
  max_id bigint(20) NOT NULL COMMENT '当前最大id',
  step int(20) NOT NULL COMMENT '号段的步长',
  version int(20) NOT NULL COMMENT '版本号',
  `create_time` datetime DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP,
  `update_time` datetime DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`)
) 

Only when it is used up, will it reapply to the database. During the competition, the optimistic lock guarantees that only one request can succeed. An update operation:

update id_table set max_id = #{max_id+step}, version = version + 1 where version = # {version}

Focus:

• Batch acquisition, reducing database requests
• Optimistic lock to ensure data accuracy
• Obtaining can only be obtained from the database. Batch acquisition can be made into asynchronous timing tasks. If it is found to be less than a certain threshold, it will be automatically supplemented.

Redis self-increment

incr has an atomic command 0619275fb5b3fa, atomic self-increment, redis is fast, based on memory:

127.0.0.1:6379> set id 1
OK
127.0.0.1:6379> incr id      
(integer) 2

Of course, if redis has a problem with a single machine, you can also go to the cluster. You can also use the initial value + step size, you can use the INCRBY command, and several machines can basically resist high concurrency.

advantage:

• Based on memory, fast
• Natural sorting, self-increment, good for sorting search

shortcoming:

• After the step length is determined, it is more difficult to adjust the increase of the machine

• Need to pay attention to persistence, availability, etc., increase system complexity

  If redis persistence is RDB, take a snapshot for a period of time, then there may be data that fails to be persisted to the disk, and then hangs, and there may be duplicate IDs when restarting. At the same time, if the master-slave delays, the master node hangs , Master-slave switching, duplicate IDs may also appear. If you use AOF, if a command is persisted once, it may slow down the speed. If persisted once a second, you may lose up to one second of data. At the same time, data recovery will be slower. This is a process of trade-offs.

Zookeeper generates unique ID

Zookeeper can actually be used to generate a unique ID, but you don't use it because the performance is not high. Znode has a data version, which can generate a 32- or 64-bit serial number. This serial number is unique, but if the competition is relatively large, a distributed lock is needed, which is not worthwhile and is inefficient.

Meituan's Leaf

The following are from the official documents of : 1619275fb5b526 https://tech.meituan.com/2019/03/07/open-source-project-leaf.html

At the beginning of the design, Leaf adhered to several requirements:

1. Globally unique, there will never be duplicate IDs, and the overall trend of IDs is increasing.
2. Highly available, the service is completely based on a distributed architecture, even if MySQL is down, it can tolerate the unavailability of the database for a period of time.
3. High concurrency and low latency. On a CentOS 4C8G virtual machine, QPS can be called up to 5W+ remotely, and TP99 is within 1ms.
4. The access is simple, and it can be accessed directly through the company's RPC service or HTTP call.

The document is very clear, there are two versions:

• V1: ID is provided by pre-distribution, which is the number-segment distribution mentioned above. The table design is similar, meaning that ID is pulled in batches.

The disadvantage of this is that it takes a lot of time to update the number segment, and it is unavailable if it is down or master-slave replication at this time.

optimization:

• 1. First do a double buffer optimization, that is, asynchronous update, which means to create two number segments. For example, when one number segment is consumed by 10%, the next number segment will be allocated, which means that it is allocated in advance, and Asynchronous thread update
• 2. In the above scheme, the number segment may be fixed, and the span may be too large or too small, then make a dynamic change, determine the size of the next number segment according to the traffic, and adjust it dynamically

• V2: Leaf-snowflake, Leaf provides the implementation of the Java version. At the same time, the machine number generated by Zookeeper is weakly dependent. Even if there is a problem with Zookeeper, it will not affect the service. After Leaf gets the workerID from Zookeeper for the first time, it will cache a workerID file on the local file system. Even if there is a problem with ZooKeeper and the machine is restarting at the same time, the normal operation of the service can be guaranteed. In this way, the weak dependence on third-party components has improved the SLA to a certain extent.

snowflake (Snowflake Algorithm)

Snowflake is the ID generation algorithm used by Twitter's internal distributed projects. It is very popular after open source. The ID it generates is Long type, 8 bytes, a total of 64 bits, from left to right:

• 1 bit: Not used, the highest bit in the binary is 1 and both are negative numbers, but the unique IDs to be generated are all positive integers, so this 1 bit is fixed to 0.
• 41 digits: record timestamp (milliseconds), this digit can be $(2^{41}-1) / (1000 * 60 * 60 * 24 * 365) = 69$ years
• 10 digits: record the ID of the working machine, which can be machine ID or machine room ID + machine ID
• 12 digits: serial number, which is the id serial number generated at the same time on a certain machine in a certain computer room within this millisecond

Then each machine generates ID according to the above logic, it will be an increasing trend, because the time is increasing, and there is no need to make a distributed one, which is much simpler.

It can be seen that snowflakes are strongly dependent on time, because time theoretically keeps going forward, so the number of digits in this part is also increasing. But there is a problem, that is, the time is called back, that is, the time suddenly goes backwards, it may be a malfunction, or it may be a problem with the time acquisition after the restart. So how can we solve the time callback problem?

• The first solution is to judge when obtaining the time. If it is less than the last timestamp, then do not allocate it and continue to obtain the time in a loop until the time meets the conditions.
• The second solution: The above solution is only suitable for clock callbacks that are small. If the interval is too large and blocking waiting, it is definitely not advisable. Therefore, either the callback exceeding a certain size will directly report an error, denial of service, or there is a solution. Use the expansion position, add 1 to the expansion position after the callback, so that the ID can still remain unique.

Java code implementation:

public class SnowFlake {

    // 数据中心(机房) id
    private long datacenterId;
    // 机器ID
    private long workerId;
    // 同一时间的序列
    private long sequence;

    public SnowFlake(long workerId, long datacenterId) {
        this(workerId, datacenterId, 0);
    }

    public SnowFlake(long workerId, long datacenterId, long sequence) {
        // 合法判断
        if (workerId > maxWorkerId || workerId < 0) {
            throw new IllegalArgumentException(String.format("worker Id can't be greater than %d or less than 0", maxWorkerId));
        }
        if (datacenterId > maxDatacenterId || datacenterId < 0) {
            throw new IllegalArgumentException(String.format("datacenter Id can't be greater than %d or less than 0", maxDatacenterId));
        }
        System.out.printf("worker starting. timestamp left shift %d, datacenter id bits %d, worker id bits %d, sequence bits %d, workerid %d",
                timestampLeftShift, datacenterIdBits, workerIdBits, sequenceBits, workerId);

        this.workerId = workerId;
        this.datacenterId = datacenterId;
        this.sequence = sequence;
    }

    // 开始时间戳
    private long twepoch = 1420041600000L;

    // 机房号，的ID所占的位数 5个bit 最大:11111(2进制)--> 31(10进制)
    private long datacenterIdBits = 5L;

    // 机器ID所占的位数 5个bit 最大:11111(2进制)--> 31(10进制)
    private long workerIdBits = 5L;

    // 5 bit最多只能有31个数字，就是说机器id最多只能是32以内
    private long maxWorkerId = -1L ^ (-1L << workerIdBits);

    // 5 bit最多只能有31个数字，机房id最多只能是32以内
    private long maxDatacenterId = -1L ^ (-1L << datacenterIdBits);

    // 同一时间的序列所占的位数 12个bit 111111111111 = 4095  最多就是同一毫秒生成4096个
    private long sequenceBits = 12L;

    // workerId的偏移量
    private long workerIdShift = sequenceBits;

    // datacenterId的偏移量
    private long datacenterIdShift = sequenceBits + workerIdBits;

    // timestampLeft的偏移量
    private long timestampLeftShift = sequenceBits + workerIdBits + datacenterIdBits;

    // 序列号掩码 4095 (0b111111111111=0xfff=4095)
    // 用于序号的与运算，保证序号最大值在0-4095之间
    private long sequenceMask = -1L ^ (-1L << sequenceBits);

    // 最近一次时间戳
    private long lastTimestamp = -1L;


    // 获取机器ID
    public long getWorkerId() {
        return workerId;
    }


    // 获取机房ID
    public long getDatacenterId() {
        return datacenterId;
    }


    // 获取最新一次获取的时间戳
    public long getLastTimestamp() {
        return lastTimestamp;
    }


    // 获取下一个随机的ID
    public synchronized long nextId() {
        // 获取当前时间戳，单位毫秒
        long timestamp = timeGen();

        if (timestamp < lastTimestamp) {
            System.err.printf("clock is moving backwards.  Rejecting requests until %d.", lastTimestamp);
            throw new RuntimeException(String.format("Clock moved backwards.  Refusing to generate id for %d milliseconds",
                    lastTimestamp - timestamp));
        }

        // 去重
        if (lastTimestamp == timestamp) {

            sequence = (sequence + 1) & sequenceMask;

            // sequence序列大于4095
            if (sequence == 0) {
                // 调用到下一个时间戳的方法
                timestamp = tilNextMillis(lastTimestamp);
            }
        } else {
            // 如果是当前时间的第一次获取，那么就置为0
            sequence = 0;
        }

        // 记录上一次的时间戳
        lastTimestamp = timestamp;

        // 偏移计算
        return ((timestamp - twepoch) << timestampLeftShift) |
                (datacenterId << datacenterIdShift) |
                (workerId << workerIdShift) |
                sequence;
    }

    private long tilNextMillis(long lastTimestamp) {
        // 获取最新时间戳
        long timestamp = timeGen();
        // 如果发现最新的时间戳小于或者等于序列号已经超4095的那个时间戳
        while (timestamp <= lastTimestamp) {
            // 不符合则继续
            timestamp = timeGen();
        }
        return timestamp;
    }

    private long timeGen() {
        return System.currentTimeMillis();
    }

    public static void main(String[] args) {
        SnowFlake worker = new SnowFlake(1, 1);
        long timer = System.currentTimeMillis();
        for (int i = 0; i < 100; i++) {
            worker.nextId();
        }
        System.out.println(System.currentTimeMillis());
        System.out.println(System.currentTimeMillis() - timer);
    }

}
  

Baidu uid-generator

Snowflake medicine, developed by Baidu, based on the 0619275fb5b8a8 algorithm. The difference is that you can define the number of bits in each part yourself, and many optimizations and extensions have been made: https://github.com/baidu/uid-generator /blob/master/README.zh_cn.md

UidGenerator is implemented in Java, a unique ID generator Snowflake UidGenerator works in application projects in the form of components, supports custom workerId digits and initialization strategies, so it is suitable for such as automatic restart and drift of instances in virtualized environments such as docker 1619275fb5b8ee. In terms of implementation, UidGenerator uses the future time to solve the inherent concurrency limitations of the sequence; it uses RingBuffer to cache the generated UID, parallelizes the production and consumption of UID, and complements the CacheLine to avoid the hardware level caused by RingBuffer. "Pseudo-sharing" problem. The final single-machine QPS can reach 6 million.

Qin Huai's Viewpoint

Regardless of the uid generator, ensuring uniqueness is the core. Only on this core can other performance or high availability issues be considered. The overall solution is divided into two types:

• Centralization: a third-party center, such as Mysql, Redis, Zookeeper

  • Advantages: self-increasing trend
  • Disadvantages: increased complexity, usually clusters, and the step length is agreed in advance.

• Decentralization: directly generated on the local machine, snowflake, uuid

  • Advantages: simple, efficient, no performance bottleneck
  • Disadvantages: the data is relatively long, and the self-increment attribute is weak

No one is perfect. There are only solutions that meet the business and current size. There is no optimal solution in the technical solutions.

[Profile of the author] :
Qin Huai, Qinhuai Grocery Store ], the road to technology is not at a time, the mountains are high and the rivers are long, even if it is slow, it will never stop. Personal Writing direction: the Java source code parsing, JDBC , Mybatis , Spring , redis , distributed, wins the Offer, LeetCode etc., carefully write each article, do not like the title of the party, do not like bells and whistles, mostly to write a series of articles , I cannot guarantee that what I have written is completely correct, but I guarantee that what I have written has been practiced or searched for information. I hope to correct any omissions or errors.

refers to all offer solutions PDF

What did I write in 2020?

open source programming notes