Netty series: the core decoder json in netty

Introduction

There are many ways of data transmission between programs and programs, which can be transmitted through binary protocols. The more popular ones are thrift protocol or google's protobuf. These binary protocols can realize the efficient transmission of data, and the volume of data can be saved in the form of binary, which is very effective in some cases where speed and efficiency are the priority. And if different programming languages call each other, it can also be implemented through this binary protocol.

Although binary is faster and more efficient, it is not very friendly to programmers, because it is difficult for a person to read binary files directly. Although there are some text data transmission methods, such as XML, the cumbersome tags of XML lead to XML has many inconveniences in use. So a common text file transfer format json was born.

Friends who can read this article must be familiar with json. Of course, there are some more concise file formats, such as YAML. Interested friends can learn more about it.

What we want to talk about here is the decoding of json by netty.

json support in java

In java, our use of json is usually to convert an object into json for data transmission, or to parse the received json and convert it into an object.

It is a pity that a good JSON tool is not provided in JDK, so we generally need to use third-party JSON packages to realize the conversion between Object and JSON.

Commonly used are Google's GSON, Ali's FastJSON and jackson, etc.

Here we use google's GSON to introduce.

Here we mainly explain the mutual conversion of objects and json in java, so other more powerful functions in GSON will not be introduced here.

First we create a JAVA object, we define a Student class as follows:

 static class Student {
        String name;
        String phone;
        Integer age;

        public Student(String name, String phone, Integer age) {
            this.name = name;
            this.phone = phone;
            this.age = age;
        }
    }

In this class, we define several different properties and a constructor for Student. Next, let's see how to use GSON to convert this object to JSON:

 Student obj = new Student("tina","188888888",18);
        Gson gson = new Gson();
        String json = gson.toJson(obj);
        System.out.println(json);
        Student obj2 = gson.fromJson(json, Student.class);
        System.out.println(obj2);

GSON is very simple to use. After we build the Gson object, we can directly call its toJson method to convert the object into a json string.

Then call the fromJson method of json to convert the json string into an object.

The output of the above code is as follows:

 {"name":"tina","phone":"188888888","age":18}
com.flydean.JsonTest$Student@4534b60d

Decoding of json by netty

netty provides a decoder for json called JsonObjectDecoder, first look at the definition of JsonObjectDecoder:

 public class JsonObjectDecoder extends ByteToMessageDecoder

Unlike the base64, byte array explained earlier, JsonObjectDecoder inherits ByteToMessageDecoder instead of MessageToMessageDecoder.

This shows that JsonObjectDecoder is directly converted from ByteBuf to Json Object object.

We know that there is no JSON object in JDK, all objects are imported from third-party packages, netty does not introduce new objects, so the object parsed from Json in netty is still a ByteBuf object, in this ByteBuf Contains a Json object.

What is the parsing logic of JsonObjectDecoder?

First look at the 4 states defined in JsonObjectDecoder:

 private static final int ST_CORRUPTED = -1;
    private static final int ST_INIT = 0;
    private static final int ST_DECODING_NORMAL = 1;
    private static final int ST_DECODING_ARRAY_STREAM = 2;

ST_INIT represents the initial state of the decode, and ST_CORRUPTED represents the abnormal state that occurs in the decode.

ST_DECODING_NORMAL represents a normal json as follows:

 {
    "source": "web",
    "type": "product_info",
    "time": 1641967014440,
    "data": {
        "id": 30000084318055,
        "staging": false
    },
    "dataId": "123456"
}

ST_DECODING_ARRAY_STREAM represents an array. For an array, an array is also an object, so an array can also be represented by json. The following is a common json array:

 [ "Google", "Runoob", "Taobao" ]

The decoding logic of JsonObjectDecoder is relatively simple. It mainly reads the data in ByteBuf, and divides and parses the json object by judging the read data and the special braces, brackets, commas and other delimiters in json.

It should be noted that the message in the ByteBuf to be decoded by JsonObjectDecoder should be in UTF-8 encoding format. Why do you need UTF-8 format?

This is because the unique delimiters in json are stored in a byte even in UTF-8, so that we can compare the read byte value with the delimiter of json during the process of reading data. So as to determine the boundaries of different objects in json.

If it is replaced by other encoding methods, the delimiter in json may be represented by multiple bytes, which makes our parsing more difficult, because we need to know when the delimiter starts and when it is the delimiter. Finish.

Its core decoding logic is as follows, first read a byte from ByteBuf:

 byte c = in.getByte(idx);

Then call decodeByte(c, in, idx); to determine whether the current position is an open bracket or a closed bracket, in an object string, or a new object string.

First, you need to make a judgment on the current state. The state judgment calls the initDecoding method:

 private void initDecoding(byte openingBrace) {
        openBraces = 1;
        if (openingBrace == '[' && streamArrayElements) {
            state = ST_DECODING_ARRAY_STREAM;
        } else {
            state = ST_DECODING_NORMAL;
        }
    }

The next step is to compare the current state with the four custom states. If it is an ordinary json object and the object is already in the closed bracket state, it means that the object has been read and can be converted and output:

 if (state == ST_DECODING_NORMAL) {
                decodeByte(c, in, idx);
                if (openBraces == 0) {
                    ByteBuf json = extractObject(ctx, in, in.readerIndex(), idx + 1 - in.readerIndex());
                    if (json != null) {
                        out.add(json);
                    }
    ...

If state indicates that it is currently an array object, the array object may contain multiple objects, and these objects are distinguished by commas. There may also be spaces between commas, so special judgment and processing of these data are required, as follows:

 else if (state == ST_DECODING_ARRAY_STREAM) {
                decodeByte(c, in, idx);

                if (!insideString && (openBraces == 1 && c == ',' || openBraces == 0 && c == ']')) {
                    for (int i = in.readerIndex(); Character.isWhitespace(in.getByte(i)); i++) {
                        in.skipBytes(1);
                    }
                    int idxNoSpaces = idx - 1;
                    while (idxNoSpaces >= in.readerIndex() && Character.isWhitespace(in.getByte(idxNoSpaces))) {
                        idxNoSpaces--;
                    }
                    ByteBuf json = extractObject(ctx, in, in.readerIndex(), idxNoSpaces + 1 - in.readerIndex());
                    if (json != null) {
                        out.add(json);
                    }
    ....

Finally, put the parsed json object into the out list of byteBuf, and the whole parsing ends here.

Summarize

The above is the use of the json core decoder JsonObjectDecoder in netty. Its essence is to split multiple json strings by judging the separator in the json object, and then store the split json strings in ByteBuf for output.

Seeing this, you may be wondering, doesn't the decoder appear together with the encoder? Why is there only JsonObjectDecoder in netty and no JsonObjectEncoder?

In fact, the Json object here is a string containing Json characters, which is written into ByteBuf, so no special encoder is needed here.

This article has been included in http://www.flydean.com/14-3-netty-codec-json/

The most popular interpretation, the most profound dry goods, the most concise tutorials, and many tricks you don't know are waiting for you to discover!

Welcome to pay attention to my official account: "Program those things", understand technology, understand you better!