Netty series: object codecs commonly used in netty

Introduction

In addition to using common strings for data transfer in the program, we use JAVA objects the most. In JDK, if an object needs to be transmitted over the network, it must implement the Serializable interface, indicating that the object can be serialized. In this way, the object object method of the JDK itself can be called to read and write the object.

So can the object serialization method of JDK be used directly for object transfer in netty? If not, what should be done?

Today, I will take a look at the object encoder provided in netty.

what is serialization

Serialization is to organize java objects in a certain order for transmission over the network or writing to storage. Deserialization is to read stored objects from the network or storage and convert them into real java objects.

So the purpose of serialization is to transfer objects. For some complex objects, we can use excellent third-party frameworks, such as Thrift, Protocol Buffer, etc., which are very convenient to use.

JDK itself also provides serialization functions. To make an object serializable, you can implement the java.io.Serializable interface.

java.io.Serializable is an interface that has existed since JDK1.1. It is actually a marker interface, because java.io.Serializable does not have an interface that needs to be implemented. Inheriting java.io.Serializable indicates that this class object can be serialized.

 @Data
@AllArgsConstructor
public class CustUser implements java.io.Serializable{
    private static final long serialVersionUID = -178469307574906636L;
    private String name;
    private String address;
}

Above we defined a CustUser serializable object. This object has two properties: name and address.

Next, let's see how to serialize and deserialize:

 public void testCusUser() throws IOException, ClassNotFoundException {
        CustUser custUserA=new CustUser("jack","www.flydean.com");
        CustUser custUserB=new CustUser("mark","www.flydean.com");

        try(FileOutputStream fileOutputStream = new FileOutputStream("target/custUser.ser")){
            ObjectOutputStream objectOutputStream = new ObjectOutputStream(fileOutputStream);
            objectOutputStream.writeObject(custUserA);
            objectOutputStream.writeObject(custUserB);
        }
        
        try(FileInputStream fileInputStream = new FileInputStream("target/custUser.ser")){
            ObjectInputStream objectInputStream = new ObjectInputStream(fileInputStream);
            CustUser custUser1 = (CustUser) objectInputStream.readObject();
            CustUser custUser2 = (CustUser) objectInputStream.readObject();
            log.info("{}",custUser1);
            log.info("{}",custUser2);
        }
    }

In the above example, we instantiate two CustUser objects, and use objectOutputStream to write the objects to a file, and finally use ObjectInputStream to read the objects from the file.

The above is the most basic use. Note that there is a serialVersionUID field in the CustUser class.

serialVersionUID is the unique marker of the serialized object. If the serialVersionUID defined in the class is consistent with the serialVersionUID in the serialized storage, it means that the two objects are one object, and we can deserialize the stored object.

If we do not explicitly define serialVersionUID, the JVM will automatically generate it according to the fields, methods and other information in the class. Many times when I look at the code, I find that many people set serialVersionUID to 1L, which is wrong because they don't understand what serialVersionUID really means.

Refactoring serialized objects

Suppose we have a serialized object in use, but suddenly we find that the object seems to be missing a field, and we need to add it, can we add it? Can the original serialized object be converted into this new object after adding it?

The answer is yes, provided that the serialVersionUID of the two versions must be the same. The newly added field is null after deserialization.

Serialization is not encryption

Many students may think this way when using serialization. Serialization has turned the object into a binary file. Does it mean that the object has been encrypted?

This is actually a misunderstanding of serialization. Serialization is not encryption, because even if you serialize, you can still know the structure of your class from the serialized data. For example, in the RMI remote call environment, even the private field in the class can be parsed from the stream.

What if we want to encrypt some fields during serialization?

At this time, you can consider adding the writeObject and readObject methods to the serialized object:

 private String name;
    private String address;
    private int age;

    private void writeObject(ObjectOutputStream stream)
            throws IOException
    {
        //给age加密
        age = age + 2;
        log.info("age is {}", age);
        stream.defaultWriteObject();
    }

    private void readObject(ObjectInputStream stream)
            throws IOException, ClassNotFoundException
    {
        stream.defaultReadObject();
        log.info("age is {}", age);
        //给age解密
        age = age - 2;
    }

In the above example, we added an age object for CustUser, and encrypted the age in writeObject (plus 2), and decrypted the age in readObject (minus 2).

Note that both writeObject and readObject are private void methods. Their invocation is achieved through reflection.

Use real encryption

In the above example, we only encrypt the age field. If we want to encrypt the entire object, is there any good way to do it?

JDK provides us with javax.crypto.SealedObject and java.security.SignedObject to encapsulate serialized objects. This encrypts the entire serialized object.

Or take an example:

 public void testCusUserSealed() throws IOException, ClassNotFoundException, NoSuchPaddingException, NoSuchAlgorithmException, IllegalBlockSizeException, BadPaddingException, InvalidAlgorithmParameterException, InvalidKeyException {
        CustUser custUserA=new CustUser("jack","www.flydean.com");
        Cipher enCipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
        Cipher deCipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
        SecretKey secretKey = new SecretKeySpec("saltkey111111111".getBytes(), "AES");
        IvParameterSpec iv = new IvParameterSpec("vectorKey1111111".getBytes());
        enCipher.init(Cipher.ENCRYPT_MODE, secretKey, iv);
        deCipher.init(Cipher.DECRYPT_MODE,secretKey,iv);
        SealedObject sealedObject= new SealedObject(custUserA, enCipher);

        try(FileOutputStream fileOutputStream = new FileOutputStream("target/custUser.ser")){
            ObjectOutputStream objectOutputStream = new ObjectOutputStream(fileOutputStream);
            objectOutputStream.writeObject(sealedObject);
        }

        try(FileInputStream fileInputStream = new FileInputStream("target/custUser.ser")){
            ObjectInputStream objectInputStream = new ObjectInputStream(fileInputStream);
            SealedObject custUser1 = (SealedObject) objectInputStream.readObject();
            CustUser custUserV2= (CustUser) custUser1.getObject(deCipher);
            log.info("{}",custUserV2);
        }
    }

In the above example, we constructed a SealedObject object and the corresponding encryption and decryption algorithm.

SealedObject is like a proxy, we write and read the encrypted object of this proxy. Thereby ensuring the security in the process of data transmission.

use a proxy

The above SealedObject is actually a proxy. Consider such a situation, if there are many fields in the class, and these fields can be automatically generated from one of the fields, then we don't actually need to serialize all the fields. , we only need to serialize that one field, and other fields can be derived from this field.

In this case, we need to use the proxy function of the serialized object.

First, the serialized object needs to implement the writeReplace method, which means to replace it with the object you really want to write:

 public class CustUserV3 implements java.io.Serializable{

    private String name;
    private String address;

    private Object writeReplace()
            throws java.io.ObjectStreamException
    {
        log.info("writeReplace {}",this);
        return new CustUserV3Proxy(this);
    }
}

Then in the Proxy object, you need to implement the readResolve method to reconstruct the serialized object from the serialized data. As follows:

 public class CustUserV3Proxy implements java.io.Serializable{

    private String data;

    public CustUserV3Proxy(CustUserV3 custUserV3){
        data =custUserV3.getName()+ "," + custUserV3.getAddress();
    }

    private Object readResolve()
            throws java.io.ObjectStreamException
    {
        String[] pieces = data.split(",");
        CustUserV3 result = new CustUserV3(pieces[0], pieces[1]);
        log.info("readResolve {}",result);
        return result;
    }
}

Let's see how to use it:

 public void testCusUserV3() throws IOException, ClassNotFoundException {
        CustUserV3 custUserA=new CustUserV3("jack","www.flydean.com");

        try(FileOutputStream fileOutputStream = new FileOutputStream("target/custUser.ser")){
            ObjectOutputStream objectOutputStream = new ObjectOutputStream(fileOutputStream);
            objectOutputStream.writeObject(custUserA);
        }

        try(FileInputStream fileInputStream = new FileInputStream("target/custUser.ser")){
            ObjectInputStream objectInputStream = new ObjectInputStream(fileInputStream);
            CustUserV3 custUser1 = (CustUserV3) objectInputStream.readObject();
            log.info("{}",custUser1);
        }
    }

Note that we are both writing and reading the CustUserV3 object.

Difference between Serializable and Externalizable

Finally, let's talk about the difference between Externalizable and Serializable. Externalizable inherits from Serializable, which needs to implement two methods:

 void writeExternal(ObjectOutput out) throws IOException;
 void readExternal(ObjectInput in) throws IOException, ClassNotFoundException;

When do you need to use writeExternal and readExternal?

With Serializable, Java automatically does object serialization for the objects and fields of the class, which may take up more space. Externalizable requires us to completely control how to write/read, which is more troublesome, but if you consider performance, you can use Externalizable.

In addition, Serializable does not need to execute the constructor for deserialization. The Externalizable needs to execute the constructor to construct the object, and then call the readExternal method to populate the object. So Externalizable objects require a no-argument constructor.

Object transfer in netty

In the serialization section above, we already know that for a defined JAVA object, we can use ObjectOutputStream and ObjectInputStream to achieve object read and write work, then can we use the same method to perform object reading and writing in netty? What about reading and writing?

Unfortunately, the object read and write methods in the JDK cannot be directly used in netty, and we need to transform them.

This is because we need a general object encoder and decoder. If we use ObjectOutputStream and ObjectInputStream, because the structure of different objects is different, we need to know the object type of the read data when reading the object to be perfect convert.

What we need in netty is a more general codec, so what should we do?

Remember the LengthFieldBasedFrameDecoder we talked about in the general frame decoder? By adding the length of the data in front of the real data, the purpose of distinguishing frames according to the length of the data is achieved.

The codec names provided in netty are called ObjectEncoder and ObjectDecoder. Let's first look at their definitions:

 public class ObjectEncoder extends MessageToByteEncoder<Serializable> {

 public class ObjectDecoder extends LengthFieldBasedFrameDecoder {

You can see that ObjectEncoder inherits from MessageToByteEncoder, and the generic type is Serializable, which means that the encoder is encoded from a serializable object to ByteBuf.

And ObjectDecoder inherits from LengthFieldBasedFrameDecoder as we said above, so a length field can be used to distinguish the actual length of the object to be read.

Let's take a closer look at how these two classes work.

ObjectEncoder

Let's first look at how ObjectEncoder serializes an object into a ByteBuf.

According to the definition of LengthFieldBasedFrameDecoder, we need an array to save the length of the real data, here is a 4-byte byte array called LENGTH_PLACEHOLDER, as shown below:

 private static final byte[] LENGTH_PLACEHOLDER = new byte[4];

Let's look at the implementation of its encode method:

 protected void encode(ChannelHandlerContext ctx, Serializable msg, ByteBuf out) throws Exception {
        int startIdx = out.writerIndex();

        ByteBufOutputStream bout = new ByteBufOutputStream(out);
        ObjectOutputStream oout = null;
        try {
            bout.write(LENGTH_PLACEHOLDER);
            oout = new CompactObjectOutputStream(bout);
            oout.writeObject(msg);
            oout.flush();
        } finally {
            if (oout != null) {
                oout.close();
            } else {
                bout.close();
            }
        }
        int endIdx = out.writerIndex();
        out.setInt(startIdx, endIdx - startIdx - 4);
    }

Here, a ByteBufOutputStream is first created, then a 4-byte length field is written to this Stream, and then the ByteBufOutputStream is encapsulated into a CompactObjectOutputStream.

CompactObjectOutputStream is a subclass of ObjectOutputStream, which overrides the writeStreamHeader and writeClassDescriptor methods.

CompactObjectOutputStream writes the final data msg into the stream, and an encoding process is almost complete.

Why do you say it's almost done? Because the length field is still empty.

At the beginning, we just wrote a placeholder with a length, this placeholder is empty, and there is no data, this data is written in the last step out.setInt:

 out.setInt(startIdx, endIdx - startIdx - 4);

This implementation also gives us an idea. When we don't know the real length of the message, if we want to write the length of the message before the message, we can occupy a place first, and when all the messages are read, we can know the real length. After that, replace the data.

At this point, the object data has been fully encoded, let's take a look at how to read the object from the encoded data.

ObjectDecoder

I said before that ObjectDecoder inherits from LengthFieldBasedFrameDecoder, and its decode method is like this:

 protected Object decode(ChannelHandlerContext ctx, ByteBuf in) throws Exception {
        ByteBuf frame = (ByteBuf) super.decode(ctx, in);
        if (frame == null) {
            return null;
        }

        ObjectInputStream ois = new CompactObjectInputStream(new ByteBufInputStream(frame, true), classResolver);
        try {
            return ois.readObject();
        } finally {
            ois.close();
        }
    }

First call the decode method of LengthFieldBasedFrameDecoder, according to the length of the object, read the real object data and put it into ByteBuf.

Then read the real object from ByteBuf through a custom CompactObjectInputStream and return it.

CompactObjectInputStream inherits from ObjectInputStream, which is the opposite of CompactObjectOutputStream.

ObjectEncoderOutputStream and ObjectDecoderInputStream

ObjectEncoder and ObjectDecoder are conversions between objects and ByteBuf. Netty also provides ObjectEncoderOutputStream and ObjectDecoderInputStream which are compatible with ObjectEncoder and ObjectDecoder. These two classes can encode and decode objects from streams, and are fully compatible with ObjectEncoder and ObjectDecoder.

Summarize

The above are the object encoders and decoders provided in netty. If you want to pass objects in netty, then these two encoders and decoders provided by netty are the best choices.

This article has been included in http://www.flydean.com/14-8-netty-codec-object/

The most popular interpretation, the most profound dry goods, the most concise tutorials, and many tricks you don't know are waiting for you to discover!

Welcome to pay attention to my official account: "Program those things", understand technology, understand you better!