Collection of IO Series Articles for Little Junior Sisters-Attached PDF Download

The most exciting part of java is IO and NIO. The full name of IO is input output, which is a bridge between the java program and the outside world. IO refers to all the classes in the java.io package. They have existed since java1.0. NIO is called new IO, which is a new generation of IO introduced in java1.4.

What is the nature of IO? What is the difference between it and NIO? How should we learn IO and NIO?

This series will describe in detail the process of learning java IO with the help of the perspective of the little sister. I hope you will enjoy it.

Who is the little sister? The name is unknown, but he is diligent and loves to learn and has unlimited potential. Let's take a look.

The example of this article https://github.com/ddean2009/learn-java-io-nio

The article is too long, you can directly download the PDF of this article: download link java-io-all-in-one.pdf

Chapter One The Essence of IO

The essence of IO

The function of IO is to read data from the external system to the java program, or to write the data output from the java program back to the external system. The external system here may be a disk, a network stream, and so on.

Because the processing of all external data is implemented by the operating system kernel, for a java application, it just calls the corresponding interface method in the operating system to interact with the external data.

The essence of all IO is the processing of the Buffer. We put data into the Buffer for the system to write external data, or read the data read from the external system from the system Buffer. As shown below:

User space means that our own java program has a buffer, and system space also has a buffer. Therefore, the system space will cache data. In this case, the system space will directly return the data in the buffer to improve the reading speed.

DMA and virtual address space

Before continuing, let's explain the basic concepts in the two operating systems to facilitate our understanding of IO later.

Modern operating systems have a component called DMA (Direct Memory Access). What does this component do?

Generally speaking, the reading and writing of the memory is done by the CPU. Without DMA, if the program performs IO operations, then all the CPU time will be occupied, and the CPU cannot respond to other tasks. Can wait for IO execution to complete. This is unimaginable in modern applications.

If DMA is used, the CPU can transfer the IO operation to other operating system components, such as the data manager, and only after the data manager has completed the operation, will the CPU be notified of the completion of the IO operation. Modern operating systems basically implement DMA.

Virtual address space is also called (Virtual address space). In order to isolate different programs from each other and ensure the certainty of addresses in programs, modern computer systems introduce the concept of virtual address space. To put it simply, it can be seen as a mapping with an actual physical address. By using segmentation or paging technology, the actual physical address is mapped to the virtual address space.

For the basic flow chart of IO above, we can map the buffer in the system space and the buffer in the user space to the same place in the virtual address space at the same time. In this way, the step of copying from system space to user space is omitted. The speed will be faster.

At the same time, in order to solve the problem that virtual space is larger than physical memory space, modern computer technology generally uses paging technology.

The paging technology is to divide the virtual space into many pages, and only allocate the page to the mapping of physical memory when it is needed, so that the physical memory can actually be regarded as a cache of the virtual space address.

The impact of virtual space address paging on IO is that the operation of IO is also based on page.

Commonly used page sizes are: 1,024, 2,048, and 4,096 bytes.

IO classification

IO can be divided into two types: File/Block IO and Stream I/O.

For File/Block IO, the data is stored in the disk, and the disk is managed by the filesystem. We can define the file name, path, file attributes, etc. through the filesystem.

The filesystem is managed by dividing the data into individual data blocks. Some blocks store file metadata, and some blocks store real data.

Finally, the filesystem also performs paging in the process of processing data. The page size of the filesystem can be the same as the size of the memory page, or a multiple of it, such as 2,048 or 8,192 bytes.

Not all data exists in the form of blocks. We also have a type of IO called stream IO.

Stream IO is like a pipeline stream, the data in it is the sequence to be consumed.

The difference between IO and NIO

IO in java1.0 is streaming IO, it can only process data one byte by one byte, so IO is also called Stream IO.

NIO was born to improve the efficiency of IO, and it reads data in the form of blocks.

In Stream IO, input inputs one byte, and output outputs one byte. Because it is a Stream, a filter or filter chain can be added. Think about the filter chain in the web framework. In Stream IO, data can only be processed once, and you cannot roll back data in Stream.

In Block IO, data is processed in the form of blocks, so its processing speed is faster than Stream IO, and at the same time it can process data back. But you need to deal with the buffer yourself, so the complexity is higher than that of Stream IO.

Generally speaking, Stream IO is a blocking IO. When a thread performs a read or write operation, the thread will be blocked.

NIO is generally non-blocking, which means that other operations can be performed during the read or write process, and the completion of the NIO operation will be notified after the read or write operation is completed.

In IO, it is mainly divided into DataOutPut and DataInput, which correspond to the out and in of IO respectively.

There are three major categories of DataOutPut, namely Writer, OutputStream and ObjectOutput.

Look at the inheritance relationship among them:

DataInput also has three major categories, namely ObjectInput, InputStream and Reader.

Look at their inheritance relationship:

ObjectOutput and ObjectInput classes are relatively small, so they are not listed here.

Count about 20 classes and figure out the usefulness of these 20 classes. Congratulations, you can understand the java IO!

For NIO, it is a bit more complicated. First of all, in order to process the block information, the data needs to be read into the buffer. Therefore, in NIO, Buffer is a very important concept. Let's look at the Buffer in NIO:

From the figure above, we can see that NIO has prepared a variety of buffer types for us to use.

Another very important concept is channel, which is the channel through which NIO obtains data:

The number of classes that NIO needs to master is slightly more than that of IO, after all, NIO is a bit more complicated.

With just dozens of categories, we have mastered IO and NIO, and I feel excited to think about it.

Summarize

In the next article, we will introduce the little junior sister to you. It just so happens that she is also learning java IO, so let's go with her in the future, so stay tuned.

Chapter 2 try with and its underlying principles

Introduction

Little Junior Sister is a java beginner and is learning to use java IO recently. As a big brother, I naturally want to give her the most powerful support. Let's take a look at what problems she encountered and how the problems were solved.

IO closed problem

On this day, the little sister asked me with a depressed look: Brother F, I have been learning Java IO for many days. I recently wrote an example. There is no problem reading one file, but reading many files will tell me: "Can 't open so many files", can you help me see what the problem is?

For more information, please visit www.flydean.com

Of course, I can’t refuse the request of Junior Sister. I immediately responded: Maybe there are too many open files. I will teach you two commands to check the maximum file opening limit.

One command is ulimit -a

The second command is

ulimit -n
256

It seems that your maximum file limit is too small, there are only 256, just adjust it to a larger size.

Little Junior Sister said: No, Brother F, I read the files one by one, and I didn't open so many files at the same time.

Well, take a look at the code you wrote:

BufferedReader bufferedReader = null;
        try {
            String line;
            bufferedReader = new BufferedReader(new FileReader("trywith/src/main/resources/www.flydean.com"));
            while ((line = bufferedReader.readLine()) != null) {
                log.info(line);
            }
        } catch (IOException e) {
            log.error(e.getMessage(), e);
        }

After reading the code, the problem has been found. Junior sister, your IO is not closed. You should close your reader in finally after using it.

The following piece of code will do:

BufferedReader bufferedReader = null;
        try {
            String line;
            bufferedReader = new BufferedReader(new FileReader("trywith/src/main/resources/www.flydean.com"));
            while ((line = bufferedReader.readLine()) != null) {
                log.info(line);
            }
        } catch (IOException e) {
            log.error(e.getMessage(), e);
        } finally {
            try {
                if (bufferedReader != null){
                 bufferedReader.close();
                }
            } catch (IOException ex) {
                log.error(ex.getMessage(), ex);
            }
        }

Little Junior Sister thanked her and went to change the code silently.

Use try with resource

After half an hour, the younger sister came to me again, brother F, now every piece of code must be manually added finally, it is too troublesome, many times I am afraid that I forget to close the IO, causing unexpected exceptions in the program . You also know that I am always afraid of trouble. Is there any simple way to solve this problem?

So what JDK version do you use, Junior Sister?

The little sister said with embarrassment: Although the latest JDK has reached 14, I still use JDK8.

JDK8 is enough. In fact, starting from JDK7, Java has introduced the new function of try with resource. You put the resource that you want to close after use into the try, and the JVM will automatically close it for you. Is it very convenient? Look at the following Snippet of code:

try (BufferedReader br = new BufferedReader(new FileReader("trywith/src/main/resources/www.flydean.com")))
        {
            String sCurrentLine;
            while ((sCurrentLine = br.readLine()) != null)
            {
                log.info(sCurrentLine);
            }
        } catch (IOException e) {
            log.error(e.getMessage(), e);
        }

The principle of try with resource

That’s great, the younger sister was very happy, and then she started asking me again: Brother F, what is a resource? Why don't you need to close by yourself when you put it in the try?

Resource is a resource, which can be opened and closed. We can call all classes that implement the java.lang.AutoCloseable interface resource.

First look at the definition of AutoCloseable:

public interface AutoCloseable {
        void close() throws Exception;
}

AutoCloseable defines a close() method. When we open the AutoCloseable resource in the try with resource, when the try block execution ends, the JVM will automatically call this close() method to close the resource.

Let's see how the close method in the above BufferedReader is implemented:

public void close() throws IOException {
    synchronized (lock) {
        if (in == null)
            return;
        in.close();
        in = null;
        cb = null;
    }
}

Custom resource

The little sister suddenly realized: Brother F, can we implement AutoCloseable to create our own resources?

Of course, we can give you an example. For example, after answering this question for you, I will go to dinner. We define such a resource class:

public class CustResource implements AutoCloseable {

    public void helpSister(){
        log.info("帮助小师妹解决问题！");
    }

    @Override
    public void close() throws Exception {
        log.info("解决完问题，赶紧去吃饭！");
    }

    public static void main(String[] args) throws Exception {
       try( CustResource custResource= new CustResource()){
           custResource.helpSister();
       }
    }
}

Run output:

[main] INFO com.flydean.CustResource - 帮助小师妹解决问题！
[main] INFO com.flydean.CustResource - 解决完问题，赶紧去吃饭！

Summarize

In the end, the problem of Junior Sister was solved, and I can also eat on time.

Chapter 3 File System

Introduction

Little Junior Sister has encountered a problem again. This time the problem is related to file creation, file permissions and file system related issues. Fortunately, the answers to these questions are in my mind, let's take a look.

File permissions and file system

As soon as I arrived at the company in the morning, the younger sister came over and asked me mysteriously: Brother F, I have put some important files on the server, which are very, very important. Is there any way to protect it? Have a little privacy?

For more information, please visit www.flydean.com

What file is so important? It won't be your picture, don't worry, no one will be interested.

Little Junior Sister said: Of course not, I want to put my learning experience up, but Brother F, you know, I have just started learning, and many of my ideas are not mature enough. I want to keep a secret first and make it public later.

Seeing how self-motivated younger junior sister is, I always burst into tears, and I feel very comforted. Let's get started.

You know, there are two types of operating systems in this world, windows and linux (unix) systems. The two systems are quite different, but both systems have a concept of a file. Of course, the scope of files in Linux is more extensive, and almost all resources can be regarded as files.

If there are files, there are corresponding file systems. These file systems are supported by the system kernel. We don't need to repeat the wheel in the java program, just call the system kernel interface directly.

Junior sister: Brother F, I understand this. We don't make wheels again, we are just porters of wheels. So how does java call the system kernel to create files?

The most common way to create a file is to call the createNewFile method in the File class. Let's look at the implementation of this method:

public boolean createNewFile() throws IOException {
        SecurityManager security = System.getSecurityManager();
        if (security != null) security.checkWrite(path);
        if (isInvalid()) {
            throw new IOException("Invalid file path");
        }
        return fs.createFileExclusively(path);
    }

Inside the method, the security check is performed first, and if the security check is passed, the createFileExclusively method of FileSystem is called to create the file.

In my mac environment, the implementation class of FileSystem is UnixFileSystem:

public native boolean createFileExclusively(String path)
        throws IOException;

see it? The createFileExclusively in UnixFileSystem is a native method, which will call the underlying system interface.

Junior sister: Wow, after the file is created, we can assign permissions to the file, but are the permissions on windows and linux the same?

This question is a good question. Java code is cross-platform. Our code needs to be executed on the JVM on windows and linux at the same time, so we must find the common ground of their permissions.

Let's take a look at the permissions of the windows file first:

It can be seen that the permissions of a windows file can be modified, read and executed. We don't need to consider special permissions first, because we need to find the common ground between windows and linux.

Look at the permissions of the linux file:

 ls -al www.flydean.com 
-rw-r--r--  1 flydean  staff  15 May 14 15:43 www.flydean.com

Above, I used an ll command to list the detailed information of the www.flydean.com file. The first column is the file permissions.

The basic file permissions of Linux can be divided into three parts, namely owner, group, and others. Each part has read, write and execute permissions like windows, which are represented by rwx.

The three parts of permissions are linked together to become rwxrwxrwx. Compared with our output above, we can see that the file www.flydean.com is readable and writable for the owner, read-only for the group user, and also for other users Read-only.

If you want to make the file only readable by yourself, you can execute the following command:

chmod 600 www.flydean.com

The younger sister became excited immediately: Brother F, I understand this, 6 is 110 in binary, and 600 is 110000000 in binary, which corresponds to rw-------.

I am very satisfied with the comprehension ability of Junior Junior Sister.

File creation

Although we are no longer in the era of Kong Yiji, we don't need to know the four ways of writing fenzi, but it is necessary to have one more knowledge and one more way, and to make sufficient preparations.

Junior sister, then do you know what kinds of file creation methods are there in java?

Little Junior Sister whispered: Brother F, I only know a new File method.

I stroked my beard with satisfaction, showing my superior aura.

As we said before, there are three types of IO, one is Reader/Writer, the other is InputStream/OutputStream, and the last is ObjectReader/ObjectWriter.

In addition to using the first new File, we can also use OutputStream to achieve, of course, we also need to use the try with resource feature mentioned earlier to make the code more concise.

Look at the first way first:

public void createFileWithFile() throws IOException {
        File file = new File("file/src/main/resources/www.flydean.com");
        //Create the file
        if (file.createNewFile()){
            log.info("恭喜，文件创建成功");
        }else{
            log.info("不好意思，文件创建失败");
        }
        //Write Content
        try(FileWriter writer = new FileWriter(file)){
            writer.write("www.flydean.com");
        }
    }

Look at the second way:

public  void createFileWithStream() throws IOException
    {
        String data = "www.flydean.com";
        try(FileOutputStream out = new FileOutputStream("file/src/main/resources/www.flydean.com")){
            out.write(data.getBytes());
        }
    }

The second way looks more brief than the first way.

Junior sister: Wait a minute, Brother F, NIO has already appeared in JDK7, can I use NIO to create files?

This question is certainly not difficult for me:

public void createFileWithNIO()  throws IOException
    {
        String data = "www.flydean.com";
        Files.write(Paths.get("file/src/main/resources/www.flydean.com"), data.getBytes());

        List<String> lines = Arrays.asList("程序那些事", "www.flydean.com");
        Files.write(Paths.get("file/src/main/resources/www.flydean.com"),
                lines,
                StandardCharsets.UTF_8,
                StandardOpenOption.CREATE,
                StandardOpenOption.APPEND);
    }

The Files tool class is provided in NIO to realize the writing operation of files. When writing, we can also bring some parameters, such as character encoding, whether to replace the file or append to the back of the file, and so on.

Permissions of files in the code

Junior sister has a problem again: Brother F, after talking for a long time, I haven't told me about the authority.

Don't worry, let's talk about permissions now:

public void fileWithPromission() throws IOException {
        File file = File.createTempFile("file/src/main/resources/www.flydean.com","");
        log.info("{}",file.exists());

        file.setExecutable(true);
        file.setReadable(true,true);
        file.setWritable(true);
        log.info("{}",file.canExecute());
        log.info("{}",file.canRead());
        log.info("{}",file.canWrite());

        Path path = Files.createTempFile("file/src/main/resources/www.flydean.com", "");
        log.info("{}",Files.exists(path));
        log.info("{}",Files.isReadable(path));
        log.info("{}",Files.isWritable(path));
        log.info("{}",Files.isExecutable(path));
    }

As we mentioned above, for the purpose of general use, JVM can only take the functions that both windows and linux have, that is to say, the permissions are only read and write and execute permissions, because windows can also distinguish this user or other users, so whether it is the permission of this user Also retained.

In the above example, we used the traditional File and Files in NIO to update file permissions.

Summarize

Okay, let's talk about file permissions first.

Chapter 4 File Reading Those Things

Introduction

Little Junior Sister is a little bit confused about the reader and stream in java IO recently. I don't know which one to use. How to read the file is the correct posture? Today, Brother F answered her on the spot.

Characters and bytes

Junior sister has been confused recently: Brother F, last time you mentioned that IO reading is divided into two categories, namely Reader and InputStream. Is there any difference between these two categories? Why do I see some classes that are both Reader and Stream? For example: InputStreamReader?

Junior sister, do you know the ultimate three questions of philosophers? who are you? Where did it come from? Where to go?

Brother F, are you confused? I'm asking you java, what philosophy are you talking about.

Little Junior Sister, in fact, philosophy is the foundation of all learning. Do you know how to translate scientific principles in English? the philosophy of science, the principle of science is philosophy.

What do you think of the nature of the code in the computer? The essence of the code is a long series of binary numbers composed of 0 and 1. The combination of so many binary numbers becomes the code in the computer, that is, the binary code that can be recognized by the JVM.

For more information, please visit www.flydean.com

Little Junior Sister looked admired: Brother F seemed to make a lot of sense, but what does this have to do with Reader and InputStream?

Don't worry, there is a certain number in the secret, let me ask you a question first, what is the smallest unit of storage in java?

Junior sister: Let me think about it, the smallest one in java should be boolean, true and false correspond exactly to binary 1, 0.

Half right, although boolean is also the smallest unit of storage in java, it needs to take up a byte of space. The smallest storage unit in java is actually Byte. If you don’t believe it, you can use the JOL tool I introduced before to verify it:

[main] INFO com.flydean.JolUsage - java.lang.Boolean object internals:
 OFFSET  SIZE      TYPE DESCRIPTION                               VALUE
      0    12           (object header)                           N/A
     12     1   boolean Boolean.value                             N/A
     13     3           (loss due to the next object alignment)
Instance size: 16 bytes
Space losses: 0 bytes internal + 3 bytes external = 3 bytes total

The above is the boxed Boolean. You can see that although Boolean occupies 16 bytes at the end, the boolean inside is only 1 byte.

Byte is translated into Chinese as byte, and byte is the basic unit of storage in java.

With bytes, we can interpret characters. Characters are composed of bytes. According to different encoding methods, characters can consist of 1, 2 or more bytes. We humans can recognize Chinese characters with the naked eye, and English can be regarded as characters.

The Reader is the characters read according to a certain encoding format, and the InputStream is the lower-level bytes that are directly read.

Junior sister: I understand. If it is a text file, we can use Reader, and if it is a non-text file, we can use InputStream.

Ruzi can teach, and the younger sister is making rapid progress.

Read by character

Junior sister, next Brother F will tell you several ways to read files by character. The first is to use FileReader to read File, but FileReader itself does not provide any method to read data, and you want to read it. To fetch data, we still need to use BufferedReader to connect to FileReader. BufferedReader provides a buffer for reading, which can read one line at a time:

public void withFileReader() throws IOException {
        File file = new File("src/main/resources/www.flydean.com");

        try (FileReader fr = new FileReader(file); BufferedReader br = new BufferedReader(fr)) {
            String line;
            while ((line = br.readLine()) != null) {
                if (line.contains("www.flydean.com")) {
                    log.info(line);
                }
            }
        }
    }

Every time you read a line, you can connect these lines to form a stream. Through Files.lines, we get a stream. In the stream, we can use lambda expressions to read files. This is called the second type. Way:

public void withStream() throws IOException {
        Path filePath = Paths.get("src/main/resources", "www.flydean.com");
        try (Stream<String> lines = Files.lines(filePath))
        {
            List<String> filteredLines = lines.filter(s -> s.contains("www.flydean.com"))
                    .collect(Collectors.toList());
            filteredLines.forEach(log::info);
        }
    }

The third type is actually not commonly used, but I also want to teach you. This way is to use Scanner in the tool category. The Scanner can be used to split the file by line breaks, which is also good to use:

public void withScanner() throws FileNotFoundException {
        FileInputStream fin = new FileInputStream(new File("src/main/resources/www.flydean.com"));
        Scanner scanner = new Scanner(fin,"UTF-8").useDelimiter("\n");
        String theString = scanner.hasNext() ? scanner.next() : "";
        log.info(theString);
        scanner.close();
    }

Read by byte

The younger sister was very satisfied, and she urged me: Brother F, I understand how to read characters, let's read the bytes.

I nodded, Junior Sister, do you remember the essence of philosophy? Bytes are the essence of java storage. Only by mastering the essence can all hypocrisy be broken.

Remember the Files tool class mentioned before? This tool class provides a lot of methods related to file operations, among them there is a method to read all bytes, the little sister should pay attention, here is to read all the bytes at once! It must be used with caution, it can only be used in scenes with fewer files, remember to remember.

public void readBytes() throws IOException {
        Path path = Paths.get("src/main/resources/www.flydean.com");
        byte[] data = Files.readAllBytes(path);
        log.info("{}",data);
    }

If it is a relatively large file, you can use FileInputStream to read a certain number of bytes at a time:

public void readWithStream() throws IOException {
        File file = new File("src/main/resources/www.flydean.com");
        byte[] bFile = new byte[(int) file.length()];
        try(FileInputStream fileInputStream  = new FileInputStream(file))
        {
            fileInputStream.read(bFile);
            for (int i = 0; i < bFile.length; i++) {
                log.info("{}",bFile[i]);
            }
        }
    }

Stream reads are read byte by byte. This will be slower. We use FileChannel and ByteBuffer in NIO to speed up some reading:

public void readWithBlock() throws IOException {
        try (RandomAccessFile aFile = new RandomAccessFile("src/main/resources/www.flydean.com", "r");
             FileChannel inChannel = aFile.getChannel();) {
            ByteBuffer buffer = ByteBuffer.allocate(1024);
            while (inChannel.read(buffer) > 0) {
                buffer.flip();
                for (int i = 0; i < buffer.limit(); i++) {
                    log.info("{}", buffer.get());
                }
                buffer.clear();
            }
        }
    }

Junior sister: If it is a very, very large file reading, is there a faster way?

Of course, remember the virtual address space mapping we talked about last time:

We can directly map the user's address space and the system's address space to the same virtual address memory at the same time, so as to avoid the performance overhead caused by copying:

public void copyWithMap() throws IOException{
        try (RandomAccessFile aFile = new RandomAccessFile("src/main/resources/www.flydean.com", "r");
             FileChannel inChannel = aFile.getChannel()) {
             MappedByteBuffer buffer = inChannel.map(FileChannel.MapMode.READ_ONLY, 0, inChannel.size());
             buffer.load();
            for (int i = 0; i < buffer.limit(); i++)
            {
                log.info("{}", buffer.get());
            }
            buffer.clear();
        }
    }

Find the number of rows with errors

Little Junior Sister: That's awesome! Brother F, you said so well. I have another problem with Junior Sister: I’m doing file parsing recently. Some of the files are not standardized, and the parsing fails halfway through the parsing, but there is no error prompting which line is wrong. It’s very difficult. Is there any good solution to the positioning problem?

See if it's getting late, brother will teach you another method. There is a class in java called LineNumberReader. Using it to read files can print the line number. Does it meet your needs:

public void useLineNumberReader() throws IOException {
        try(LineNumberReader lineNumberReader = new LineNumberReader(new FileReader("src/main/resources/www.flydean.com")))
        {
            //输出初始行数
            log.info("Line {}" , lineNumberReader.getLineNumber());
            //重置行数
            lineNumberReader.setLineNumber(2);
            //获取现有行数
            log.info("Line {} ", lineNumberReader.getLineNumber());
            //读取所有文件内容
            String line = null;
            while ((line = lineNumberReader.readLine()) != null)
            {
                log.info("Line {} is : {}" , lineNumberReader.getLineNumber() , line);
            }
        }
    }

Summarize

Today, I explained the character stream and byte stream to the younger sister, and also explained the basic method of file reading, which is worthwhile.

Chapter 5 File Writing Those Things

Introduction

Little Junior Sister also put a lot of strange demands on Senior Brother F. To format the output, to output a specific code, and to locate the output by yourself, what? Want to burn after reading it? Let's see how Brother F takes the move one by one.

Character output and byte output

Junior sister: Brother F, the last time you talked about IO is halfway, file reading is basically finished, but file writing has not been discussed yet. When will I give the junior sister a more popular science?

Junior sister: Brother F, you know that I have always been a model of diligence and studious. I am a good student in the eyes of teachers, a good role model in the hearts of classmates, and a good, well-behaved child around my parents. When I was climbing the peak of science forever, I found out that half of the knowledge hadn't been acquired. It really made me sigh, Brother F, quickly pass the knowledge to me.

Junior sister, your request, brother, I should do my best, but how do I remember that several days have passed since the last time I talked about IO file reading? Why did you come to me today?

Little Junior Sister blushed: Brother F, this is not when I encountered some problems when using it, so I wanted to review your knowledge again.

Let's go over the structure of the output class first:

The above are the two major output systems: Writer and OutputStream.

Writer is mainly for characters, while Stream is mainly for Bytes.

The most commonly used Writer is FileWriter and BufferedWriter. Let's look at the next most basic writing example:

public void useBufferedWriter() throws IOException {
        String content = "www.flydean.com";
        File file = new File("src/main/resources/www.flydean.com");

        FileWriter fw = new FileWriter(file);
        try(BufferedWriter bw = new BufferedWriter(fw)){
            bw.write(content);
        }
    }

BufferedWriter is an encapsulation of FileWriter, it provides a certain buffer mechanism, which can improve the efficiency of writing.

In fact, BufferedWriter provides three writing methods:

public void write(int c)
public void write(char cbuf[], int off, int len)
public void write(String s, int off, int len)

The first method passes in an int, the second method passes in the character array and the position and length to start reading, and the third method passes in the string and the position and length to start reading. Is it very simple and completely understandable?

Junior sister: No, Brother F, the parameters of the last two methods, whether char or String are characters, I can understand. What the hell is the first method passing int?

Little Junior Sister, have you forgotten all the truths I told you before? The underlying storage of int is bytes, and the underlying storage of char and String is also bytes. We just need to cast int and char as a coercion. Let's see how it is converted:

public void write(int c) throws IOException {
        synchronized (lock) {
            ensureOpen();
            if (nextChar >= nChars)
                flushBuffer();
            cb[nextChar++] = (char) c;
        }
    }

Remember how many bytes does an int take up? 4, char needs to occupy 2 bytes. This forced conversion from int to char will cause precision loss. Only the low 2 bytes of data will be retained, and the high two bytes of data will be discarded. This needs to be paid attention to during use.

After reading Writer, let's take a look at Stream:

public void useFileOutputStream() throws IOException {
        String str = "www.flydean.com";
        try(FileOutputStream outputStream = new FileOutputStream("src/main/resources/www.flydean.com");
            BufferedOutputStream bufferedOutputStream= new BufferedOutputStream(outputStream)){
            byte[] strToBytes = str.getBytes();
            bufferedOutputStream.write(strToBytes);
        }
    }

Like Writer, BufferedOutputStream is also an encapsulation of FileOutputStream. Let's look at the write method provided in BufferedOutputStream:

public synchronized void write(int b)
public synchronized void write(byte b[], int off, int len)

Compare the difference with Writer, the method of BufferedOutputStream is synchronized, and BufferedOutputStream directly operates on byte.

The int parameter passed to the first write method also needs to be intercepted, but this time it is converted from int to byte.

Formatted output

Junior sister: Brother F, the System.out.println we often use can directly output the formatted string to the standard output. Does the file writing have a similar function?

There must be, PrintWriter is used for formatted output:

public void usePrintWriter() throws IOException {
        FileWriter fileWriter = new FileWriter("src/main/resources/www.flydean.com");
        try(PrintWriter printWriter = new PrintWriter(fileWriter)){
            printWriter.print("www.flydean.com");
            printWriter.printf("程序那些事 %s ", "非常棒");
        }
    }

Export other objects

Junior sister: Brother F, we can output String, char and Byte. Can we output basic types such as Integer and Long?

Yes, you can do it with DataOutputStream:

public void useDataOutPutStream()
            throws IOException {
        String value = "www.flydean.com";
        try(FileOutputStream fos = new FileOutputStream("src/main/resources/www.flydean.com")){
            DataOutputStream outStream = new DataOutputStream(new BufferedOutputStream(fos));
            outStream.writeUTF(value);
        }
    }

DataOutputStream provides writeLong, writeDouble, writeFloat, etc. methods, and you can also writeUTF!

Write in a specific location

Junior sister: Brother F, sometimes we don't need to write to the file from the beginning every time, can we customize where to write it?

Just use RandomAccessFile:

public void useRandomAccess() throws IOException {
        try(RandomAccessFile writer = new RandomAccessFile("src/main/resources/www.flydean.com", "rw")){
            writer.seek(100);
            writer.writeInt(50);
        }
    }

RandomAccessFile can be located by seek, and then write from the specified location by the write method.

Lock file

Junior Sister: Brother F, there is one last question. How can I ensure that when I write files, others will not overwrite what I have written and that there will be no conflicts?

FileChannel can call tryLock method to obtain a FileLock lock, through this lock, we can control file access.

public void useFileLock()
            throws IOException {
        try(RandomAccessFile stream = new RandomAccessFile("src/main/resources/www.flydean.com", "rw");
        FileChannel channel = stream.getChannel()){
            FileLock lock = null;
            try {
                lock = channel.tryLock();
            } catch (final OverlappingFileLockException e) {
                stream.close();
                channel.close();
            }
            stream.writeChars("www.flydean.com");
            lock.release();
        }
    }

Summarize

Today, I gave the little sister a lot of ways to write documents, which will be enough for her to study for a while.

Chapter 6 Directory or File

Introduction

Directory and file are silly and unclear, what is the nature of directory and file? How to manipulate the directory in java, how to traverse the directory. In this article, Brother F will tell you one by one.

Files and directories in linux

Junior sister: Brother F, I recently had a doubt. It seems that there are only files and no directories in the java code. Isn't it the god who invented java in the first place?

Brother F: Junior sister is so courageous, dare to question authority is the most important step from a small worker to an expert. Think about me, Brother F. Since I was young, no one has mentioned anything. I believe what the teacher says, and I listen to what the expert says: the stock market must go up to 10,000 points. The house is for people to live in, not for speculation. Of course, crude oil treasures It is a must-have product for Xiaobai's financial management... Then, there is no more.

For more information, please visit www.flydean.com

Although there is no concept of directory in java, only File file, but File can actually represent directory:

public boolean isDirectory()

There is an isDirectory method in File, which can determine whether the File is a directory.

File and directory are silly and unclear. Little Junior Sister, do you think of anything?

Junior sister: Brother F, I remember you mentioned last time that all resources under Linux can be regarded as files. Are the essence of files and directories under Linux the same?

Yes, files under Linux are first-class citizens, and all resources are distinguished in the form of files.

We won't talk about the underlying structure such as sectors, logical blocks, and pages. Let's first consider what content a file should contain. In addition to the data of the file itself, there are also many metadata things, such as file permissions, owner, group, creation time and other information.

In the Linux system, these two parts are stored separately. The block that stores the data itself is called the block, and the block that stores the metadata is called the inode.

The address of the block is stored in the inode, and the block address where the actual data of the file is stored can be found through the inode to access the file. Consider that large files may occupy many blocks, so one inode can store the addresses of multiple blocks, and one file is usually enough to use one inode.

In order to show the hierarchical relationship and facilitate the management of files, the data files of the directory store the files in the directory and the inode addresses of the files, thus forming a chain relationship of ring-to-ring, ring-to-ring.

The above figure lists a ring-in-ring layout for searching files under it through a directory.

I think the reason why a category is not listed separately in the java directory may be based on the underlying file layout of Linux.

Basic operation of the directory

Because directories and files in Java share the File class, all the basic operations of File are used for directories.

Basically, the following three types of methods should be paid more attention to when comparing directories and files:

public boolean isDirectory()
public File[] listFiles() 
public boolean mkdir() 

Why are there three categories? Because there are several methods that are relatively close to them, I won't list them all here.

isDirectory determines whether the file is a directory. listFiles lists all the files under the directory. mkdir creates a file directory.

Junior sister: Brother F, before we used the directory traversal to take a long time. After you explain the data structure of the directory, I feel that listFiles is not a time-consuming operation, all the data is ready, read it directly Just take it out.

Yes, do not look at the surface when looking at the problem, but at the essential connotation hidden on the surface. You see, brother, I don't usually show up. In fact, I am a real mainstay and can be called a model of the company's outstanding employees.

Little Junior Sister: Brother F, you don't usually pay attention to commend you or something? Oh, I see, it must be the boss who is afraid that commending you will cause others to be jealous, and it will collapse your image as a good big brother. It seems that the boss really understands you.

Advanced operation of the directory

Okay, junior sister, as long as you understand, Brother F will tell you about the advanced operations of the directory, for example, how do we copy a directory?

Junior sister, Brother F, who has a simple copy of the directory, you taught me last time:

cp -rf

Wouldn't the matter of one order be solved? Is there some secret hidden inside?

Cough cough cough cough, there is no secret, little sister, I remember you said last time that you want to complete java. Today, brother introduces you to a method of copying file directories in java.

In fact, the Files tool class has provided us with an excellent method of copying files:

public static Path copy(Path source, Path target, CopyOption... options)

Using this method, we can copy files.

If you want to copy a directory, you can traverse the files in the directory and call the copy method in a loop.

Junior Sister: Wait a minute, Brother F, if there are still directories under the directory, what should I do if there are still directories under the directory?

This is a trap. Let me solve it with a recursive method:

public void useCopyFolder() throws IOException {
        File sourceFolder = new File("src/main/resources/flydean-source");
        File destinationFolder = new File("src/main/resources/flydean-dest");
        copyFolder(sourceFolder, destinationFolder);
    }

    private static void copyFolder(File sourceFolder, File destinationFolder) throws IOException
    {
        //如果是dir则递归遍历创建dir，如果是文件则直接拷贝
        if (sourceFolder.isDirectory())
        {
            //查看目标dir是否存在
            if (!destinationFolder.exists())
            {
                destinationFolder.mkdir();
                log.info("目标dir已经创建: {}",destinationFolder);
            }
            for (String file : sourceFolder.list())
            {
                File srcFile = new File(sourceFolder, file);
                File destFile = new File(destinationFolder, file);
                copyFolder(srcFile, destFile);
            }
        }
        else
        {
            //使用Files.copy来拷贝具体的文件
            Files.copy(sourceFolder.toPath(), destinationFolder.toPath(), StandardCopyOption.REPLACE_EXISTING);
            log.info("拷贝目标文件: {}",destinationFolder);
        }
    }

The basic idea is that I traverse a directory when I encounter it, and copy a file when I encounter it.

Back pain operation of the catalog

Junior sister: Brother F, if I want to delete files in a directory, or we want to count how many files there are in this directory, what should I do?

Although these operations are a bit painful, they can be solved. There is a method in the Files tool class called walk, which returns a Stream object. We can use the Stream API to process files.

Delete Files:

public void useFileWalkToDelete() throws IOException {
        Path dir = Paths.get("src/main/resources/flydean");
        Files.walk(dir)
                .sorted(Comparator.reverseOrder())
                .map(Path::toFile)
                .forEach(File::delete);
    }

Statistics file:

 public void useFileWalkToSumSize() throws IOException {

        Path folder = Paths.get("src/test/resources");
        long size = Files.walk(folder)
                .filter(p -> p.toFile().isFile())
                .mapToLong(p -> p.toFile().length())
                .sum();
        log.info("dir size is: {}",size);
    }

Summarize

This article introduces some very common and useful operations of the directory.

Chapter 7 File System and WatchService

Introduction

This time, Junior Sister encountered the problem of monitoring file changes. Senior Brother F introduced WatchService introduced in JDK7 nio to Junior Sister. Unexpectedly, she also popularized the concept of file system by the way. She never expected it.

Pain points of monitoring

Junior sister: Brother F, have you felt a little difficulty breathing recently, the back collar is a little bit chilly, the kind of talking a little bit unsmoothly?

No, Junior Sister, did you wear the Qiuyi upside down?

Junior sister: No, Brother F. What I'm talking about is the feeling in my heart, that kind of unnecessarily pressure, and a trace of throbbing entangled in my heart.

Don't go around the corner, Junior Sister, are you having problems again?

For more information, please visit www.flydean.com

Junior sister: Brother F still understands me. This is not very easy to use the Properties file last time. It is really painful to restart the java application every time I modify the Properties file. Is there any other way?

Of course there is a way. The most basic way is to open a thread to monitor the last modification time of the attribute file regularly, and reload it if it is modified. This will not work.

Junior sister: Writing threads is so troublesome, is there any easier way?

I know you want to ask this question. Fortunately, I have prepared enough. Today I will introduce you a class WatchService introduced by JDK7 in nio.

WatchService and file system

WatchService is an interface introduced in nio by JDK7:

The monitored service is called WatchService, and the monitored object is called Watchable:

WatchKey register(WatchService watcher,
                      WatchEvent.Kind<?>[] events,
                      WatchEvent.Modifier... modifiers)
        throws IOException;
WatchKey register(WatchService watcher, WatchEvent.Kind<?>... events)
        throws IOException;

Watchable registers the WatchEvent of the object to WatchService through register. Since then, as long as a WatchEvent occurs on the Watchable object, the WatchService will be notified.

There are four types of WatchEvent:

1. ENTRY_CREATE target is created
2. ENTRY_DELETE target is deleted
3. ENTRY_MODIFY target was modified
4. OVERFLOW A special Event, which means that the Event is abandoned or lost

The WatchKey returned by the register is the collection of WatchEvents monitored.

Now look at the 4 methods of WatchService:

1. close close watchService
2. poll gets the next watchKey, if not, returns null
3. The poll with time parameters obtains the next watchKey within a certain period of time.
4. take to get the next watchKey, if not, wait forever

Junior sister: Brother F, how can we build a WatchService?

Remember the file system mentioned in the last article, there is a way to get WatchService in FileSystem:

public abstract WatchService newWatchService() throws IOException;

Let's look at the structure diagram of FileSystem:

On my mac system, FileSystem can be divided into three categories, UnixFileSystem, JrtFileSystem and ZipFileSystem. I guess there should be corresponding windows-related file systems on windows. If you are interested, you can check it out, Junior Sister.

Little sister: UnixFileSystem is used to process files under Unix, and ZipFileSystem is used to process zip files. What is JrtFileSystem used for?

Oh, this is going to be farther away, why do you have to pull to the horizon every time you ask a question...

When the JDK was still 9, a very big change was made called modular JPMS (Java Platform Module System). This Jrt is for modular systems. Let’s take an example:

public void useJRTFileSystem(){
        String resource = "java/lang/Object.class";
        URL url = ClassLoader.getSystemResource(resource);
        log.info("{}",url);
    }

In the above piece of code, we get the url of the Object class. Let's see if it is in JDK8, what is the output:

jar:file:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/rt.jar!/java/lang/Object.class

The output result is jar:file indicating that this Object class is placed in the jar file, followed by the path of the jar file.

If it is after JDK9:

jrt:/java.base/java/lang/Object.class

The result is the beginning of jrt, java.base is the name of the module, followed by the path of the Object. It seems that it is more concise and clear than the traditional jar path.

With the file system, we can get the corresponding WatchService while getting the default file system of the system:

WatchService watchService = FileSystems.getDefault().newWatchService();

The use and realization of WatchSerice essence

Junior sister: Brother F, how did WatchSerice come true? It’s so amazing and saved us so much work.

In fact, the purpose of JDK providing so many classes is to prevent us from repeating the wheel. I told you that the easiest way to monitor files is to open a separate thread to monitor file changes? Actually... WatchService does just that!

PollingWatchService() {
        // TBD: Make the number of threads configurable
        scheduledExecutor = Executors
            .newSingleThreadScheduledExecutor(new ThreadFactory() {
                 @Override
                 public Thread newThread(Runnable r) {
                     Thread t = new Thread(null, r, "FileSystemWatcher", 0, false);
                     t.setDaemon(true);
                     return t;
                 }});
    }

The above method is the method to generate WatchService. The little sister sees that it does not. Its essence is to open a daemon thread to receive monitoring tasks.

Let's see how to register a file to WatchService:

private void startWatcher(String dirPath, String file) throws IOException {
        WatchService watchService = FileSystems.getDefault().newWatchService();
        Path path = Paths.get(dirPath);
        path.register(watchService, ENTRY_MODIFY);

        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            try {
                watchService.close();
            } catch (IOException e) {
                log.error(e.getMessage());
            }
        }));

        WatchKey key = null;
        while (true) {
            try {
                key = watchService.take();
                for (WatchEvent<?> event : key.pollEvents()) {
                    if (event.context().toString().equals(fileName)) {
                        loadConfig(dirPath + file);
                    }
                }
                boolean reset = key.reset();
                if (!reset) {
                    log.info("该文件无法重置");
                    break;
                }
            } catch (Exception e) {
                log.error(e.getMessage());
            }
        }
    }

The key method above is path.register, where Path is a Watchable object.

Then use watchService.take to get the generated WatchEvent, and finally process the file according to the WatchEvent.

Summarize

Dao begets one, one life two, two begets three, three begets everything. A simple function is actually hidden behind...the Tao Te Ching, oh, no, behind it is the philosophy of Tao.

Chapter 8 File and Path

Introduction

What is the relationship between file and path? What secrets are hidden in files and paths? Under the management of the file system, what are the ways to create a path? Today, Senior Brother F and Junior Sister will give you another wonderful performance.

File and path

Junior sister: Brother F, I have a question. The file File in java is a class that can be understood because the file contains a lot of other information, but why should the path be a separate class? Isn't it simpler to use only one String?

For more information, please visit www.flydean.com

Everything has a reason, there is no love without a reason, and no hate without a reason. Everything is really wonderful.

Let's look at the definition of File and path:

public class File
   implements Serializable, Comparable<File>

public interface Path
    extends Comparable<Path>, Iterable<Path>, Watchable

First of all, File is a class, which represents the attributes and functions that all file systems have. Whether you are windows or linux, the File objects in them should be the same.

File contains Path. Let’s take a look at it. Path is an interface. Why is it an interface? Because Path can be divided into JrtPath, UnixPath and ZipPath according to different situations. The FileSystem corresponding to the three Paths has been discussed in the previous article. So the realization of Path is different, but the File containing Path is the same.

Junior sister: Brother F, why is this so awkward? Give me a straightforward explanation.

In this case, let me explain: the patriotic version, maybe we belong to different nationalities, but we are all Chinese. In the popular version, everyone is a cultural person, so why are you so dragging? The cultural version, in the same nine years, Ru Hexiu?

Looking at the implementation interfaces of the two, File implements Serializable to indicate that it can be serialized, and implements Comparable, which indicates that it can be sorted.

Path inherits Comparable, which means it can be sorted. Inheriting Iterable means that it can be traversed, and it can be traversed because Path can represent a directory. Inherit Watchable, which means it can be registered in WatchService for monitoring.

Different paths in the file

Junior sister: Brother F, there are several get methods for Path in File. Can you tell me the difference?

Directly on the code:

public void getFilePath() throws IOException {
        File file= new File("../../www.flydean.com.txt");
        log.info("name is : {}",file.getName());

        log.info("path is : {}",file.getPath());
        log.info("absolutePath is : {}",file.getAbsolutePath());
        log.info("canonicalPath is : {}",file.getCanonicalPath());
    }

There are three Path-related methods in File, namely getPath, getAbsolutePath and getCanonicalPath.

The result returned by getPath is the path passed in when new File is entered, and what is returned is what is entered.

What getAbsolutePath returns is the absolute path, that is, the current path is added in front of getPath.

What getCanonicalPath returns is the simplified AbsolutePath, that is, the reference symbols such as. or .. are removed.

Look at the output:

 INFO com.flydean.FilePathUsage - name is : www.flydean.com.txt
 INFO com.flydean.FilePathUsage - path is : ../../www.flydean.com.txt
 INFO com.flydean.FilePathUsage - absolutePath is : /Users/flydean/learn-java-io-nio/file-path/../../www.flydean.com.txt
 INFO com.flydean.FilePathUsage - canonicalPath is : /Users/flydean/www.flydean.com.txt