Distributed | How to contribute source code for DBLE? What are the skills required for middleware development?

Author: Lulu

A technical person who loves technology and is willing to share. Currently, he is mainly engaged in the research of database-related technologies. Update the personal public account WU from time to time.

Source of this article: original submission

* Produced by the Aikesheng open source community, original content is not allowed to be used without authorization, please contact the editor and indicate the source for reprinting.

Preface

A good open source product first needs a mature and stable version, and secondly it needs an active community. Mature and stable software can attract users to build a product community, which in turn can promote the further development and improvement of the product.

Therefore, the product and the community have an interactive and complementary relationship.

Today I want to introduce you to the skills required for middleware development. Maybe you can attract students who are interested in middleware development, and then become a source code contributor to DBLE.

This article first introduces the general skills required for middleware development. These skills are not only required for DBLE development, but are also required for all middleware development. Then go a step further and introduce the special skills required for database middleware development, which may be different from other middleware development skills.

General skills required for middleware development

Middleware is an independent system software service program that serves specific applications. Common middleware includes database middleware, message middleware, and cache middleware.

The technical requirements for middleware development can be said to be relatively high. The following is my summary of the more important technologies required for middleware development.

1. Multithreading

Multi-threading technology is one of the core technologies that must be mastered in middleware development because of its effect on performance.

Multi-threading can make full use of the performance of multi-core CPUs, thereby increasing the throughput of the system, and the thread pool technology can reduce the overhead of thread creation and destruction, thereby further improving the performance of the system.

Of course, multi-threading also brings some problems. Visibility, orderliness, and liveness are all problems caused by multi-threading technology. Some people say that performance improvement is the only thing he can tolerate these problems caused by multi-threading. reason.

In any case, multi-threading technology is a necessary skill for middleware development. Of course, learning multi-threading technology is not so easy. Don’t believe that an article on the Internet is proficient in multi-threading. There is never an article that can make you proficient in multi-threading. There is no shortcut to learning knowledge.

For JAVA multi-threaded learning, I recommend "JAVA Concurrent Programming Practice". After all, the author is a member of the JCP expert group, and JAVA programmers can learn multi-threading by reading the JUC source code package completely.

2. Network programming

Network programming can also be said to be one of the most important technologies for the development of middleware.

The establishment of network connections, the reading and writing of network data packets, and the processing of network protocols are the foundation and difficulty of middleware. Of course, there are some network frameworks such as Netty that can help us deal with these problems well, but for middleware developers, it is better to be familiar with network programming, so that when facing some network problems, we can also not The confidence of being in a panic, knowing what is happening, and knowing why, should be able to be well reflected at this time.

For DBLE network processing, I have written a series of source code interpretation articles before. DBLE's network modules are all implemented with JDK native IO packages. Reading through it should help you get a general understanding of network programming.

3. Design pattern

Middleware is different from business development. It cannot simply add code and logic as soon as it meets new requirements. Instead, it must be considered from the perspective of the overall architecture. If necessary, existing modules need to be reconstructed to add new function extensions.

Therefore, for middleware development, we need to master various design patterns, and we also need to master refactoring-related skills.

The book "Design Patterns: Fundamentals of Reusable Object-Oriented Software" is recommended for design patterns, and the book "Refactoring: Improving the Design of Existing Code" is recommended for refactoring skills. Similarly, there is no shortcut to knowledge learning, you still need to be patient to finish reading, and to understand through practice, in order to master it.

4. Asynchronous programming

High-performance frameworks generally have a lot of asynchronous logic, and process control is achieved through asynchronous callbacks. While asynchronous programming achieves high performance, it also loses a certain degree of readability, because the logic of the code may be fragmented. But for high-performance applications, asynchronous programming also needs to be learned.

5. Algorithm and data structure

Algorithms and data structures are the basic disciplines of computers, so for middleware, they are naturally inseparable.

6. Test

How to ensure the quality of the product, in addition to the standardization of the software development process, code review, etc., testing can be said to be the most important part.

Testing includes the most basic unit testing, integration testing, performance testing, fatigue testing, etc. For middleware developers, these are all testing skills that need to be mastered.

7. Performance tuning

What determines the system performance bottleneck is the serial part of the system and the interaction part with the external system.
The first step in performance tuning is to locate the performance bottleneck of the system, which of course requires you to have a deep understanding of the system, as well as the assistance of measurement tools.

The idea of performance tuning includes parallelizing serial tasks and asynchronous tasks.

The above are the general skills required for middleware development in my personal opinion, which may not be complete, but the most basic and important content should be included.

Special skills required for database middleware development

What are the more keywords that database middleware compares with middleware? The answer is "database". Therefore, the special skills required for database middleware development are actually skills related to database technology.

In fact, for database middleware developers, they will find themselves getting closer and closer to the database in the future.

The special skills required for database middleware development are as follows.

1、 SQL

SQL routing, SQL rewriting and other functions of database middleware are all based on the understanding of SQL. So SQL parsing is one of the important functions of database middleware.

SQL as a database query language, the analysis of SQL is the same as that of any other language, including lexical analysis and syntax analysis. I also wrote an article about DBLE SQL parsing before. You can search for it in the official account for further reading.

After SQL is parsed, the next step is naturally to execute. Although most scenarios of database middleware can push SQL down to the database node for execution, some complex SQL still needs to be processed at the middleware layer, and execution is involved here. Optimization of the plan, return result processing algorithm, etc., these contents can be said to belong to the important and difficult content.

2. Affairs

Transaction is the core concept of database, and transaction is also important for database middleware. Especially in the scenario of distributed transactions, database middleware acts as a transaction coordinator. Understanding of transactions can help you understand how database middleware guarantees data consistency in this scenario.

3. Database Agreement

The database middleware actually acts as a database server externally. This involves the processing of the database protocol.

DBLE implements the MySQL protocol, so when you connect to DBLE, you will find the same effect as connecting to MySQL.
For the development of DBLE, you need to have a certain understanding of the MySQL protocol. For the MySQL protocol, you can refer to the link below.

https://dev.mysql.com/doc/internals/en/client-server-protocol.html

4. Copy

The high availability of the database can only be achieved through replication.

For database middleware, the read-write separation function can use the database to read from the node, thereby increasing the throughput of the application and reducing the load on the master node. Therefore, database middleware developers also need to understand the master-slave replication of the database.

At last

This article lists some of the skills required for middleware development, hoping to give everyone a certain understanding of middleware development, and further hope that it can help students who are interested in middleware development to start learning.

Some students may think that there are so many things to learn, when will they really have the skills to develop middleware?

In fact, we don't need to wait for things to be completely prepared before doing it, because you may feel that the preparations are never sufficient. So you might as well find a simple entry point to start practicing. For example, for DBLE a small bug fix, a configuration modification, try to participate in the modification of the source code, and I believe that the official DBLE team will also give you corresponding help in the process.

I hope today's article can help you understand the development of middleware.