[DBA 100 Persons] Hu Zhonghao: The Making of a Domestic Distributed Database DBA

The "DBA 100" exclusive interview program is an exclusive interview activity conducted by OceanBase around senior DBAs (Database Administrators). The idea of technology trends, I hope their growth path can give DBAs in various industries some advice and thinking.

The second issue of "DBA 100 People" will take you to know Hu Zhonghao, a database management engineer in the Information Technology Department of a bank in Yunnan - with Oracle OCP, OCM certification, MySQL OCP certification, OceanBase OBCP certification expert, Jushan database SCDD certification expert, I hope his Experience can bring you reference value.

---

With the launch of the new core system, Hu Zhonghao, engineer of the operation and maintenance center of the information technology department of a bank in Yunnan, also has a new label - domestic distributed database DBA. Hu Zhonghao participated in the construction of the new core system database of a bank in Yunnan throughout the process, from early market research, mid-term POC and bidding, and later deployment and launch, so he has become one of the few DBAs in the banking system who truly understands the domestic distributed database. . Although he has experienced many challenges and tortures in the process, Hu Zhonghao thinks it is very worthwhile. It is these that make Hu Zhonghao change from an Oracle DBA to a domestic distributed database DBA.

into the "pit"

Hu Zhonghao graduated from North China University of Technology in 2014, majoring in geographic information system. Although he has learned some database knowledge in university, including Microsoft's SQL Server and geographic database ArcCatalog, etc., before he took part in the graduation practice in his senior year, Hu Zhonghao had no special interest in databases, and he never thought that he would become a DBA in the future. This is the last time The internship gave Hu Zhonghao the idea of becoming a DBA.

"In the unit where my senior year practiced in Beijing, there was an engineer in charge of Oracle's technical support, which left a deep impression on me. At the time, I felt that he was very powerful, and only he could solve many problems, and everyone respected him. It can be seen that, He is also proud to be an Oracle DBA. I wanted to be like him, so I came up with the idea of being an Oracle DBA." Hu Zhonghao said.

It happened that Hu Zhonghao had an acquaintance who worked in Oracle. He supported Hu Zhonghao's idea very much. He introduced Hu Zhonghao to an Oracle technical training class through his own relationship and participated in a 4-month practical training. 4 months later, Hu Zhonghao officially took up the position of DBA. To this day, although the job position has changed a lot, he has always been engaged in database-related work.

Hu Zhonghao said that in these years of work experience, the most unforgettable one is the more than one year that he worked as a resident engineer in Yunnan Mobile as a database operation and maintenance personnel. Talking about that experience, Hu Zhonghao still has a fresh memory and is full of gratitude. It was the most stressful, busiest, most rewarding and fastest growing period of time he had ever experienced.

"Basically, I have encountered all the failure scenarios that Oracle may have encountered in the market. What is particularly important is that my psychological endurance has been greatly exercised." Hu Zhonghao said.

At that time, Hu Zhonghao was in a state of being able to work at any time, even when he was eating a bowl of noodles outside, he also brought his computer with him. Once you receive a call, you may have to turn on the hotspot of your mobile phone and start working. As for being woken up in the middle of the night, it is more commonplace. One of these experiences still makes Hu Zhonghao feel natural. That time, Yunnan Mobile updated a system, just in time for the mobile membership day, the transaction volume was very large, the system was stuck, and the surrounding systems were affected. In the end, people could not pay the phone bills, and the complaint calls exploded, and the leaders were very anxious. .

"At that time, the customer leader was standing behind me, watching me find the cause of the failure. At that moment, the pressure was too great." Hu Zhonghao said.

Fortunately, Hu Zhonghao, with his solid basic skills, quickly found out the reason and gave a solution: two indexes were established, and the problem was quickly solved. At that moment, Hu Zhonghao seemed to have found the status of the Oracle technical support staff he had seen during his internship. It was this experience that gave Hu Zhonghao a personal understanding of his ability to withstand pressure.

During that time, he also delivered the databases of many large enterprises in Yunnan Province, such as: Power Grid, Fudian Bank, Yunnan Baiyao, Chengtai Insurance, Yunnan Telecom, logistics, hospitals, etc., which has accumulated very rich practical experience for Hu Zhonghao, It also gave him full confidence. When a bank in Yunnan recruited a DBA in 2018, Hu Zhonghao passed the interview with his previous efforts and became a DBA of a bank in Yunnan.

core system to be replaced

After entering a bank in Yunnan, Hu Zhonghao and another colleague were responsible for more than 100 sets of Oracle databases and dozens of DB2 and MySQL databases of the bank. The daily work is smooth and orderly. Compared with the previous resident engineer, the pressure is much less, but occasionally there are some challenging tasks, such as building the first Oracle RAC cluster for a bank in Yunnan.

This smooth and orderly day came to an abrupt end with the new core system project. In October 2020, the leader told Hu Zhonghao that the bank's core system needs to be replaced. For the consideration of localization and the company's future development, the new core system is going to use a domestic distributed database, so Hu Zhonghao will prepare.

Hu Zhonghao is not surprised that the core system needs to be changed. Now this core system is built on the IBM AIX small computer. The system and database are all on this machine. It has been running for many years. It has become a bottleneck and urgently needs to be upgraded, but it was unexpected to directly access the domestic distributed database.

After receiving the task, Hu Zhonghao found that all his work experience in these years was around Oracle, DB2 and MySQL, and his knowledge of domestic databases was really limited. What is the current state of the domestic database, who are the mainstream players in the market, what are its own advantages and disadvantages compared with Oracle, and how is the ecological construction? These problems are waiting for Hu Zhonghao to understand and learn.

Fortunately, there are many years of experience in using Oracle database as the basis, these problems can not help Hu Zhonghao. After a period of research and contact with manufacturers, I have a preliminary understanding of the market, and 4 candidate manufacturers including OceanBase have been finalized to come to a bank in Yunnan for on-site testing and POC.

During this time, all Hu Zhonghao's work energy was devoted to the new database, and he was faced with new knowledge every day, which made Hu Zhonghao feel full and excited, and also gave Hu Zhonghao a more comprehensive understanding of the domestic database.

He said that domestic distributed databases can be roughly divided into two categories: one is a distributed database developed based on an open source database engine and implemented using sub-databases and sub-tables, and the other is a native distributed database. Databases based on sub-databases and sub-tables have better linear scalability, and most of the bottom layers use open source engines, so the development cost is relatively low, but there are shortcomings in the implementation of distributed transactions, global clocks, and global indexes. The native distributed database has the characteristics of high availability and high elasticity, and because it does not need to be divided into databases and tables, it is not necessary to perceive the distribution of data, and it is less intrusive to applications.

"At that time, after we tested and evaluated, we tended to the cloud-native distributed database." Hu Zhonghao said. Later, in order to make the preparations more solid, a bank in Yunnan specially visited a bank that has deployed a domestic distributed database. In addition to understanding their use experience, it focused on global distributed transactions, global clocks and global indexes. These are very crucial questions. An open tender was held in April 2021, and OceanBase was selected.

"The final selection of OceanBase is a decision made by combining various factors, in addition to technology and ecology, including the strength of manufacturers. After all, we don't want to find a company in a few years." Hu Zhonghao said.

The domestic distributed database is amazing

From October 2020, Hu Zhonghao began to pay attention to the domestic database. By the end of May, OceanBase was successfully launched. In the past two years, Hu Zhonghao devoted himself to the domestic database. Talking about the domestic database, he said that the domestic database gave him a lot of surprises. He never thought that the technology of the domestic database would advance so fast.

Hu Zhonghao introduced that compared with Oracle, the three characteristics of the domestic distributed database left a deep impression on him. These are high availability, high elasticity and high concurrency load capacity, which have brought great advantages to the database operation and maintenance of a bank in Yunnan Comes with a lot of convenience.

For example, the core system of a bank in Yunnan adopts the deployment mode of two places, three centers, and five copies. Now any local system is down without affecting the business, and the domestic distributed database can easily realize active-active. The fault can be automatically switched, and the maintenance is very convenient.

"During the test, we have used many extreme measures, including killing processes, shutting down servers, disconnecting the network, and pulling out hard drives. Basically, the business can be restored within 30 seconds." Hu Zhonghao said.

Similarly, the domestic database's ability to support elastic expansion and contraction and high concurrency also makes the DBA's work much easier. For example, when OceanBase expands, you only need to connect the device and configure it, and it will not interfere with the normal operation of the business. In the past, when a centralized database was used, the time for adding a node and adding a storage device must be carefully selected, because the expansion and contraction It will seriously affect the normal operation of the business.

"Although the stand-alone performance is weaker, and the domestic distributed database is not as good as the foreign centralized database, the domestic database can increase equipment almost infinitely, and the performance will increase linearly, which is very convenient." Hu Zhonghao said.

DBAs need to stay awake at all times

Since he entered the DBA industry under the guidance of a friend, Hu Zhonghao has been struggling in the database field for years, and he is no longer the ignorant teenager who just entered the industry. Years of experience have given him a deeper understanding of this profession. Looking back on the experience of these years, he is very grateful to the big brother who brought him into the industry, and made him take a lot of detours.

He said that DBA is a profession with relatively high requirements for practitioners, and it is necessary to have a clear understanding of this before entering the industry.

"DBA first requires rigorous, careful and strong psychological quality, which is more important than professional technical knowledge, followed by continuous learning and good communication skills." He said.

Hu Zhonghao especially emphasized learning ability. He said that in the past, being an Oracle DBA was very prestigious, and the job was easy to find, but today's localization driven by new technologies and the rise of domestic databases themselves have made great changes in the database industry, and DBAs must continue to learn Keeping up with changes is what makes learning so important today.

"When you have time, you must read more official documents, build a test environment yourself, conduct tests, pay more attention to summarizing, and must summarize more." Hu Zhonghao suggested.

Hu Zhonghao revealed that he has a habit that he has maintained for many years, that is, to write work notes, to record and summarize what he has learned every day, including the selection of the new database and the entire process of going online, Hu Zhonghao has recorded it. "This is a summary and a record. When you look back, you can see your progress little by little."

Talking about future plans, Hu Zhonghao said that the short-term goal is to get the highest level of OceanBase certification OBCE (Database Certified Master) this year; the long-term goal is to learn more about business in addition to mastering database technology, and turn yourself into a knowledgeable An expert in the business, not just a techie.

"Through continuous learning, the technology has grown, and at the same time, the psychological endurance has kept up. Today, no matter what problems I encounter today, I can face it confidently and calmly. This is my biggest gain over the years." Hu Zhonghao concluded.