Distributed | dble metadata update synchronization

Author: Wu Jinling

A member of the Ecoson dble project team, mainly responsible for the daily testing of dble, and is good at troubleshooting problems in dble. I love testing, and I want to carry it through to the end for the rest of my life.

Source of this article: original submission

* Produced by the Aikesheng open source community, original content is not allowed to be used without authorization, please contact the editor and indicate the source for reprinting.

---

Synchronize metadata, reload @@config_all or reload @@metadata?

1. This article takes the dble 3.21.06.0 version as an example. First, let's take a look at several types of problems that the community often encounters that tables or table fields cannot be found
The key configuration of sharding.xml is as follows:

Question 1 : After starting dble, the table sharding_2_t1 is created on all back-end nodes, and then the reload @@config_all command is executed on the dble management terminal to query the table sharding_2_t1 data and insert data into the table sharding_2_t1, but the table is reported to be missing Metadata information (as shown in the figure below), why?

Question 2 : Add an age field to the table sharding_4_t1 on the back-end node, and ensure that all back-end nodes have added this field, and then execute reload @@config_all to query the table and use the new field is no problem, but Using order by, it prompts that this field cannot be found (as shown in the figure below), why?

2. Before answering the reasons for the above problems, let us first take a look at the instructions for reload @@config_all and reload @@metadata (for details, please refer to the official dble document: https://github.com/actiontech/dble- docs-cn/blob/master/2.Function/2.1_manager_cmd/2.1.07_reload.md), so that we can better understand the problem

1.reload @@config_all (The function is completely equivalent to reload @@config after version

Reload all configurations, involving 3 xml configuration files (user.xml, db.xml, sharding.xml ), and at the same time reload @@config_all has 3 optional parameters: reload @@config_all [-s] [- f] [-r];

-s Skip testing the back-end link. By default, the back-end link in the configuration will be tested without adding this parameter. If the test fails, ERROR will be returned;

-f Close all changed dbGroups (all dbGroups will be regarded as changes if the -r parameter is added at the same time) related front-end links in the transaction. If there is no such parameter, only the corresponding back-end links will be put into the old link pool by default.

-r Do not make intelligent judgments, and reload all back-end connection pools. If this parameter is not added, the new configuration will be intelligently judged, and only the connection pool that has been added, deleted, or modified will be changed, and the connection pool that has not been changed will not be affected.

related impact : When this command is executed, when the configuration information of the table is changed, the meta information of the involved table will be reloaded, otherwise the original table meta information will be maintained. In addition, if the -r parameter is included, the above judgment will not be made, and all meta data will be reloaded. If the -r is not included but the -s parameter is included, when calculating whether the metadata needs to be reloaded, all dbGroup/dbInstance changes are ignored. Note that some changes that cannot be reflected in configuration changes cannot be reloaded in metadata. For example, a schema with a default shardingNode tries to change the split table or global table into a non-split table by deleting the configuration, which is not in compliance with the specification. This operation should be avoided.

2.reload @@metadata

The function of reload @@metadata is to reload all metadata information, and there are also two forms of extended filter expressions:

reload @@metadata where schema=? [and table=? ]: Reload all tables in the specified schema or the metadata information of the specified table.

reload @@metadata where table in ('schema1.table1' ,'schema2.table2','schema1.table3',...): reload the metadata information of table1, table3 and table2 in schema1.

reload @@config_all and reload @@metadata, it can be seen that the main difference between the two when loading metadata is: reload @@config_all will only reload metadata for changes in the configuration, and configure Items that have not been changed need to be manually reload @@metadata to synchronize the changed metadata.

3. After understanding the difference between reload @@config_all and reload @@metadata, let's analyze the reasons for the first two problems.

It is not difficult to see that the errors of question 1 and question 2 are all due to the fact that after the table is built or the table structure is modified on the back-end node when the configuration has not changed, it is believed that the meta of the changed table can be synchronized by executing reload @@config_all Caused by data. Users here have a common misunderstanding. They think that executing reload @@config_all in all scenarios will trigger the reloading of metadata, but in fact the metadata will be updated synchronously only when the configuration changes at the same time. Other scenarios require manual reload @ @metadata synchronizes the metadata of the backend table to dble.

In addition, why the select age from sharding_4_t1 in question 2 can be successfully queried? This is because this query is a simple query. You can directly issue the statement to the back-end node for execution. The dble does not need to use the age field, and the back-end node has an age field. , So the execution was successful.

4. In order to facilitate the troubleshooting of metadata-related errors, you can use check full @@metadata to view the metadata information of all tables in the current dble.

check full @@metadata also supports filtering by multiple conditions (for details, please refer to dble's official document: https://github.com/actiontech/dble-docs-cn/blob/master/2.Function/2.1_manager_cmd/2.1 .17_check_meta_data.md)

Through the above query results, it is not difficult to find that the metadata of table sharding_2_t1 and table sharding_4_t1 are not synchronized to the update.

After executing reload @@metadata, execute check full @@metadata again to query metadata information:

From the above results, you can see that the table sharding_2_t1 and table sharding_4_t1 have been synchronized to the correct metadata information. At this time, you can perform routine operations on the table without reporting an error: