In-depth analysis of the entire process of Dubbo 3.0 server exposure

Introduction to With the advent of the cloud native era, a very important goal of Dubbo 3.0 is to fully embrace cloud native. Because of this, Dubbo 3.0 evolved the original interface-level service discovery mechanism into an application-level service discovery mechanism in order to better adapt to cloud native.

Author introduction

Xiong hire, Github account pinxiong, Apache Dubbo contributor, focusing on RPC, Service Mesh and cloud native fields. Currently working in the R&D team of Ctrip International Division, responsible for marketing, cloud native and other related work.

background

With the advent of the cloud native era, a very important goal of Dubbo 3.0 is to fully embrace cloud native. Because of this, Dubbo 3.0 evolved the original interface-level service discovery mechanism into an application-level service discovery mechanism in order to better adapt to cloud native.

Based on the application-level service discovery mechanism, Dubbo 3.0 can greatly reduce the additional resource consumption brought by the framework and greatly increase the resource utilization rate, which is mainly reflected in:

• Single-machine resident memory decreased by 75%
• The cluster instance scale that can be supported is millions of clusters
• The overall data volume of the registry dropped by more than 90%

At present, there are many technical articles about the Dubbo server-side exposure process, but they are all interpreted based on the Dubbo interface-level service discovery mechanism. Under Dubbo 3.0's application-level service discovery mechanism, the server-side exposure process has changed a lot from before. This article hopes to analyze the whole process of server-side exposure by understanding the source code of Dubbo 3.0.

What is application-level service discovery

To put it simply, Dubbo used to register all interface information in the registration center, and an application instance generally has multiple interfaces, so the amount of registered data is much larger and redundant. The mechanism of application-level service discovery is that the same application instance only registers one piece of data in the registry. This mechanism mainly solves the following problems:

• Align with mainstream microservice models, such as Spring Cloud
• Support Kubernetes native service. The maintenance and scheduling services in Kubernetes are all based on the application instance level and do not support the interface level
• Reduce the data storage capacity of the registry and reduce the pressure of address change push

Suppose that the application dubbo-application deploys 3 instances (instance1, instance2, instance3), and provides 3 external interfaces (sayHello, echo, getVersion) with different timeout periods. Under the interface-level and application-level service discovery mechanisms, the data registered in the registry is completely different. As shown below:

• Data in the registry under the interface-level service discovery mechanism

"sayHello": [
  {"application":"dubbo-application","name":"instance1", "ip":"127.0.0.1", "metadata":{"timeout":1000}},
  {"application":"dubbo-application","name":"instance2", "ip":"127.0.0.2", "metadata":{"timeout":2000}},
  {"application":"dubbo-application","name":"instance3", "ip":"127.0.0.3", "metadata":{"timeout":3000}},
],
"echo": [
  {"application":"dubbo-application","name":"instance1", "ip":"127.0.0.1", "metadata":{"timeout":1000}},
  {"application":"dubbo-application","name":"instance2", "ip":"127.0.0.2", "metadata":{"timeout":2000}},
  {"application":"dubbo-application","name":"instance3", "ip":"127.0.0.3", "metadata":{"timeout":3000}},
],
"getVersion": [
  {"application":"dubbo-application","name":"instance1", "ip":"127.0.0.1", "metadata":{"timeout":1000}},
  {"application":"dubbo-application","name":"instance2", "ip":"127.0.0.2", "metadata":{"timeout":2000}},
  {"application":"dubbo-application","name":"instance3", "ip":"127.0.0.3", "metadata":{"timeout":3000}}
]

• Data in the registry under the application-level service discovery mechanism

"dubbo-application": [
  {"name":"instance1", "ip":"127.0.0.1", "metadata":{"timeout":1000}},
  {"name":"instance2", "ip":"127.0.0.2", "metadata":{"timeout":2000}},
  {"name":"instance3", "ip":"127.0.0.3", "metadata":{"timeout":3000}}
]

Through comparison, we can find that the application-level service discovery mechanism does reduce the amount of data in the registry a lot, and the original interface-level data is stored in the metadata center.

whole process of server exposure **

After the introduction of the application-level service discovery mechanism, the whole process of Dubbo 3.0 server exposure is very different from before. The core code of the whole process of exposing the server is in DubboBootstrap#doStart, as follows:

private void doStart() {
    // 1. 暴露Dubbo服务
    exportServices();
    // If register consumer instance or has exported services
    if (isRegisterConsumerInstance() || hasExportedServices()) {
        // 2. 暴露元数据服务
        exportMetadataService();
        // 3. 定时更新和上报元数据
        registerServiceInstance();
        ....
    }
    ......
}

Assuming that Zookeeper is used as an example to expose the services of the Triple protocol in the registration process, the sequence diagram of the whole process of server exposure is as follows:

We can see that the entire exposure process is quite complicated and can be divided into four parts:

• Expose the service of the injvm protocol
• Register service-discovery-registry agreement
• Expose the services of the Triple agreement and register the registry agreement
• Expose MetadataService service

The entire process of service exposure will be explained in detail from these four parts below.

1. Expose the service of the injvm protocol

The service of the injvm protocol is exposed locally. The main reason is that there are often both Service (exposed service) and Reference (service reference) in an application, and the service referenced by the Reference is the Service exposed on the application. In order to support this usage scenario, Dubbo provides the injvm protocol to expose the Service locally, and Reference can directly call the Service locally without going to the network.

Overall timing diagram

Since this part of the content is similar in the previous interface-level service discovery mechanism, the relevant core code will not be discussed here.

2. Register the service-discovery-registry agreement

The core purpose of registering the service-discovery-registry protocol is to register metadata related to the service. By default, metadata is stored in local memory and local files through InMemoryWritableMetadataService.

Overall timing diagram

The core code is in ServiceConfig#exportRemote, as follows:

• Entrance to register service-discovery-registry agreement

private URL exportRemote(URL url, List<URL> registryURLs) {
    if (CollectionUtils.isNotEmpty(registryURLs)) {
        // 如果是多个注册中心，通过循环对每个注册中心进行注册
        for (URL registryURL : registryURLs) {
            // 判断是否是service-discovery-registry协议
            // 将service-name-mapping参数的值设置为true
            if (SERVICE_REGISTRY_PROTOCOL.equals(registryURL.getProtocol())) {
                url = url.addParameterIfAbsent(SERVICE_NAME_MAPPING_KEY, "true");
            }
            ......
            // 注册service-discovery-registry协议复用服务暴露流程
            doExportUrl(registryURL.putAttribute(EXPORT_KEY, url), true);
        }
    ......
    return url;
}

• Wrap Metadata in invoker

The core code is in ServiceConfig#doExportUrl, as follows:

private void doExportUrl(URL url, boolean withMetaData) {
    Invoker<?> invoker = PROXY_FACTORY.getInvoker(ref, (Class) interfaceClass, url);
    // 此时的withMetaData的值为true
    // 将invoker包装成DelegateProviderMetaDataInvoker
    if (withMetaData) {
        invoker = new DelegateProviderMetaDataInvoker(invoker, this);
    }
    Exporter<?> exporter = PROTOCOL.export(invoker);
    exporters.add(exporter);
}

• Convert Invoker into Exporter through RegistryProtocol

The core code is in ProtocolListenerWrapper#export, as follows:

public <T> Exporter<T> export(Invoker<T> invoker) throws RpcException {
    // 此时的protocol为RegistryProtocol类型
    if (UrlUtils.isRegistry(invoker.getUrl())) {
        return protocol.export(invoker);
    }
    ......
}

• RegistryProtocol transforms Invoker into Exporter's core process

The core code is in RegistryProtocol#export, as follows:

public <T> Exporter<T> export(final Invoker<T> originInvoker) throws RpcException {
    URL registryUrl = getRegistryUrl(originInvoker);
    URL providerUrl = getProviderUrl(originInvoker);
    ......
    // 再次暴露Triple协议的服务
    final ExporterChangeableWrapper<T> exporter = doLocalExport(originInvoker, providerUrl);
    // registryUrl中包含service-discovery-registry协议
    // 通过该协议创建ServiceDiscoveryRegistry对象
    // 然后组合RegistryServiceListener监听器，
    // 最后包装成ListenerRegistryWrapper对象
    final Registry registry = getRegistry(registryUrl);
    final URL registeredProviderUrl = getUrlToRegistry(providerUrl, registryUrl);
    boolean register = providerUrl.getParameter(REGISTER_KEY, true);
    if (register) {
        // 注册service-discovery-registry协议
        // 触发RegistryServiceListener的onRegister事件
        register(registry, registeredProviderUrl);
    }
    ......
    // 触发RegistryServiceListener的onRegister事件
    notifyExport(exporter);
    return new DestroyableExporter<>(exporter);
}

• Exposing the service of the Triple agreement

The core code is in RegistryProtocol#doLocalExport, as follows:

private <T> ExporterChangeableWrapper<T> doLocalExport(final Invoker<T> originInvoker, URL providerUrl) {
    String key = getCacheKey(originInvoker);
    // 此时的protocol为Triple协议的代理类
    // 和暴露injvm协议的PROTOCOL相同
    return (ExporterChangeableWrapper<T>) bounds.computeIfAbsent(key, s -> {
        Invoker<?> invokerDelegate = new InvokerDelegate<>(originInvoker, providerUrl);
        return new ExporterChangeableWrapper<>((Exporter<T>) protocol.export(invokerDelegate), originInvoker);
    });
}

• Register service-discovery-registry agreement

The core code is in ServiceDiscoveryRegistry#register and ServiceDiscoveryRegistry#doRegister, as follows:

1、ServiceDiscoveryRegistry#register

public final void register(URL url) {
    // 只有服务端（Provider）才需要注册
    if (!shouldRegister(url)) {
        return;
    }
    // 注册service-discovery-registry协议
    doRegister(url);
}

2、ServiceDiscoveryRegistry#doRegister

public void doRegister(URL url) {
    url = addRegistryClusterKey(url);
    // 注册元数据
    if (writableMetadataService.exportURL(url)) {
        if (logger.isInfoEnabled()) {
            logger.info(format("The URL[%s] registered successfully.", url.toString()));
        }
    } else {
        if (logger.isWarnEnabled()) {
            logger.warn(format("The URL[%s] has been registered.", url.toString()));
        }
    }
}

• Register metadata

The core code is in InMemoryWritableMetadataService#exportURL, as follows:

public boolean exportURL(URL url) {
    // 如果是MetadataService，则不注册元数据
    if (MetadataService.class.getName().equals(url.getServiceInterface())) {
        this.metadataServiceURL = url;
        return true;
    }
    updateLock.readLock().lock();
    try {
        String[] clusters = getRegistryCluster(url).split(",");
        for (String cluster : clusters) {
            MetadataInfo metadataInfo = metadataInfos.computeIfAbsent(cluster, k -> new MetadataInfo(ApplicationModel.getName()));
            // 将Triple协议的服务中接口相关的数据生成ServiceInfo
            // 将ServiceInfo注册到MetadataInfo中
            metadataInfo.addService(new ServiceInfo(url));
        }
        metadataSemaphore.release();
        return addURL(exportedServiceURLs, url);
    } finally {
        updateLock.readLock().unlock();
    }
}

• Post onRegister event

The core code is in ListenerRegistryWrapper#register, as follows:

public void register(URL url) {
    try {
        // registry为ServiceDiscoveryRegistry对象
        // 此时已经调用完ServiceDiscoveryRegistry#registry方法
        registry.register(url);
    } finally {
        if (CollectionUtils.isNotEmpty(listeners) && !UrlUtils.isConsumer(url)) {
            RuntimeException exception = null;
            for (RegistryServiceListener listener : listeners) {
                if (listener != null) {
                    try {
                        // 注册完service-discovery-registry协议后发布onRegister事件
                        listener.onRegister(url, registry);
                    } catch (RuntimeException t) {
                        logger.error(t.getMessage(), t);
                        exception = t;
                    }
                }
            }
            if (exception != null) {
                throw exception;
            }
        }
    }
}

• Publish service registration events

The core code is in RegistryProtocol#notifyExport, as follows:

private <T> void notifyExport(ExporterChangeableWrapper<T> exporter) {
    List<RegistryProtocolListener> listeners = ExtensionLoader.getExtensionLoader(RegistryProtocolListener.class)
        .getActivateExtension(exporter.getOriginInvoker().getUrl(), "registry.protocol.listener");
    if (CollectionUtils.isNotEmpty(listeners)) {
        for (RegistryProtocolListener listener : listeners) {
            // 发布RegistryProtocolListener的onExport事件
            listener.onExport(this, exporter);
        }
    }
}

We can see that the core purpose of registering the service-discovery-registry protocol is to store information related to the service interface in memory. Considering both compatibility and smooth migration, the community adopts the method of reusing ServiceConfig's exposure process when implementing it.

3. Expose the Triple agreement service and register the registry agreement

Exposing the services of the Triple protocol and registering the registry protocol is the core process of Dubbo 3.0 service exposure, which is divided into two parts:

• Exposing the service of the Triple agreement
• Register the registry agreement

Since the process of exposing the Triple protocol service is the same as the process of exposing the Injvm protocol service, it will not be repeated. The process of registering the registry protocol only registers the information related to the application instance, which is the application-level service discovery mechanism mentioned earlier.

Overall timing diagram

• Convert Invoker into Exporter through InterfaceCompatibleRegistryProtocol

The core code is in ProtocolListenerWrapper#export, as follows:

public <T> Exporter<T> export(Invoker<T> invoker) throws RpcException {
    // 此时的protocol为InterfaceCompatibleRegistryProtocol类型（继承了RegistryProtocol）
    // 注意：在注册service-discovery-registry协议的时候protocol为RegistryProtocol类型
    if (UrlUtils.isRegistry(invoker.getUrl())) {
        return protocol.export(invoker);
    }
    ......
}

• RegistryProtocol transforms Invoker into Exporter's core process

The core code is in RegistryProtocol#export, as follows:

public <T> Exporter<T> export(final Invoker<T> originInvoker) throws RpcException {
    URL registryUrl = getRegistryUrl(originInvoker);
    URL providerUrl = getProviderUrl(originInvoker);
    ......
    // 再次暴露Triple协议的服务
    final ExporterChangeableWrapper<T> exporter = doLocalExport(originInvoker, providerUrl);
    // registryUrl中包含registry协议
    // 通过该协议创建ZookeeperRegistry对象
    // 然后组合RegistryServiceListener监听器，
    // 最后包装成ListenerRegistryWrapper对象
    // 注意：
    // 1. service-discovery-registry协议对应的是ServiceDiscoveryRegistry
    // 2. registry协议对应的是ZookeeperRegistry
    final Registry registry = getRegistry(registryUrl);
    final URL registeredProviderUrl = getUrlToRegistry(providerUrl, registryUrl);
    boolean register = providerUrl.getParameter(REGISTER_KEY, true);
    if (register) {
        // 注册registry协议
        // 触发RegistryServiceListener的onRegister事件
        register(registry, registeredProviderUrl);
    }
    ......
    // 发布RegistryProtocolListener的onExport事件
    notifyExport(exporter);
    return new DestroyableExporter<>(exporter);
}

• Register the registry agreement

The core code is in FailbackRegistry#register and ServiceDiscoveryRegistry#doRegister (ZookeeperRegistry inherits FailbackRegistry), as follows:

1、FailbackRegistry#register

public void register(URL url) {
    if (!acceptable(url)) {
        ......
        try {
            // 注册registry协议
            doRegister(url);
        } catch (Exception e) {
            ......
        }
    }
}

2、ServiceDiscoveryRegistry#doRegister

public void doRegister(URL url) {
    try {
        // 在zookeeper上注册Provider
        // 目录：/dubbo/xxxService/providers/***
        // 数据：dubbo://192.168.31.167:20800/xxxService?anyhost=true&
        //      application=application-name&async=false&deprecated=false&dubbo=2.0.2&
        //      dynamic=true&file.cache=false&generic=false&interface=xxxService&
        //      metadata-type=remote&methods=hello&pid=82470&release=&
        //      service-name-mapping=true&side=provider&timestamp=1629588251493
        zkClient.create(toUrlPath(url), url.getParameter(DYNAMIC_KEY, true));
    } catch (Throwable e) {
        throw new RpcException("Failed to register " + url + " to zookeeper " + getUrl() + ", cause: " + e.getMessage(), e);
    }
}

• Subscription address change

The core code is in FailbackRegistry#subscribe and ZookeeperRegistry#doSubscribe, as follows:

1、FailbackRegistry#subscribe

public void subscribe(URL url, NotifyListener listener) {
    ......
    try {
        // 调用ZookeeperRegistry#doSubscribe
        doSubscribe(url, listener);
    } catch (Exception e) {
    ......
}

2、ZookeeperRegistry#doSubscribe

public void doSubscribe(final URL url, final NotifyListener listener) {
    try {
        if (ANY_VALUE.equals(url.getServiceInterface())) {
            ......
        } else {
            ......
            for (String path : toCategoriesPath(url)) {
                ConcurrentMap<NotifyListener, ChildListener> listeners = zkListeners.computeIfAbsent(url, k -> new ConcurrentHashMap<>());
                ChildListener zkListener = listeners.computeIfAbsent(listener, k -> new RegistryChildListenerImpl(url, path, k, latch));
                if (zkListener instanceof RegistryChildListenerImpl) {
                    ((RegistryChildListenerImpl) zkListener).setLatch(latch);
                }
                // 创建临时节点用来存储configurators数据
                // 目录：/dubbo/xxxService/configurators
                // 数据：应用的配置信息，可以在dubbo-admin中进行修改，默认为空
                zkClient.create(path, false);
                // 添加监听器，用来监听configurators中的变化
                List<String> children = zkClient.addChildListener(path, zkListener);
                if (children != null) {
                    urls.addAll(toUrlsWithEmpty(url, path, children));
                }
            }
            ......
        }
    } catch (Throwable e) {
        ......
    }
}

• Establish the connection between the exposed Triple protocol service and Metadata

The core code is in ServiceConfig#exportUrl, MetadataUtils#publishServiceDefinition, InMemoryWritableMetadataService#publishServiceDefinition, RemoteMetadataServiceImpl#publishServiceDefinition and MetadataReport#storeProviderMetadata, as follows:

1、ServiceConfig#exportUrl

private void exportUrl(URL url, List<URL> registryURLs) {
    ......
    if (!SCOPE_NONE.equalsIgnoreCase(scope)) {
        ......
        if (!SCOPE_LOCAL.equalsIgnoreCase(scope)) {
            url = exportRemote(url, registryURLs);
            // 发布事件，更新服务接口相关的数据
            MetadataUtils.publishServiceDefinition(url);
        }
    }
    ......
}

2、MetadataUtils#publishServiceDefinition

public static void publishServiceDefinition(URL url) {
    // 将服务接口相关的数据存在到InMemoryWritableMetadataService中
    WritableMetadataService.getDefaultExtension().publishServiceDefinition(url);
    // 将服务接口相关的数据存在到远端的元数据中心
    if (REMOTE_METADATA_STORAGE_TYPE.equalsIgnoreCase(url.getParameter(METADATA_KEY))) {
        getRemoteMetadataService().publishServiceDefinition(url);
    }
}

3、InMemoryWritableMetadataService#publishServiceDefinition

public void publishServiceDefinition(URL url) {
    try {
        String interfaceName = url.getServiceInterface();
        if (StringUtils.isNotEmpty(interfaceName)
            && !ProtocolUtils.isGeneric(url.getParameter(GENERIC_KEY))) {
            Class interfaceClass = Class.forName(interfaceName);
            ServiceDefinition serviceDefinition = ServiceDefinitionBuilder.build(interfaceClass);
            Gson gson = new Gson();
            String data = gson.toJson(serviceDefinition);
            // 存储服务接口相关数据
            // 数据格式：
            // {
            //   "canonicalName": "xxxService",
            //   "codeSource": "file:/Users/xxxx",
            //   "methods": [{
            //       "name": "hello",
            //       "parameterTypes": ["java.lang.String"],
            //       "returnType": "java.lang.String",
            //       "annotations": []
            //   }],
            //   "types": [{
            //       "type": "java.lang.String"
            //    }],
            //  "annotations": []
            // } 
            serviceDefinitions.put(url.getServiceKey(), data);
            return;
        } else if (CONSUMER_SIDE.equalsIgnoreCase(url.getParameter(SIDE_KEY))) {
            ......
        }
        ......
    } catch (Throwable e) {
        ......
    }
}

4、RemoteMetadataServiceImpl#publishServiceDefinition

public void publishServiceDefinition(URL url) {
    checkRemoteConfigured();
    String side = url.getSide();
    if (PROVIDER_SIDE.equalsIgnoreCase(side)) {
        // 发布服务端（Provider）的服务接口信息到元数据中心
        publishProvider(url);
    } else {
        ......
    }
}
RemoteMetadataServiceImpl#publishProvider
private void publishProvider(URL providerUrl) throws RpcException {
    ......
    try {
        String interfaceName = providerUrl.getServiceInterface();
        if (StringUtils.isNotEmpty(interfaceName)) {
            ......
            for (Map.Entry<String, MetadataReport> entry : getMetadataReports().entrySet()) {
                // 获取MetadataReport服务，该服务用来访问元数据中心
                MetadataReport metadataReport = entry.getValue();
                // 将服务接口信息存储到元数据中心
                metadataReport.storeProviderMetadata(new MetadataIdentifier(providerUrl.getServiceInterface(),
                    providerUrl.getVersion(), providerUrl.getGroup(),
                    PROVIDER_SIDE, providerUrl.getApplication()), fullServiceDefinition);
            }
            return;
        }
        ......
    } catch (ClassNotFoundException e) {
        ......
    }
}

5、AbstractMetadataReport#storeProviderMetadata

public void storeProviderMetadata(MetadataIdentifier providerMetadataIdentifier, ServiceDefinition serviceDefinition){
    if (syncReport) {
        storeProviderMetadataTask(providerMetadataIdentifier, serviceDefinition);
    } else {
        // 异步存储到元数据中心
        reportCacheExecutor.execute(() -> storeProviderMetadataTask(providerMetadataIdentifier, serviceDefinition));
    }
}
private void storeProviderMetadataTask(MetadataIdentifier providerMetadataIdentifier, ServiceDefinition serviceDefinition) {
    try {
        ......
        allMetadataReports.put(providerMetadataIdentifier, serviceDefinition);
        failedReports.remove(providerMetadataIdentifier);
        Gson gson = new Gson();
        // data的数据格式：
        // {
        //   "parameters": {
        //       "side": "provider", 
        //       "interface": "xxxService",
        //       "metadata-type": "remote",
        //       "service-name-mapping": "true",
        //   },
        //   "canonicalName": "xxxService",
        //   "codeSource": "file:/Users/xxxx",
        //   "methods": [{
        //       "name": "hello",
        //       "parameterTypes": ["java.lang.String"],
        //       "returnType": "java.lang.String",
        //       "annotations": []
        //   }],
        //   "types": [{
        //       "type": "java.lang.String"
        //    }],
        //  "annotations": []
        // } 
        String data = gson.toJson(serviceDefinition);
        // 存储到元数据中心，实例中的元数据中心是ZookeeperMetadataReport
        // 目录：元数据中心Metadata-report的/dubbo/metadata/xxxService/provider/${application-name}节点下
        doStoreProviderMetadata(providerMetadataIdentifier, data);
        // 存储到本地文件
        // 路径：xxxService:::provider:${application-name} 
        saveProperties(providerMetadataIdentifier, data, true, !syncReport);
    } catch (Exception e) {
        ......
    }
}

• Establish the relationship between the Triple protocol service and the MetadataReport service

The core code is in ServiceConfig#exported, MetadataServiceNameMapping#map and ZookeeperMetadataReport#registerServiceAppMapping, as follows:

1、ServiceConfig#exported

protected void exported() {
    exported = true;
    List<URL> exportedURLs = this.getExportedUrls();
    exportedURLs.forEach(url -> {
        // 判断URL中是否标记有service-name-mapping的字段
        // 标记有该字段的服务是需要将暴露的服务与元数据中心关联起来
        // Consumer可以通过元数据中心的消息变更感知到Provider端元数据的变更
        if (url.getParameters().containsKey(SERVICE_NAME_MAPPING_KEY)) {
            ServiceNameMapping serviceNameMapping = ServiceNameMapping.getDefaultExtension();
            // 建立关系
            serviceNameMapping.map(url);
        }
    });
    onExported();
}

2、MetadataServiceNameMapping#map

public void map(URL url) {
    execute(() -> {
        String registryCluster = getRegistryCluster(url);
        // 获取MetadataReport，也就是元数据中心的访问路径
        MetadataReport metadataReport = MetadataReportInstance.getMetadataReport(registryCluster);
        ......
        int currentRetryTimes = 1;
        boolean success;
        String newConfigContent = getName();
        do {
            // 获取元数据中心中存储的应用的版本信息
            ConfigItem configItem = metadataReport.getConfigItem(serviceInterface, DEFAULT_MAPPING_GROUP);
            String oldConfigContent = configItem.getContent();
            if (StringUtils.isNotEmpty(oldConfigContent)) {
                boolean contains = StringUtils.isContains(oldConfigContent, getName());
                if (contains) {
                    break;
                }
                newConfigContent = oldConfigContent + COMMA_SEPARATOR + getName();
            }
            // 在元数据中心创建mapping节点，并将暴露的服务数据存到元数据中心，这里的元数据中心用zookeeper实现的
            // 目录：/dubbo/mapping/xxxService
            // 数据：configItem.content为${application-name}，configItem.ticket为版本好
            success = metadataReport.registerServiceAppMapping(serviceInterface, DEFAULT_MAPPING_GROUP, newConfigContent, configItem.getTicket());
        } while (!success && currentRetryTimes++ <= CAS_RETRY_TIMES);
    });
}

3、ZookeeperMetadataReport#registerServiceAppMapping

public boolean registerServiceAppMapping(String key, String group, String content, Object ticket) {
    try {
        if (ticket != null && !(ticket instanceof Stat)) {
            throw new IllegalArgumentException("zookeeper publishConfigCas requires stat type ticket");
        }
        String pathKey = buildPathKey(group, key);
        // 1. 创建/dubbo/mapping/xxxService目录，存储的数据为configItem
        // 2. 生成版本号
        zkClient.createOrUpdate(pathKey, content, false, ticket == null ? 0 : ((Stat) ticket).getVersion());
        return true;
    } catch (Exception e) {
        logger.warn("zookeeper publishConfigCas failed.", e);
        return false;
    }
}

At this point, the process of exposing the services of the Triple agreement and registering the registry agreement is over. The main purpose is to split the data (application instance data + service interface data) registered in the registry in the previous interface-level service discovery mechanism. The registration protocol part registers the application instance data to the registration center. After the Exporter has exposed it, the service interface data is registered to the metadata center by calling MetadataUtils#publishServiceDefinition.

4. Expose MetadataService service

MetadataService mainly provides an API that can obtain metadata on the Consumer side. The exposure process is a service exposure process that reuses the Triple protocol.

Overall timing diagram

• Expose the entrance of MetadataService

The core code is in DubboBootstrap#exportMetadataService, as follows:

private void exportMetadataService() {
    // 暴露MetadataServer
    metadataServiceExporter.export();
}

• Expose MetadataService

The core code is in ConfigurableMetadataServiceExporter#export, as follows:

public ConfigurableMetadataServiceExporter export() {
    if (!isExported()) {
        // 定义MetadataService的ServiceConfig
        ServiceConfig<MetadataService> serviceConfig = new ServiceConfig<>();
        serviceConfig.setApplication(getApplicationConfig());
        // 不会注册到注册中心
        serviceConfig.setRegistry(new RegistryConfig("N/A"));
        serviceConfig.setProtocol(generateMetadataProtocol());
        serviceConfig.setInterface(MetadataService.class);
        serviceConfig.setDelay(0);
        serviceConfig.setRef(metadataService);
        serviceConfig.setGroup(getApplicationConfig().getName());
        serviceConfig.setVersion(metadataService.version());
        serviceConfig.setMethods(generateMethodConfig());
        // 用暴露Triple协议服务的流程来暴露MetadataService
        // 采用的是Dubbo协议
        serviceConfig.export();
        this.serviceConfig = serviceConfig;
    }
    return this;
}

Since the process of exposing MetadataService is to reuse the process of exposing Triple protocol services mentioned earlier, there are a few differences in the whole process. These differences have been marked in the code above, so I won't repeat them.

• Register ServiceInstance instance

The purpose of registering ServiceInstance is to update the Metadata regularly. When there is an update, the version number will be updated through the MetadataReport to make the Consumer end aware.

The core code is in DubboBootstrap#registerServiceInstance and DubboBootstrap#doRegisterServiceInstance, as follows:

private void registerServiceInstance() {
    ....
    // 创建ServiceInstance
    // ServiceInstance中包含以下字段
    // 1. serviceName：${application-name}
    // 2. host: 192.168.31.167
    // 3. port: 2080
    // 4. metadata: 服务接口级相关的数据，比如：methods等数据
    // 同时，还会对ServiceInstance数据中的字段进行补充，分别调用下面4个ServiceInstanceCustomizer实例
    // 1）ServiceInstanceMetadataCustomizer
    // 2）MetadataServiceURLParamsMetadataCustomizer
    // 3）ProtocolPortsMetadataCustomizer
    // 4）ServiceInstanceHostPortCustomizer
    ServiceInstance serviceInstance = createServiceInstance(serviceName);
    boolean registered = true;
    try {
        // 注册ServiceInstance
        doRegisterServiceInstance(serviceInstance);
    } catch (Exception e) {
        registered = false;
        logger.error("Register instance error", e);
    }
    // 如果注册成功，定时更新Metadata，没10s更新一次
    if(registered){
        executorRepository.nextScheduledExecutor().scheduleAtFixedRate(() -> {
            ......
            try {
                // 刷新Metadata和ServiceInstance
                ServiceInstanceMetadataUtils.refreshMetadataAndInstance(serviceInstance);
            } catch (Exception e) {
                ......
            } finally {
                ......
            }
        }, 0, ConfigurationUtils.get(METADATA_PUBLISH_DELAY_KEY, DEFAULT_METADATA_PUBLISH_DELAY), TimeUnit.MILLISECONDS);
    }
}

DubboBootstrap#doRegisterServiceInstance

private void doRegisterServiceInstance(ServiceInstance serviceInstance) {
    if (serviceInstance.getPort() > 0) {
        // 发布Metadata数据到远端存储元数据中心
        // 调用RemoteMetadataServiceImpl#publishMetadata，
        // 内部会调用metadataReport#publishAppMetadata
        publishMetadataToRemote(serviceInstance);
        logger.info("Start registering instance address to registry.");
        getServiceDiscoveries().forEach(serviceDiscovery ->{
            ServiceInstance serviceInstanceForRegistry = new DefaultServiceInstance((DefaultServiceInstance) serviceInstance);
            calInstanceRevision(serviceDiscovery, serviceInstanceForRegistry);
            ......
            // 调用ZookeeperServiceDiscovery#doRegister注册serviceInstance实例
            // 将应用服务信息注册到注册中心中
            // 目录：/services/${application-name}/192.168.31.167:20800
            // 数据：serviceInstance序列化后的byte数组
            serviceDiscovery.register(serviceInstanceForRegistry);
        });
    }
}

Through the above analysis, we can easily know

• ServiceInstance contains Metadata
• Metadata is the metadata stored in InMemoryWritableMetadataService, which occupies local memory space
• InMemoryWritableMetadataService is used to update Metadata
• ServiceInstance is a data structure stored in the remote metadata registry
• RemoteMetadataServiceImpl will call metadataReport to update ServiceInstance data to the remote metadata registry

summary

Through the analysis of the entire process of Dubbo 3.0 server-side exposure, it can be seen that although the implementation of the application-level service discovery mechanism is much more complicated, Dubbo 3.0 is compatible with the 2.7.x version in order to allow users to migrate smoothly. In many places, the previous process is reused as much as possible.

Judging from the Benchmark data released by Dubbo 3.0 recently, the performance and resource utilization of Dubbo 3.0 have indeed improved a lot. Dubbo 3.0 still has a long way to go on the road to embracing cloud native. The community is sorting out and optimizing the core processes in Dubbo 3.0. Follow-up plans to support the deployment of multi-instance applications. I hope you are interested in witnessing the road to Dubbo cloud native. Students can actively participate in community contributions!

Copyright Notice: content of this article is contributed spontaneously by Alibaba Cloud real-name registered users. The copyright belongs to the original author. The Alibaba Cloud Developer Community does not own its copyright and does not assume corresponding legal responsibilities. For specific rules, please refer to the "Alibaba Cloud Developer Community User Service Agreement" and the "Alibaba Cloud Developer Community Intellectual Property Protection Guidelines". If you find suspected plagiarism in this community, fill in the infringement complaint form to report it. Once verified, the community will immediately delete the suspected infringing content.