trino -- 服务发现解析

trino的服务发现是通过airlift来实现的。
DiscoveryClientConfig中discovery.uri指定了向哪个服务发送自己的信息。Coordinator配置的是localhost，也就是自己，Worker节点配置的是Coordinator的地址。
此外还指定了DiscoveryAnnouncementClient实现类是HttpDiscoveryAnnouncementClient。
Server启动的时候，会调用Announcer的start方法，开始了服务注册。

injector.getInstance(Announcer.class).start();

注册

start中，会调用HttpDiscoveryAnnouncementClient的announce方法进行注册，可以看到不管成功和失败，定时任务都会持续的注册。

ListenableFuture<Duration> future = announcementClient.announce(getServiceAnnouncements());

Futures.addCallback(future, new FutureCallback<Duration>()
{
    @Override
    public void onSuccess(Duration expectedDelay)
    {
        errorBackOff.success();

        // wait 80% of the suggested delay
        expectedDelay = new Duration(expectedDelay.toMillis() * 0.8, MILLISECONDS);
        scheduleNextAnnouncement(expectedDelay);
    }

    @Override
    public void onFailure(Throwable t)
    {
        Duration duration = errorBackOff.failed(t);
        scheduleNextAnnouncement(duration);
    }
}, executor);

HttpDiscoveryAnnouncementClient的announce方法会封装Announcement对象，这里包含了要上报的信息，比如节点id之类的，比如当前节点id是ffffffff-ffff-ffff-ffff-ffffffffffff，发给localhost，那http请求就是http://localhost:8080/v1/announcement/ffffffff-ffff-ffff-ffff...。

// 其他代码略
Announcement announcement = new Announcement(nodeInfo.getEnvironment(), nodeInfo.getNodeId(), nodeInfo.getPool(), nodeInfo.getLocation(), services);
        Request request = preparePut()
                .setUri(createAnnouncementLocation(uri, nodeInfo.getNodeId()))
                .setHeader("User-Agent", nodeInfo.getNodeId())
                .setHeader("Content-Type", MEDIA_TYPE_JSON.toString())
                .setBodyGenerator(jsonBodyGenerator(announcementCodec, announcement))
                .build();

处理注册请求

ServerSecurityModule中配置了DynamicAnnouncementResource，用来处理注册请求。
这里可以看到把节点id和节点信息保存到dynamicStore中，这个dynamicStore是ReplicatedDynamicStore。

@PUT
@Consumes({"application/json"})
public Response put(@PathParam("node_id") Id<Node> nodeId, @Context UriInfo uriInfo, DynamicAnnouncement announcement) {
    if (!this.nodeInfo.getEnvironment().equals(announcement.getEnvironment())) {
        return Response.status(Status.BAD_REQUEST).entity(String.format("Environment mismatch. Expected: %s, Provided: %s", this.nodeInfo.getEnvironment(), announcement.getEnvironment())).build();
    } else {
        String location = (String)MoreObjects.firstNonNull(announcement.getLocation(), "/somewhere/" + nodeId.toString());
        DynamicAnnouncement announcementWithLocation = DynamicAnnouncement.copyOf(announcement).setLocation(location).build();
        this.dynamicStore.put(nodeId, announcementWithLocation);
        return Response.status(Status.ACCEPTED).build();
    }
}

ReplicatedDynamicStore会把节点信息保存DistributedStore中并设置了过期时间。
DistributedStore会同时把数据保存RemoteStore和LocalStore中。他有一个定时任务，会把过期的信息从LocalStore中删除掉。

public void put(Id<Node> nodeId, DynamicAnnouncement announcement) {
    List<Service> services = FluentIterable.from(announcement.getServiceAnnouncements()).transform(DynamicServiceAnnouncement.toServiceWith(nodeId, announcement.getLocation(), announcement.getPool())).toList();
    byte[] key = nodeId.getBytes();
    byte[] value = this.codec.toJsonBytes(services);
    this.store.put(key, value, this.maxAge);
}

LocalStore保存会判断原先是否有数据，如果有数据且和新数据不一样，就会把旧的数据替换掉。

@Override
public void put(Entry entry)
{
    ByteBuffer key = ByteBuffer.wrap(entry.getKey());

    boolean done = false;
    while (!done) {
        Entry old = map.putIfAbsent(key, entry);

        done = true;
        if (old != null) {
            entry = resolver.resolve(old, entry);

            if (entry != old) {
                done = map.replace(key, old, entry);
            }
        }
    }
}

发现

ServerMainModule中指定了InternalNodeManager的实现类是DiscoveryNodeManager。
DiscoveryNodeManager初始化的时候，会调用refreshNodesInternal方法。
这个类启动的时候，会调用pollWorkers方法，并启动了定时任务每5秒去调用。pollWorkers方法也会调用refreshNodesInternal方法。

@PostConstruct
public void startPollingNodeStates()
{
    nodeStateUpdateExecutor.scheduleWithFixedDelay(() -> {
        try {
            pollWorkers();
        }
        catch (Exception e) {
            log.error(e, "Error polling state of nodes");
        }
    }, 5, 5, TimeUnit.SECONDS);
    pollWorkers();
}

refreshNodesInternal主要是拉取节点的信息并缓存到本地。
下面代码片段就是拉取信息的地方。
这个serviceSelector是MergingServiceSelector。

Set<ServiceDescriptor> services = serviceSelector.selectAllServices().stream()
    .filter(service -> !failed.contains(service))
    .collect(toImmutableSet());

在selectAllServices方法中，会通过selector获取节点信息，这个selector是CachingServiceSelector。

@Override
public List<ServiceDescriptor> selectAllServices()
{
    return merge(announcer.getServiceAnnouncements(), selector.selectAllServices());
}

CachingServiceSelector只是从本地缓存把值返回来，那这个本地缓存哪里来的呢？

@Override
public List<ServiceDescriptor> selectAllServices()
{
    ServiceDescriptors serviceDescriptors = this.serviceDescriptors.get();
    if (serviceDescriptors == null) {
        return ImmutableList.of();
    }
    return serviceDescriptors.getServiceDescriptors();
}

CachingServiceSelector初始化的时候，会调用start方法。这里会调用refresh方法。refresh会会调用http请求，不管请求成功还是失败，还会10s后继续调用refresh方法，所以CachingServiceSelector的缓存10s还会继续更新。

@PostConstruct
public void start()
{
    if (started.compareAndSet(false, true)) {
        checkState(!executor.isShutdown(), "CachingServiceSelector has been destroyed");

        // if discovery is available, get the initial set of servers before starting
        try {
            refresh().get(1, TimeUnit.SECONDS);
        }
        catch (Exception ignored) {
        }
    }
}

这个lookupClient是HttpDiscoveryLookupClient，最终调用http://localhost:8080/v1/service/trino/general获取节点信息。

ServiceDescriptors oldDescriptors = this.serviceDescriptors.get();

ListenableFuture<ServiceDescriptors> future;
if (oldDescriptors == null) {
    future = lookupClient.getServices(type, pool);
}
else {
    future = lookupClient.refreshServices(oldDescriptors);
}

处理发现请求

ServiceResource用来接收并处理发现请求。这个dynamicStore实际上就是上面处理注册请求的ReplicatedDynamicStore，也就是说注册的信息从这里返回。

@GET
@Path("{type}/{pool}")
@Produces(MediaType.APPLICATION_JSON)
public Services getServices(@PathParam("type") String type, @PathParam("pool") String pool)
{
    return new Services(node.getEnvironment(), union(dynamicStore.get(type, pool), staticStore.get(type, pool)));
}

ReplicatedDynamicStore中，查询数据是有缓存的，默认缓存1s。

@Override
public Set<Service> get(String type, String pool)
{
    return ImmutableSet.copyOf(filter(getAll(), and(matchesType(type), matchesPool(pool))));
}
@Override
public Set<Service> getAll()
{
    return servicesSupplier.get();
}
// 构造方法里servicesSupplier的定义
servicesSupplier = cachingSupplier(servicesSupplier(), config.getStoreCacheTtl());

private Supplier<Set<Service>> servicesSupplier()
{
    return new Supplier<Set<Service>>()
    {
        @Override
        public Set<Service> get()
        {
            ImmutableSet.Builder<Service> builder = ImmutableSet.builder();
            for (Entry entry : store.getAll()) {
                builder.addAll(codec.fromJson(entry.getValue()));
            }
            return builder.build();
        }
    };
}

上面的store是DistributedStore，DistributedStore的localStore就是InMemoryStore，所以取的数据就是把InMemoryStore的缓存map信息返回，这个map就是缓存注册的节点信息。

public Iterable<Entry> getAll()
{
    return Iterables.filter(localStore.getAll(), and(not(expired()), not(tombstone())));
}

@Override
public Iterable<Entry> getAll()
{
    return map.values();
}