聊聊PowerJob的ServerDeployContainerRequest

序

本文主要研究一下PowerJob的ServerDeployContainerRequest

ServerDeployContainerRequest

tech/powerjob/common/request/ServerDeployContainerRequest.java

@Data
@NoArgsConstructor
@AllArgsConstructor
public class ServerDeployContainerRequest implements PowerSerializable {

    /**
     * 容器ID
     */
    private Long containerId;
    /**
     * 容器名称
     */
    private String containerName;
    /**
     * 文件名（MD5值），用于做版本校验和文件下载
     */
    private String version;
    /**
     * 下载地址
     */
    private String downloadURL;
}

ServerDeployContainerRequest定义了containerId、containerName、version、downloadURL属性

onReceiveServerDeployContainerRequest

tech/powerjob/worker/actors/WorkerActor.java

    @Handler(path = WORKER_HANDLER_DEPLOY_CONTAINER)
    public void onReceiveServerDeployContainerRequest(ServerDeployContainerRequest request) {
        OmsContainerFactory.deployContainer(request);
    }

WorkerActor的onReceiveServerDeployContainerRequest用于处理ServerDeployContainerRequest，它委托给了OmsContainerFactory.deployContainer

deployContainer

tech/powerjob/worker/container/OmsContainerFactory.java

    public static synchronized void deployContainer(ServerDeployContainerRequest request) {

        Long containerId = request.getContainerId();
        String containerName = request.getContainerName();
        String version = request.getVersion();

        log.info("[OmsContainer-{}] start to deploy container(name={},version={},downloadUrl={})", containerId, containerName, version, request.getDownloadURL());

        OmsContainer oldContainer = CARGO.get(containerId);
        if (oldContainer != null && version.equals(oldContainer.getVersion())) {
            log.info("[OmsContainer-{}] version={} already deployed, so skip this deploy task.", containerId, version);
            return;
        }

        String filePath = CONTAINER_DIR + containerId + "/" + version + ".jar";
        // 下载Container到本地
        File jarFile = new File(filePath);

        try {
            if (!jarFile.exists()) {
                FileUtils.forceMkdirParent(jarFile);
                FileUtils.copyURLToFile(new URL(request.getDownloadURL()), jarFile, 5000, 300000);
                log.info("[OmsContainer-{}] download jar successfully, path={}", containerId, jarFile.getPath());
            }

            // 创建新容器
            OmsContainer newContainer = new OmsJarContainer(containerId, containerName, version, jarFile);
            newContainer.init();

            // 替换容器
            CARGO.put(containerId, newContainer);
            log.info("[OmsContainer-{}] deployed new version:{} successfully!", containerId, version);

            if (oldContainer != null) {
                // 销毁旧容器
                oldContainer.destroy();
            }

        } catch (Exception e) {
            log.error("[OmsContainer-{}] deployContainer(name={},version={}) failed.", containerId, containerName, version, e);
            // 如果部署失败，则删除该 jar（本次失败可能是下载jar出错导致，不删除会导致这个版本永久无法重新部署）
            CommonUtils.executeIgnoreException(() -> FileUtils.forceDelete(jarFile));
        }
    }

deployContainer方法先找到旧的OmsContainer，然后判断version是否一样，一样就不用重新部署，否则先从本地查找jar包，找不到则根据downloadURL去下载，然后创建OmsJarContainer，执行其init方法，若存在旧的OmsContainer则执行其destroy方法

OmsContainer

tech/powerjob/worker/container/OmsContainer.java

public interface OmsContainer extends LifeCycle {

    /**
     * 获取处理器
     * @param className 全限定类名
     * @return 处理器（可以是 MR、BD等处理器）
     */
    BasicProcessor getProcessor(String className);

    /**
     * 获取容器的类加载器
     * @return 类加载器
     */
    OhMyClassLoader getContainerClassLoader();

    Long getContainerId();
    Long getDeployedTime();
    String getName();
    String getVersion();

    /**
     * 尝试释放容器资源
     */
    void tryRelease();
}

OmsContainer接口定义了getProcessor、getContainerClassLoader、getContainerId、getDeployedTime、getName、getVersion、tryRelease方法

OmsJarContainer

tech/powerjob/worker/container/OmsJarContainer.java

@Slf4j
public class OmsJarContainer implements OmsContainer {

    private final Long containerId;
    private final String name;
    private final String version;
    private final File localJarFile;
    private final Long deployedTime;

    // 引用计数器
    private final AtomicInteger referenceCount = new AtomicInteger(0);

    private OhMyClassLoader containerClassLoader;
    private ClassPathXmlApplicationContext container;

    private final Map<String, BasicProcessor> processorCache = Maps.newConcurrentMap();

    public OmsJarContainer(Long containerId, String name, String version, File localJarFile) {
        this.containerId = containerId;
        this.name = name;
        this.version = version;
        this.localJarFile = localJarFile;
        this.deployedTime = System.currentTimeMillis();
    }

    //......
}    

OmsJarContainer实现了OmsContainer接口

getProcessor

    public BasicProcessor getProcessor(String className) {

        BasicProcessor basicProcessor = processorCache.computeIfAbsent(className, ignore -> {
            Class<?> targetClass;
            try {
                targetClass = containerClassLoader.loadClass(className);
            } catch (ClassNotFoundException cnf) {
                log.error("[OmsJarContainer-{}] can't find class: {} in container.", containerId, className);
                return null;
            }

            // 先尝试从 Spring IOC 容器加载
            try {
                return (BasicProcessor) container.getBean(targetClass);
            } catch (BeansException be) {
                log.warn("[OmsJarContainer-{}] load instance from spring container failed, try to build instance directly.", containerId);
            } catch (ClassCastException cce) {
                log.error("[OmsJarContainer-{}] {} should implements the Processor interface!", containerId, className);
                return null;
            } catch (Exception e) {
                log.error("[OmsJarContainer-{}] get bean failed for {}.", containerId, className, e);
                return null;
            }

            // 直接实例化
            try {
                Object obj = targetClass.getDeclaredConstructor().newInstance();
                return (BasicProcessor) obj;
            } catch (Exception e) {
                log.error("[OmsJarContainer-{}] load {} failed", containerId, className, e);
            }
            return null;
        });

        if (basicProcessor != null) {
            // 引用计数 + 1
            referenceCount.getAndIncrement();
        }
        return basicProcessor;
    }

getProcessor方法会先通过containerClassLoader.loadClass去加载对应的processor类，加载不到则直接返回，之后根据targetClass去spring容器查找，若查找不到则直接通过targetClass.getDeclaredConstructor().newInstance()尝试实例化

init

    public void init() throws Exception {

        log.info("[OmsJarContainer-{}] start to init container(name={},jarPath={})", containerId, name, localJarFile.getPath());

        URL jarURL = localJarFile.toURI().toURL();

        // 创建类加载器（父类加载为 Worker 的类加载）
        this.containerClassLoader = new OhMyClassLoader(new URL[]{jarURL}, this.getClass().getClassLoader());

        // 解析 Properties
        Properties properties = new Properties();
        try (InputStream propertiesURLStream = containerClassLoader.getResourceAsStream(ContainerConstant.CONTAINER_PROPERTIES_FILE_NAME)) {

            if (propertiesURLStream == null) {
                log.error("[OmsJarContainer-{}] can't find {} in jar {}.", containerId, ContainerConstant.CONTAINER_PROPERTIES_FILE_NAME, localJarFile.getPath());
                throw new PowerJobException("invalid jar file because of no " + ContainerConstant.CONTAINER_PROPERTIES_FILE_NAME);
            }

            properties.load(propertiesURLStream);
            log.info("[OmsJarContainer-{}] load container properties successfully: {}", containerId, properties);
        }
        String packageName = properties.getProperty(ContainerConstant.CONTAINER_PACKAGE_NAME_KEY);
        if (StringUtils.isEmpty(packageName)) {
            log.error("[OmsJarContainer-{}] get package name failed, developer should't modify the properties file!", containerId);
            throw new PowerJobException("invalid jar file");
        }

        // 加载用户类
        containerClassLoader.load(packageName);

        // 创建 Spring IOC 容器（Spring配置文件需要填相对路径）
        // 需要切换线程上下文类加载器以加载 JDBC 类驱动（SPI）
        ClassLoader oldCL = Thread.currentThread().getContextClassLoader();
        Thread.currentThread().setContextClassLoader(containerClassLoader);
        try {
            this.container = new ClassPathXmlApplicationContext(new String[]{ContainerConstant.SPRING_CONTEXT_FILE_NAME}, false);
            this.container.setClassLoader(containerClassLoader);
            this.container.refresh();
        }finally {
            Thread.currentThread().setContextClassLoader(oldCL);
        }

        log.info("[OmsJarContainer-{}] init container(name={},jarPath={}) successfully", containerId, name, localJarFile.getPath());
    }

init方法根据jar包地址创建OhMyClassLoader，然后先解析oms-worker-container.properties，执行properties.load(propertiesURLStream)，接着获取配置的packageName，执行containerClassLoader.load(packageName)加载类，然后根据oms-worker-container-spring-context.xml创建spring的ClassPathXmlApplicationContext，设置其classLoader，执行其refresh方法

destroy

    public void destroy() throws Exception {

        // 没有其余引用时，才允许执行 destroy
        if (referenceCount.get() <= 0) {
            try {
                if (localJarFile.exists()) {
                    FileUtils.forceDelete(localJarFile);
                }
            }catch (Exception e) {
                log.warn("[OmsJarContainer-{}] delete jarFile({}) failed.", containerId, localJarFile.getPath(), e);
            }
            try {
                processorCache.clear();
                container.close();
                containerClassLoader.close();
                log.info("[OmsJarContainer-{}] container destroyed successfully", containerId);
            }catch (Exception e) {
                log.error("[OmsJarContainer-{}] container destroyed failed", containerId, e);
            }
            return;
        }

        log.warn("[OmsJarContainer-{}] container's reference count is {}, won't destroy now!", containerId, referenceCount.get());
    }

destroy方法在referenceCount小于等于0时会先删除localJarFile，然后执行processorCache.clear()、ClassPathXmlApplicationContext的close、OhMyClassLoader的close

JarContainerProcessorFactory

tech/powerjob/worker/processor/impl/JarContainerProcessorFactory.java

@Slf4j
public class JarContainerProcessorFactory implements ProcessorFactory {

    private final WorkerRuntime workerRuntime;

    public JarContainerProcessorFactory(WorkerRuntime workerRuntime) {
        this.workerRuntime = workerRuntime;
    }

    @Override
    public Set<String> supportTypes() {
        return Sets.newHashSet(ProcessorType.EXTERNAL.name());
    }

    @Override
    public ProcessorBean build(ProcessorDefinition processorDefinition) {

        String processorInfo = processorDefinition.getProcessorInfo();
        String[] split = processorInfo.split("#");
        String containerName = split[0];
        String className = split[1];

        log.info("[ProcessorFactory] try to load processor({}) in container({})", className, containerName);

        OmsContainer omsContainer = OmsContainerFactory.fetchContainer(Long.valueOf(containerName), workerRuntime);
        if (omsContainer != null) {
            return new ProcessorBean()
                    .setProcessor(omsContainer.getProcessor(className))
                    .setClassLoader(omsContainer.getContainerClassLoader());
        } else {
            log.warn("[ProcessorFactory] load container failed. processor info : {}", processorInfo);
        }
        return null;
    }
}

JarContainerProcessorFactory的build方法它根据#来解析出containerId及className，然后通过OmsContainerFactory.fetchContainer去查找容器，然后通过omsContainer.getProcessor(className)获取对应的processor；JarContainerProcessorFactory的supportTypes为EXTERNAL(外部处理器（动态加载）)

小结

WorkerActor的onReceiveServerDeployContainerRequest用于处理ServerDeployContainerRequest，它委托给了OmsContainerFactory.deployContainer；deployContainer方法先找到旧的OmsContainer，然后判断version是否一样，一样就不用重新部署，否则先从本地查找jar包，找不到则根据downloadURL去下载，然后创建OmsJarContainer，执行其init方法，若存在旧的OmsContainer则执行其destroy方法；init方法根据jar包地址创建OhMyClassLoader，创建ClassPathXmlApplicationContext，设置其classLoader，执行其refresh方法；destroy方法在referenceCount小于等于0时会先删除localJarFile，然后执行processorCache.clear()、ClassPathXmlApplicationContext的close、OhMyClassLoader的close。