WePack —— Facilitate Enterprise's Progressive DevOps Transformation

This article is a sharing made by Yu Dian, senior product manager of CODING, at the Tencent Cloud CIF Engineering Efficiency Summit. At the end of the article, you can go to the official website of the summit to watch the replay and download the PPT.

Hello everyone, I am Yu Dian, Senior Product Manager of CODING. DevOps should not be an unfamiliar concept to everyone. It brings faster delivery speed, response to changes and better team collaboration methods for R&D teams, helping companies to deliver and deploy software within days to weeks. In the DevOps process, you may be more likely to pay attention to the performance upgrades brought about by tools such as code, pipeline, testing, and release. So what are we building, what is the source of the test application, and what is released? ——The answers to these three questions are all "product packages."

Product management in software development is like warehouse management in traditional manufacturing, which connects the two stages of product production and release. A few days ago, I briefly counted the product data of the CODING R&D team. In one day, the entire team pushed more than 6,000 products and generated more than 10,000 product pull operations. The products are not only large in quantity, but also have different attributes due to different development languages and different facing environments, which require separate management. The product's push-pull stability, safety and information accuracy determine the production quality of the entire product. So around the various complex problems faced by enterprise-level product management, I will introduce to you today the design concept and implementation practice of CODING, an enterprise-level product management service independently developed by 161b31e297d527

Speaking of pain points, everyone can join me to evaluate the pain index of your team. The first is the category of users with the highest pain index: we have met many users before, because the entire R&D team has different technology stacks such as Java, C++, and front-end, and the types of products produced are also different. Many users use traditional Ftp or SVN warehouses to manage software compression packages, and build a Nexus to manage open source third-party Maven and Npm dependencies. With the popularity of container images, the R&D team has gradually produced some Docker images that need to be managed, so users set up a set of Harbor to manage the images.

Each of these systems has its own set of management system and account system, which brings higher management costs to operation and maintenance personnel. The product management has not yet been opened up, and it is even more difficult to get through with other DevOps modules. Therefore, centralized management of products is the first step for enterprise users to establish a DevOps product management system.

Many users are aware of this problem, so they start to build themselves based on open source product libraries such as Nexus, or purchase a unified product management platform such as JFrog to manage products. At this time, centralized management has brought new problems: Although these tools provide unified account and authority management functions, so that we can manage all team products in a unified manner, as the scale of enterprise products expands, they are centralized. How to divide the management authority of the products has become a problem. example, once a new project is born, or a temporary project or outsourcing project appears at this time, the product administrator needs to reconfigure the various permissions of these new projects for the user group of the entire enterprise; product management without project isolation will also occur Security issues will bring a great burden to the operation and maintenance team, making product management a bottleneck in the acceleration of the DevOps process.

After we solved the single-point problem of the previous two points of product management, a new problem came: Because of the linking attributes of product management, it is necessary to integrate information and function calls with upstream and downstream tools. Many companies choose different tools in each link of DevOps, and then several integrations and dockings will be required. Many companies choose to buy one-stop products such as foreign Azure, Gitlab, or domestic CODING DevOps to solve multiple integration problems, and finally get through the DevOps tool chain, but new problems arise.

When we face many privatized customers, we have mentioned the demand for product security scanning function. The concept of DevSecOps has emerged, whether it is due to national security regulations or the company's own information security requirements.

Some customers have purchased powerful security tools such as Black Duck and Checkmarx, which can scan for various vulnerabilities and provide multi-dimensional statistical reports, but this report only stays in the security tool system, and can be displayed at most in Jenkins plug-ins middle. How to split these vulnerabilities to the corresponding project team to solve, find a person in charge for this report, and track the resolution of the vulnerabilities in the report? these tools actually do not provide a solution. This is security tool and the DevOps tool chain. At the same time, because of this split, it is difficult for us to establish an automated control process to limit the right shift of security issues.

Another key issue is that the vulnerabilities scanned by security tools such . We cannot avoid the status quo that most R&D teams have limited security capabilities. Perhaps a team has 100 R&D and only one security officer, and he cannot take into account the security issues introduced by the entire team. Moreover, the vulnerability information and solutions provided by security tools are still relatively professional. It may be difficult for an ordinary developer to understand the repair suggestions provided by security software. He repaired according to the suggestions. In fact, there is no more professional person to verify the results. English information also increases the difficulty of this process, so fixing security vulnerabilities is much harder than finding them.

Focusing on these progressive pain points, we hope to provide users with a complete set of product management solutions. Therefore, the enterprise-level product management service-"WePack" . Next, I will introduce the design concept of WePack. .

When designing WePack, we first considered the product management-the problem of centralized management . We hope that WePack can not only unify the product types, but also manage the internal self-developed first-party construction package, the second-party platform basic product package, and the third-party open source components from the external network to ensure self-developed products. The life cycle information of R&D, testing, and release is recorded, and the source and use information of third-party products can be traced in the WePack system to establish the company's only trusted source of products.

At the same time, due to the requirements of enterprise product management for fine-grained permissions and project resource isolation, WePack continues to use CODING's user-verified permission management system. Ordinary product management tools, such as JFrog Artifactory, Nexus user management system. Several project groups under the enterprise share all product resources, and products are only isolated through warehouses. In the face of expanding project management needs, users need to increase the number of nodes to achieve project authority isolation. Recently, these tools have also realized the problems of multi-project management and have launched Project-based product capabilities, but the number of projects is limited by the payment level. WePack inherits the management level of the CODING project and provides users with the management level of the namespace. The namespace can also be understood as a project or R&D team, which can be used to manage different user groups and configure different product operations for these users. Permissions. Users in the system can join or exit the namespace at any time, and there is no limit to the number of namespaces, so enterprises do not need to worry about the management costs of temporary projects and outsourced projects.

WePack's current permission granularity is accurate to the warehouse level . Enterprise administrators can set the visible range of the warehouse and user groups to achieve fine-grained control over the product operation permissions of the specified warehouse, and meet the complex permissions management requirements of the enterprise. We are also planning to further refine this permission granularity to the product level.

After introducing the basic design concept of WePack, some friends who are familiar with coding may have a question. Whether it is a public cloud or a private deployment version of CODING DevOps, there is already a product management module. Why do you need to design a privatization separately? What about the product management single product system in the scene? We mainly have the following three considerations.

First of all, CODING product management can help users realize basic product management functions, connect upstream and downstream CI/CD, gather information, and open up the entire DevOps process. For users who are isolated from multiple environments such as software development, testing, and production, he can choose to deploy multiple sets of Wepack in multiple environments, and connect each environment through the product synchronization function. Or products need to be distributed to users in multiple regions. It is obviously inappropriate to deploy a set of coding in each environment or region. WePack can help them realize centralized management of products in various environments and regions.

On the other hand, if many users have self-built DevOps workflows, WePack can also provide good open capabilities to help users easily connect product management modules and self-built tools to achieve information aggregation.

Finally, the product is actually the same as the code. It is the core resource of the enterprise. Whether it is a public cloud or a private cloud, such a system may be required to store product resources separately in a relatively isolated environment.

Next, I will introduce the functional design of WePack in detail. The first is some basic functions of : WePack has basically covered mainstream product types, and can realize unified management of multiple types of products. And unlike most product management tools on the market that only provide product file structure information, WePack also displays the product’s own description, label, size, Readme and other metadata information. When the product is pushed to WePack, the product can be displayed. information. In the version management of the product, the user can set whether the product version can be overwritten through a flexible version management strategy. In support of the management function of third-party dependent packages, WePack provides a product agency function, which can proxy public products on the external network into the system, and cooperate with the product scanning function to ensure the safety, operability and traceability of open source components.

For the above-mentioned multi-environment and multi-regional management requirements, we provide product synchronization functions to realize the cross-environment and cross-regional circulation of products. The product synchronization function supports pushing products in the system to a remote warehouse, or pulling products from a remote warehouse to a local warehouse. Detailed practical cases will be explained later, which can also deepen the understanding of this function.

WePack supports flexible docking of various object storage products, and also supports users to clean up old products to free up storage space. It is worth mentioning that we have supported the non-stop server physical deletion of the Docker version.

The last basic function is product scanning. We have chosen to cooperate with Tencent's security professional team in terms of scanning capabilities. They provide support for different security capabilities. This makes our security capabilities highly accurate and professional, and meets strong demands for security. The product management needs of the enterprise.

This is the basic function WePack, and then I will introduce WePack on DevOps workflow of some applications . In addition to metadata, WePack also provides the function of product attributes. Any information such as code, Commit, build environment information, test pass information, quality inspection information, etc. can be recorded in the product. At the same time, the product push plugin of CODING can not only help users push products to the WePack product warehouse through CODING CI, but also automatically write the information in the build environment into the product management system. At the same time, we also provide a Jenkins plugin to support this function. .

So when it comes to the quality control process, whether it is the product management in coding or the WePack system, we have opened the product scanning module for vulnerability scanning of open source components in the current environment. Our work-in-process scanning function has added the ability to control the process, which can prohibit the download of problematic products when the scanning results come out, so that only safe products that pass the inspection can flow into the next link. At this time, if the development finds that the dependent component fails to be pulled, it can be traced back to the cause of the failure to solve the problematic product. Thanks to the process control of the discovery of loopholes and the flow of products, we can effectively manage and control security issues to the right.

In terms of product quality and safety responsibilities, we have avoided making the product scanning function independent of the DevOps process, but instead belongs to the management level of the entire WePack team/project. We can configure a scanning scheme for a project group in a namespace, or a product warehouse, or configure a scanning scheme based on some filtering rules, so that the person in charge of the product can only focus on the product vulnerability results within his authority, and who can do it Who will solve the problem.

So in terms of our functional design, help the R&D team to fix security vulnerabilities better and more conveniently? First of all, in the vulnerability information, users can locate the dependent components to which the vulnerability belongs. Through the dependency analysis function we provide, users can find out which production products the problematic component is dependent on, so as to locate and fix the vulnerability entry. At the same time, in order to avoid problems such as inaccuracy of open source vulnerabilities, lack of Chinese support for commercial users, and insufficient transparency of information, we have cooperated with Tencent Security and the Joint Laboratory. Tencent Security’s professional operation team has conducted security research based on a common security vulnerability library. Will contribute their research results to open source vulnerability libraries such as NVD. According to their research results, Tencent Security has established a self-developed software open source vulnerability signature database, which greatly reduces the false alarm rate of vulnerabilities, while providing Chinese vulnerability information, vulnerability component repair version information, and so on.

After our product scanning engine analyzes the product dependencies, it will access the service of the vulnerability library, output the product scan report, and then write it into the WePack system. After optimizing the vulnerability information, we hope to prevent users from paying attention to too many unimportant vulnerabilities at one time, so we define the priority of vulnerability repair. In addition to the security level provided by CVSS, Tencent Security also combines the dynamic security level of the vulnerability, that is, whether there is a public vulnerability exploiting POC disclosure, to define whether the vulnerability is now prioritized to recommend users to fix it. In this way, users can prioritize and fix vulnerabilities that are truly threatening to the R&D environment. The above are the basic functions of WePack.

Next, I will use two cases to demonstrate the implementation of WePack. The first is the landing practice case of financial industry The customer has strong control regulations and hopes that his R&D environment cannot access the external network, but also hopes to be able to proxy some open source components of the external network, so we have helped users establish a network isolation zone and deploy WePack in it. Open source components can be proxied from here to the WePack system in the network quarantine area and scanned here. Only the scanned products can enter the R&D test environment. At the same time, the customer has also deployed a WePack in its production environment, which is isolated from other environments. Through the product synchronization function we provide, products that can be released in the R&D and test environment can be pushed to the production environment for release.

The second case is in the game industry . The customer's game software package needs to be distributed to multiple regions. Therefore, we helped them deploy WePack at multiple nodes overseas. Through the product synchronization distribution function, the product package they produced WePacks that can be distributed to multiple VPCs, through the production clusters of different VPCs to pull products for cross-regional deployment.

The above are the landing cases of two specific customers. Regarding the future planning of WePack, it is mainly divided into two aspects: First, on the security control capability , we hope to provide users with more fine-grained authority control capabilities, which can help users achieve resource-granular authority control. At the same time, we will continue to deepen cooperation in safety capabilities and continuously improve product safety inspection and control capabilities. The second point is that we hope that can make the information collection and access functions more complete , so that WePack can be combined with the enterprise's R&D management requirements, and transfer R&D-construction-quality inspection-release information flow. At the same time, we will also improve the compatibility of upstream and downstream tools and deployment platforms to meet the diverse tool chains and deployment needs of enterprises.

Click to watch the CIF summit replay