To be a standardized tool in the cloud-native era to achieve efficient cloud R&D workflow

This article is about CODING R&D Director Wang Zhenwei , shared at the Tencent Cloud CIF Engineering Efficiency Summit.

At the end of the article, you can go to the summit official website to watch the replay and download the PPT.

Hello everyone, I am Wang Zhenwei, CODING R&D Director. I am very pleased to share with you CODING's product thinking and upgrades in the past period of time, and introduce to you the blockbuster new products after the CODING strategic upgrade.

First, let's take a look at the panoramic product matrix CODING. The important product updates made by CODING in the past year are identified here. A large number of products have been newly launched and some existing products have been upgraded. This picture includes almost all the existing important product modules and modules of CODING. Function. We focus on the management of the software development phase, from team collaboration to project planning, from code collaboration to construction integration, from test management to product management and deployment and launch. After deployment and launch, the cloud infrastructure IaaS and PaaS are fully connected. It can be understood that if customers use coding and use the cloud, a full-link cloud native system can be formed.

In the past year, CODING insisted on being customer-centric, moving towards making development easier, and facing the cloud-native future industry, large-scale enhancement of existing products and creation of some very exciting new products. I will focus on some of the new products and important product upgrades in the next section.

Project collaboration 2.0-methodical

We all hope that all things will progress in an orderly rhythm. Then we know that software development work is highly complex, which requires the collaboration of different types of professionals, such as: product managers, back-end engineers, test engineers, front-end engineers, agile coaches, UI designers, etc., different Companies also have different role settings. But if you simply put these people together and lack a definite work communication mechanism, I think their collaboration will fall into a mess.

The project collaboration provided by coding for the developer team has always been the top priority of our products. The project collaboration product team has just completed the 2.0 major upgrade of the product, hoping to make the collaboration orderly. Here I will briefly introduce several important features of Project Collaboration 2.0.

The first is to add custom collaboration mode . We all know that project management software is overwhelming and there may be many choices, but traditional project management software is often based on a specific management theory, such as agile or waterfall, and in-depth design for a specific scenario . In fact, this design method obviously lacks flexibility. For most companies, internal matters cannot be accurately classified as "needs", "tasks" or "defects."

For example, for a financial company, he may need to manage the concept of "risk"; for a start-up company, new products do not yet have complete product requirements, and he may need a concept called "idea" or "brain hole" ; For a software company, there may be a special type of task called "delivery" or "implementation." Then you can see that in fact, different companies cannot use the concepts in traditional project management software to express their business models.

Whether it’s "risk", "imagination" or "delivery", Project Collaboration 2.0 can respond very flexibly. use a custom collaboration model not only to break out of the limitations of agile and waterfall collaboration models, but also to freely base on the actual business of the enterprise. The concept of work items 16193812ca4d6c. A custom concept such as risk can not only set general attribute types such as text, numbers, and radio buttons, but Project Collaboration 2.0 also supports a lot of internal information associations. For example, you can set a brain hole proposer as a member of the project members, and even set the expected realization period of a brain hole in which iterations. With these customized concepts and customized attributes, coupled with the customized process, a set of project collaboration processes exclusive to the enterprise is completely established.

The status of the items can also be customized. For example, companies can set different types of work items according to their own conditions, such as product review phase, design review phase, architecture review phase, under development, tested, to be launched, released, etc. Any state. If a new concept is defined as "risk", then you can set the "risk": different states such as discovered, identified, evaluated, processed, eliminated, and so on. And when these concepts are transferred between different states, we can define their transfer rules and set that only certain types of members can transfer a certain state. For example, only testers can transfer tasks from "under development" to "tested". The flow of corporate matters brings great flexibility and convenience.

The second is the brand new project set function brought by Project Collaboration 2.0. The project set is used to manage the progress of cross-project matters outside the project hierarchy. As shown in the figure, the requirements in the program can be decomposed into different projects of different teams, and can be centrally managed and tracked in the program view.

The following figure is a classic project page information. It can be clearly seen that unlike projects, the project has a clear time limit and several milestones will be set. The mileage progress of the entire project set is clear here. The progress of tasks in this project set in different projects is also very clearly recorded. From the perspective of the promoter of the program, such transparent information and clear time schedule planning can effectively promote cross-team collaboration and effectively solve the problem of corporate departmental walls.

Time is limited, the new upgraded features of Project Collaboration 2.0 are introduced here, and other functions and experiences will not be listed here. We believe that highly flexible attribute and process configuration, clear and intuitive information display, and rule and transparent circulation settings can make collaboration orderly. All the features of Project Collaboration 2.0 can already be experienced on the coding.net public cloud. Customers and partners with privatization requests can contact us.

Continuous Deployment 2.0-Handy

The release of server software has always been a very difficult and complex problem. There are many practices in the industry to address this problem. From our perspective, this process is actually the key link between Dev and Ops in DevOps. But in fact, the concerns of developers and operators are different. Developers need to deliver quickly, while operators are more concerned about stability and reliability. DevOps is actually difficult to reconcile this contradiction, but by reshaping responsibilities, allows operators to focus on operating facility operation and maintenance, while allowing developers to undertake business operation and maintenance; allowing developers to integrate rights and responsibilities, and have the right to publish and view the target environment The program and his key information, and let the developer assume responsibility for the stability of the corresponding business . In this way, publishing can be really handy.

Coding Continuous Deployment 2.0 has completed an all-round upgrade around this goal, and the first to bear the brunt is the application console . Regarding development as the left side and operation and maintenance as the right side, we hope that the power and responsibility of business operation and maintenance will be completely transferred to development, that is, operation and maintenance will be shifted to the left. Developers and operation and maintenance personnel have different concerns. Developers are application thinking, and operation and maintenance personnel are resource thinking. Therefore, if a function that supports developers to take over application business operation and maintenance in an all-round way, it must be application-centric.

In the application console, the corresponding development team can easily see the release history, environment list, monitoring status and corresponding log records of the concerned application. Developers no longer need to find operation and maintenance personnel to query logs, and no longer need to rely on the help of operation and maintenance personnel to add an alarm rule, which truly realizes that operation and maintenance can focus on resources and facilities, and developers can focus on applications.

The software that has been developed and tested needs to be released to truly provide services to users, and risks in the release phase also follow. Although the vast majority of releases are mainly based on the version upgrade of the program itself, the actual situation is that a large number of release processes are also mixed with uncontrollable database changes and configuration item changes, which are often ignored in large numbers. In the past continuous deployment, almost all software and implementation teams only focused on the changes of the application itself, which led to the actual release scenario still requiring a lot of manual intervention. Coding Continuous Deployment 2.0 proposes the multi-dimensional release concept , which can integrate the related release of programs, databases and other external services, configuration items, etc. into an organic whole, fully take over the release process, and truly realize the full automation of the release process.

Another important feature is that we have implemented the GitOps concept . GitOps is a very popular operation and maintenance practice in the past two years. Because deployment is part of the operation and maintenance, naturally GitOps can also solve the deployment problem. GitOps is based on Infrastructure as Code (IaC for short). IaC describes the infrastructure and program status of the target environment as code and stores it in the Git repository. In this case, the change to the target environment is no longer directly operated by the operation and maintenance of the target environment, but by modifying the code in the Git warehouse, and then the GitOps system completes the automatic synchronization of the code and the target environment. Applications based on Kubernetes can be very convenient to practice GitOps, because Kubernetes itself is designed to support declarative definition and final state control, and with the Git warehouse defined by the YAML file, publishing becomes handy.

Coding currently provides a complete ability to practice GitOps, using the coding code warehouse to manage the IaC source code files, using the coding merge request to review environment changes, and CODING continuous deployment to complete the synchronization of the source code and the environment.

Coding Continuous Deployment 2.0 can already accept trial applications from early users, and you can enter the CIF heavy release page to learn about and experience new products.

Nocalhost —— simplify the complex

As mentioned before, the release of server software is complicated, and the concept of cloud native to solve the problem of server software architecture is also quite complicated at the current stage. This complexity is not only reflected in the infrastructure and operation and maintenance work, it is also passed to the development and coding stage. The application of the microservice architecture often consists of dozens or hundreds of microservices, and these microservices have a clear division of labor in the program logic and maintenance personnel team. But such a loose organization has brought huge challenges to the development of coding self-tests.

CODING launched the open source cloud native development environment Nocalhost at the end of 2020. We hope that in the cloud-native era, developers can make the cloud-native microservice coding experience as primitive and pure as a stand-alone application. The development experience of microservices under the cloud is terrible. Developers cannot and do not want to run the entire set of microservices. Since there is no complete development and testing environment, self-testing and debugging cannot be easily performed; if all microservices are run, It needs to consume a lot of resources and is difficult to maintain; the localized operation is too different from the actual container environment, which often leads to the outbreak of subsequent problems. Nocalhost's major upgrade has further simplified the complexity of cloud native development coding in terms of debugging and environment preparation, making the coding simpler.

Developers who have used Nocalhost know that the currently developed microservices are running in remote container clusters. This greatly guarantees the consistency of the development self-test environment and the final target environment to a certain extent, and also greatly saves local computing resources. However, in this case, it is not easy to debug a process running in a remote container: the source code is on the local computer, but the process is in the remote container. If you want to debug, a series of complex networks must be carried out. Get through and IDE configuration. Nocalhost has added the one-click debugging function debug, wait a few seconds, and the IDE will enter the debugging mode.

As shown in the figure below, the left side is the running effect of the target webpage, and the right side is the source code of the microservice. Developers only need to set breakpoints on the corresponding lines of code and go to the web page to trigger the request. When the breakpoint is reached, they can freely control the execution of statements and the debugging process of variable calculations. All things are completely resolved by Nocalhost.

Another painful thing in the development experience is the process of waiting for the service to start. For example, after writing a piece of code, click Restart, and wait for the start to complete before refreshing the page to see the result. The restart process may be measured in minutes for some services. At this time, you suddenly find that you have written a letter incorrectly. You need to restart and wait for a few minutes to see the result. At this time, your heart may be broken.

In fact, different languages and frameworks provide real-time hot-loading capabilities, but the local source code and the process in the cloud container are different, making the real-time hot-loading capabilities provided by these languages and frameworks unavailable in most cases. Nocalhost simplifies the eliminates complicated configuration and principle understanding, allowing real-time hot reloading to truly enhance the coding experience . The developer finds the corresponding service in Nocalhost, right-clicks to start it remotely, and then waits for the start to be completed, only once, and then they can write code happily. The only thing the developer has to do is to write the code, save the file, and then refresh the page to see the result.

In the large-scale microservice architecture application, there are two contradictions that are difficult to reconcile in the development stage. If you want developers to code happily, it is best to provide each developer with a complete set of micro-service architecture application development environment, which is a waste of computing resources and the cost remains high. If you want to save computing resources, you can only find ways to provide several sets of fixed development and test environments for developers to share and use. When multiple people develop, debug, and test in one environment at the same time, the environment will be chaotic. The self-test rhythm will be severely disturbed, and the final efficiency will be greatly reduced.

Nocalhost This major upgrade realizes a logical environment built on top of the basic environment. The microservice components in the environment form a virtual environment through logical division, which not only saves resources, but also allows different developers to isolate each other. is shown in the figure below, assuming that an application is composed of five ABCDE microservices. We run these microservices, keep their functions and versions stable, and set them as the base space. If a developer wants to develop service A, and he does not care about the four services of BCDE, he only needs to give him a logical space, that is, a dedicated development space, which contains only A1 services for development and testing. And reuse the BCDE of the basic space; when another developer also needs to develop A service, he can create an A2 service and form an exclusive development space with other BCDE; another developer or group needs to develop C and D are two services, so just share the ABE. Under this rhythm, each developer can have a complete set of five microservices, instead of each actually running a complete set of services. The principle of the above-mentioned mechanism is complicated and the implementation is not easy, but Nocalhost simplifies the complexity. It only requires the user to specify the basic environment and set the microservice components that need to be used in the development space, and everything can be done.

As you can see in the figure below, normal traffic is marked as blue and will be processed by the services of the basic space and returned to the caller. The traffic dedicated to the developer (Xiao Ming) will be marked as green, transmitted to the development space dedicated to Xiao Ming, and returned to the caller after processing. The dyeing and traffic scheduling is realized by Tracing and Service Mesh , but the user does not need to know the details and can use it directly.

Nocalhost is a completely open source product and plans to donate it to well-known open source foundations this year to promote the overall development of the industry. The functions described above can be viewed on the Nocalhost official website (nocalhost.dev) for corresponding usage documents.

R&D metrics-clear and clear

The world we live in is experiencing a huge wave of digitization. From industry to agriculture, from scientific research to production, from consumption to entertainment, technologies centered on computers and software are measuring the entire world with data. However, for the production process of the software itself, the degree of digitization is not high. This is reflected in the opacity of the software production process, the inability to quantify the stage progress, it is difficult to use numbers to attribute success or failure, and it is difficult to implement bottleneck analysis. Software development is a project. A mature project should not be what it is now. We believe that a mature project should be a measurable, quantifiable, traceable, and . The data of the software development process itself is not clear enough. This leads to the above-mentioned problems and is ultimately difficult to manage.

Coding newly launched the R&D measurement , making R&D data clear.

Coding focuses on the management and efficiency improvement of the software R&D process. To make the data of the R&D process clear, it is necessary to collect data across the entire link. We categorize and collect key data indicators from the five stages of work items, development and coding, testing and verification, build integration, release and deployment, and design key data items based on a large number of industry research and case analysis. The key item data capture of the entire link ensures that the entire measurement system can master the key points, while the highly open scalability ensures the deep extension of a single item.

Simplifying complexity has always been an important feature of coding. There are as many as dozens or hundreds of data in the five major stages, and multiple dimensions such as projects, time windows, and personnel are distinguished. These data are very complicated, but R&D metrics can be configured minimally, and most of the data can be recommended in the first way. Key to generate a data view. As shown in the figure below, for issues such as event planning and human layout, R&D metrics also give a special personnel-oriented view of human saturation; in the case of multi-person cooperation, multi-task parallel, and mixed routine tasks and unexpected tasks, You can understand the work arrangement plan of each person at a glance. The construction of this capability must require comprehensive data collection.

Through a large amount of data and case analysis, we have summarized a set of digital depth of the R&D process and a maturity model of the team's R&D effectiveness. This model is directly built into the CODING R&D measurement. For users, paying attention to these key indicators can understand their own performance maturity status, learn from each other's strengths, optimize their own processes and capabilities in a targeted manner, and finally achieve a multiplier effect with half the effort.

Currently, R&D measurement has been fully launched on the CODING public cloud, and users can experience it immediately. Customers and partners with privatization requests can contact us.

Compass —— Travelling Clouds and Flowing Water

After analyzing a large amount of R&D process data collected by R&D metrics, we have been thinking about the ultimate form of software engineering? We once compared software engineering and construction engineering, software engineering and intelligent manufacturing, software engineering and scientific research, and widely accepted the thinking of programming as art, such as "The Beauty of Programming". Our conclusion is that they are a bit similar, but not exactly the same.

all, software engineering is still a project, and it should have the key features of the project, such as process, schedule, quality, risk, etc. Then superimpose the characteristics of the software on the basis of the project, such as parallel collaboration, iterative updates, etc.

Every software development team has its own special situation, but they all hope that the team has a standardized process and smooth collaboration. Coding has launched a new Compass , which completely embeds our in-depth thinking on software engineering, best practices in process specifications, and core ideas of value delivery into this product. We believe that software engineering requires rules and regulations. The setting of rules and regulations is not only an optimal state on a single point, but also must be efficient from a global perspective. Compass guides the direction of software engineering and makes R&D flow smoothly.

As the answer to our ultimate thinking about software engineering, we named this standard execution method and value delivery flow across the entire link: Compass. Compass means compass or compass. We hope that members engaged in software development can be pointed out by Compass, instead of relying on colleagues or leaders to arrange work. A sophisticated software development process, the system should tell people what they should do according to specifications and processes, instead of blindly finding the direction of work.

Process is the core of Compass. For a R&D team, in order to establish an efficient R&D specification, the first task is to confirm what process the team uses to develop and deliver software. Compass's process engine is highly flexible, and can arrange work items from the project, to code branching and merging rules, to source code quality specifications, to building test red lines, to product storage structures, and finally to the release and delivery process.

It can be said that Compass has set up a R&D full-link process definition. Once this process is defined, it is very convenient to force members of the R&D team to follow the process to execute the work content, and then meet the specifications. We believe that a clear process will give clear expectations and will also produce structured accurate measurement data. For the R&D team, each role will get a Todo List. This Todo List is automatically calculated by Compass according to the execution node of the process. Each member only needs to continue to complete the items in the Todo List. The whole process will advance automatically.

For example, a developer wakes up in the morning and turns on the computer, and sees that it is clearly stated in the Todo List listed by Compass that he has three coding tasks, two code review tasks, and one release approval confirmation pending. He can fully understand the basic upstream status of each task based on system information, and can also understand the impact of task completion time on downstream based on process calculations. Such clear work instructions are like a compass that guides the direction of work all the time. All these things are automatically driven by the system to move people forward. On this basis, it can be understood that we have almost transformed the software production process into a manufacturing workshop assembly line. The only thing the assembly line workers have to do is to wait for the output of the previous step and perform their own processing. It is finally delivered to downstream for further processing.

Because the process drives people to advance the progress of the affairs, the start and completion of each stage are recorded by the system, and all data will be reported to the R&D measurement system in a unified manner, so that managers can easily perform bottleneck analysis. For example, if too much time has been spent in testing in the past month, you can carefully analyze whether it is insufficient manpower, backward testing tools, or lack of personnel.

structured, procedural, digitized, and standardized R&D process produces a large amount of actual data, which in turn is used to optimize R&D rules and process settings, forming a two-way positive cycle, making the process and specifications themselves like products Iteratively, the company's research and development efficiency can truly step by step and gradually improve. From this perspective, Compass extends the idea of agile iteration from software product development to management systems.

Since every specific matter has specific norms and guidelines, this can be very intuitive and convenient to constrain the behavior of operators. This provides for core technical organizations such as CTOs, CIOs, technical committees, and architecture support departments to implement technical practices throughout the enterprise. A strong guarantee. For example, we all know that there are many branch model scenarios in the Git warehouse, and Compass can set the branch naming rules, merge process, merge permissions, pre-checks, post-notices, etc. in detail for the branch merge scenario. And so on, we hope that all the key links in the research and development process can be followed by rules, and finally achieve the effect of trying to find out according to the picture, so that the technical management of the CTO can be upgraded from verbal teachings to systemic restraint guidelines.

Compass is the ultimate answer to CODING's thinking in software engineering. We hope to turn software research and development into a smooth and streamlined factory assembly line. The final effect is that the machine pushes people forward, rather than people push the machine forward. Compass is above all other product modules in the coding system, and is a global and very large product system. At present, the core process engine of Compass has been built. You can visit the heavy release page of the CIF conference to learn more about Compass's features and applicable scenarios, and apply for a trial.

there are more……

Coding has made fruitful product iterations in the past year. Due to time constraints, I will not repeat them one by one. Here are some key updates——

1. Coding is about to launch the CI engine independently developed by Tencent to solve the inconvenience caused by the long-term control of Jenkins. This engine has been used within Tencent Group for many years, has been proven, powerful, and flexible. The new engine will cooperate with Tencent Cloud's secure container to achieve faster scheduling and more flexible orchestration capabilities. Coding's new CI engine is currently available for trial by early users, and you can enter the CIF heavy release page to learn about and experience the new products.
2. CODING released an independently deployed product library in 2020: WePack . At present, WePack fully integrates the security capabilities of Tencent Group, and has joined forces with well-known security teams in the industry, Yunding Security Lab and Coen Security Lab, to greatly enhance WePack's capabilities in product scanning and security reinforcement. WePack can not only use industry-public vulnerability database scanning products, such as NVD, CNVD, etc. that everyone is familiar with, but also has the ability of Tencent security team for more than 20 years. Recognized. WePack can be easily and privately deployed in the customer's environment. For more information, you can enter the CIF heavy release page to learn about new products.
3. CODING is based on the Git code base enhances the directory-based read and write permission control . Teams migrating from SVN to Git often complain that Git cannot control read permissions at the directory level. Anyone who knows the principles of Git knows that Git performs version verification based on the hash algorithm of the entire code base. If the exported files are not complete, the verification will not be completed. This basic principle causes Git itself to be unable to implement read permission control by directory. Starting from the principle, coding extends Git. On the basis of being compatible with existing Git usage and not intruding into Git, it implements directory-based permission control by extending Git. This ability is currently available for trial by early users, you can contact us to apply for a trial.
4. Cloud Studio The team has greatly improved the experience of IDE Cloud Studio on the cloud. Cloud Studio is now more convenient and faster, and at the same time the degree of convenience has surpassed the local IDE. Developers do not need to install any software, they only need to open a browser and log in to their account to start programming. Starting from opening a workspace on the cloud, it only takes 3 seconds to complete the loading process until the workspace is fully available. The new version of Cloud Studio is now fully online. You can go to the official website (cloudstudio.net) to register and use it. If you have a privatization deployment requirement, you can contact us.

Coding hopes to build a full-chain cloud native development system, and sincerely thank customers, partners, and peers for their support and help. The cloud-native development system is currently incomplete, and coding still has a long way to go. We look forward to the coming of the cloud-native era in the future, making development easier!

Click to watch the replay of the CIF summit and experience the new CODING products in depth!