Shopee's exploration of React Native architecture

1. Background

React Native (hereinafter referred to as RN) is a popular cross-end development framework in the field of hybrid applications. RN is very suitable for flexible and changeable e-commerce business. Since RN is based on client-side rendering technology, it has certain advantages in user experience compared to H5 pages.

With the rapid development of Shopee's business, the amount of RN code in our App has grown very rapidly, resulting in problems such as excessive build volume, long deployment time, and conflicting dependencies between different teams. In order to deal with these pain points, we explored the decentralized RN architecture, and combined this model to develop the system (Code Push Platform, CPP for short) and client SDK, covering the development, construction, release, operation of multiple teams, etc. Series RN R&D cycle. After nearly three years of iteration, it has access to a number of company-level core apps.

The front-end team of Shopee Merchant Service has created a variety of merchant-side applications, most of the users are merchant service personnel, they have high requirements for the high availability of the business system and timely feedback of problems, which also promotes us to have a higher architecture of React Native. requirements.

This article will introduce how we can meet the development needs of multiple teams in complex businesses step by step from the four directions of development history, architecture model, system design, and migration plan.

2. Development history

With the rapid development of our business, the number of our RN bundles has increased rapidly, and the number of apps has reached nearly ten. The entire RN project has changed in the three dimensions of development model, deployment model and architecture model, from a single team to multiple teams, from one bundle to multiple bundles, from a centralized architecture to a decentralized architecture, and finally to a Each team's business code can be independently developed, deployed, and run.

The entire development history is divided into 4 stages, namely the single-bundle centralized development mode, the single-bundle multi-business group development mode, the multi-bundle centralized publishing mode, and the multi-bundle decentralized publishing mode.

2.1 The first stage: single bundle centralized development mode

The overall technical architecture of the initial RN is relatively simple. Since the business form was not complicated at that time, in order to meet the development process of independent teams in the same code repository, the entire release process was based on CDN update release, and the configuration file was used to record the version and download address of the RN bundle file, so as to carry out resources manage. There are two products of the whole release, one is the RN resource package, and the other is the JSON configuration file for resource version management.

After each RN resource is built, these two build products will be placed in the static resource directory. The App will automatically pull the configuration file at a specific time node (such as App restart, etc.) to check the resource update status, and then pull the RN static resources from the CDN. The next time the page is opened, the app will load the latest page content.

With the development of business, more and more business teams expect to use the RN technology stack to develop business. This situation changes the existing architecture. We naturally have the idea of "multiple business groups and multiple code repositories".

2.2 The second stage: single-bundle multi-business group development model

In response to the above problems, the R&D solution for multi-service groups is the mode of host-plugin.

host is used to manage common dependencies and general logic. It manages React, React Native, Shopee RN SDK, etc. through an independent warehouse, which ensures the "singleton" (singleton mode) condition of special RN dependencies and avoids some clients Overlapping dependencies of components, which are officially not allowed by RN.

A host corresponds to multiple plugin repositories, and the business code repository is regarded as a plugin, which is connected to the main application in the form of plugins. Business teams can manage this repository according to their own coding standards. Each plugin repository is treated as an npm dependency of the host project, and its build is a centralized publishing process. All code will be integrated into the host project to execute the build script. This mode meets the requirements of super apps.

At the same time, the host-plugin model also brings a "difficulty". Business development makes the volume of RN products gradually increase. Excessive products will affect the decompression efficiency of the client and the time it takes for the RN container to load JS.

2.3 The third stage: multi-bundle centralized architecture mode

In view of the problem that the RN product is too large, we use the build tool to subdivide the packaged product into multiple bundles. This optimization is very necessary, and we call it "sub-package". The host project corresponds to the public package, and the plugin project corresponds to the business package.

The entire build occurs in the host project, and the mode of the project is still "centralized build" and "centralized release". Multi-bundle products will be published to the system, and the client will pull the hot-updated content. The client will load the corresponding bundle on demand, and the resources consumed by the RN container for a single load are greatly reduced, which solves the problem of efficiency.

But its shortcomings are also obvious. With the expansion of the business team and the expansion of business content, the multi-bundle centralized publishing model also has four disadvantages:

• For the runtime of RN , even if the subcontracting technology separates the products, they still run in the same JSContext, which may lead to dependency conflicts and environment variable pollution;
• In the process of development and debugging , the project relies heavily on the host project. Every time there is a code change, a lot of content needs to be reloaded, making development and debugging less friendly;
• In the process of project construction , the packaging speed is affected by the number of plugins. Extra-large applications even need 50 minutes to execute a build. Excessive build time will seriously affect the release efficiency;
• In the process of deployment and release , the host project maintainer is responsible for the entire App. Each business group cannot be released independently, and the release time will be bound together. When a live issue occurs, developers need to spend a lot of communication costs and can only roll back as a whole.

2.4 The fourth stage: multi-bundle decentralized architecture model

The decentralized React Native architecture model is similar to the concept of "micro front end" of web pages or "micro application" of client side, which satisfies the independent development and deployment of multi-business teams, and can run independently in each module of the same App. It covers development, building, publishing, running, and more. This model solves the four drawbacks mentioned above, and has a comprehensive upgrade for the entire R&D system. The advantages are: mutual non-interference when RN runs, efficient development and debugging, and independence of construction and release.

The following sections will focus on the project's decentralized RN architecture and system design, and how we balance flexibility and stability.

3. Decentralized RN Architecture Model

Simply put, the decentralized RN publishing model involves four parts: independent JS runtime; independent development process; independent construction process; independent publishing process. With the help of these four key links, each team controls the RN R&D process at its own pace.

3.1 Standalone JS runtime

The emergence of independent runtime (multiple JSContext, execution context environment) is the biggest feature of the decentralized architecture. The independent runtime is the perfect guarantee for independent release. By isolating the RN running code according to the plugin dimension, it can effectively avoid variable conflicts and dependency conflicts between different businesses, that is, the release of "plugin A" will never affect "plugin A". plugin B".

Its design mainly includes the following three points:

• Create JSContext in advance and preload public packages;
• Enter the plugin page, the SDK will check whether the corresponding JSContext has been instantiated. If it has been instantiated, use it directly, otherwise select an independent context from the JSContext Pool, load and execute the business package, and the runtime between each plugin is isolated;
• When you exit the business page, the JSContext will not be destroyed immediately, but will be put into a cache pool, so that you can get the ultimate experience by repeatedly entering the business.

The container of the device JSContext can be a thread or a process. In order to avoid it being created and recycled frequently, we need to maintain the buffer pool and reuse the existing JSContext as much as possible.

Here we adopt the strategy of Least Frequently Recently Used (LFRU for short). The JSContext is re-enabled when the app that just quit is reopened. This way, we were able to save 85% of the above-the-fold rendering time. The management of the number of caches is configurable, and the business side can make a reasonable estimate based on the scale of the application. When the RN page is still in use, even if it exceeds the estimated number, the context will not be reclaimed immediately. This design effectively guarantees the availability of the page.

3.2 Development Process

The debugging efficiency of the RN project is mentioned above. As the volume of business code increases, the code debugging efficiency will also decrease. The efficiency of each developer directly affects everyone's "happiness". In contrast, RN decentralized publishing is optimized for the development process.

With the emergence of the independent runtime environment, when RN enters debugging, the client can only load one plugin into the corresponding JSContext, and other plugins use the built-in cache.

This has two advantages: one is to minimize the scope of service startup and ensure the efficiency of code hot loading; the other is to ensure the consistency of the two processes of development and construction, which will expose some problems in advance in the development phase , such as compilation problems caused by missing babel plugins. Such a "decentralized" development process improves the efficiency of RN debugging.

3.3 Build Process

With the development of the business, there are 4 RN plugins for an App. The old construction process is affected by the number of plugins, and the centralized construction time exceeds 20 minutes. With the decentralized RN architecture, the construction time no longer increases with the number of plugins, but is only related to the amount of code of the plugin, which is stable at about 5 minutes.

The new architecture is also based on the host-plugin model, and the isolation of independent repositories allows each team to have free development space. Considering that the basic native dependencies within the application are unified, the host project is only used to manage the unified public dependencies. The project needs to build the common bundle first, and the system will record the dependency information in the common bundle. When each plugin project is built, the build tool will remove the public package dependency information and complete the construction of the business package. The construction products of each business package are stored in the system independently. The system has the capability of independent rollback, independent release, and independent grayscale.

The advantage of this is the minimal granularity of the build task, and the construction of each plugin will not cause the entire project to be rebuilt, so that it can truly "package on demand".

3.4 Publishing Process

The build and release of RN are two separate processes. This also means that the building link and the publishing link of the bundle are completely decoupled, and the release time point can also be flexibly arranged by the release leader of each business team. Each business group is responsible for its own code quality, flexibly controls its own version release rhythm, and will not affect the online business of other teams. The release process includes operations such as full release, joint release, grayscale release, and rollback. Subsequent chapters will detail how to ensure release stability.

4. System Design

For complex large-scale projects, the simple hot update process can no longer satisfy the cooperation of multiple business groups. We need a hot update system with complete functions, superior performance and friendly operation to meet the development of complex services. Code Push Platform is written in Node.js, with system-affiliated command-line tools and client SDK.

In order to meet the operation of the system in multi-business teams, the whole system can be divided into three parts from the functional point of view, namely:

• Multi-team authority control;
• bundle life cycle management;
• System performance is improved.

Among them, the system performance improvement function is subdivided into:

• incremental difference;
• Multi-scene entrance volume optimization;
• One-stop multi-environment integration.

4.1 Multi-team permission management

In addition to recording each build operation, the system is more important for the decentralization of the workflow, and the permissions of each plugin are isolated. Each person in charge can only operate within the system. The person in charge of plugin 1 can only trigger related builds and releases, and cannot see the operation of plugin 2. The system regulates all release processes through strict authority control, ensuring the controllability of the project.

The design goal of React Native's decentralized release is to save communication costs between different teams. The system will limit their build and release actions, and their releases will not interfere with each other.

Permissions are managed in a tree-like structure. An App corresponds to a project, and the project leader is the project leader of the App team by default. System operations such as creating a brand new plug-in require the approval of the project leader. An App contains multiple plugins, and the person in charge of each plugin is the corresponding business team leader by default, and he has the authority to assign the authority to publish and build.

4.2 Bundle Lifecycle Management

4.2.1 Client Version Control

RN is different from web applications in that it has a close dependency on the client. In the case that the underlying dependencies of the client do not change, in general, developers can update the RN code through hot update. However, when encountering major updates, such as the upgrade of the version of React Native from 59 to 63, not only JavaScript side changes are required, but the client side also needs to be upgraded and cannot continue to be backward compatible. From a technical point of view, it is unavoidable. This situation where the client is not backward compatible is called a "fault".

The system will provide the ability for client version control. When a major change occurs, the app owner should create a new "fault information" on the system, with version numbers ranging from the lowest app compatible version to the highest app version. Only in this interval can the client pull the latest RN resources of the fault.

As shown in the table below, the version 105 RN package is pulled from the app of version 2.5.0 or greater; the version 103 RN package is pulled from versions 2.0.0 to 2.5.0; the version 1.0.0 to 2.0.0 is pulled to version 100 of the RN package.

This measure can effectively avoid potential risks. The latest demand will only be launched on the latest fault, and the old fault will only be repaired online. After all, there are two sets of codes, and the maintenance of the code is costly. As users update to the latest version, the old faults should be gradually eliminated.

4.2.2 Grayscale and Rollback

The release process includes operations such as full release, grayscale release, and rollback. For large-scale demand, going online in full will bring potential risks. Generally speaking, the new version is given priority to some users, and the person in charge of the release can conduct grayscale release according to the specified users and specific scope, and gradually expand the scope of grayscale release until it reaches full volume. When a major bug is found, the publisher can use the "zero build" method to perform a "second-level" rollback.

The decentralized RN architecture supports independent release, independent grayscale, and independent rollback of each plugin, and ensures quality and avoids risks with minimal granular operations. The grayscale and rollback of the plugin dimension level can bring flexibility to different business teams. Each business team can release the version by itself, control the grayscale rhythm, and deal with online problems.

4.3 System performance improvement

4.3.1 Differential Increment

Frequent updates of the RN resource package by the app will consume user traffic. The most effective way is to use incremental updates to save traffic. The RN resource package covers compiled JavaScript products, images, translation files and other static resources. The difference between their previous and previous versions is the code or other resource files changed in this version. In order to make the difference granularity go deep into the resource package, the system provides an independent "differential service", and uses the binary difference method to differentiate the build products.

The diff (differential) operation of the RN resource package is completed on the server side, and the patch (integration) operation is completed on the App side. In the decentralized RN architecture, the difference of each plugin is independent. The release of the plugin automatically triggers the execution of the difference. The system will pull the latest five versions with the plugin as the dimension, and the Diff Server will perform the difference calculation between them and the current version in turn. If the calculation is successful, the difference result will be uploaded to the CDN and fed back to the system, otherwise continue to retry. The entire differential operation is an asynchronous process. Even in extreme situations such as "differential service" offline, the system will automatically downgrade to a full package to ensure system availability.

4.3.2 Multi-scene entrance volume optimization

Since React Native's build officially relies on metro.js, it doesn't have tree-shaking capabilities. With the expansion of business code, the optimization of package size is a very important issue.

For example, ShopeePay provides payment services for many of the company's core apps. The ShopeePay plugin has some page-level differences between different regions and different apps. The same repository contains all the code and resources, but the build script will package them all into one product. Obviously, this results in ShopeePay's release product containing a lot of redundant resources, which is not optimal, wastes download traffic, and also affects the execution efficiency of the code.

We use a self-developed multi-scene plugin (babel-plugin-scene), which sets a scene value through the injected environment variable. Babel can load different files according to the difference of the scene value, and use the default file as the downgrade. Different scenarios correspond to different entry files, and the package volume can be effectively controlled by using this form.

4.3.3 One-stop multi-environment integration

A normal development process is from the test environment, to the uat environment, and then to the live environment. Code Push Platform is connected to the test/uat/live environment of the App, so RN developers only need to perform a "one-stop" operation in this system to meet the entire R&D cycle of a requirement.

The flow of package resources in different environments is a highlight of multi-environment integration. If an RN bundle is built in the uat environment, it does not need to be rebuilt, and the bundle is seamlessly converted to the online environment for publishing. The advantages it brings are the "zero build time" and the stability of the resource bundle, because the bundle is not rebuilt, so its content has been fully verified in uat, and the release risk is less.

5. Migration plan of old business

How to migrate existing business apps is a very serious issue, especially for businesses with a heavy historical background, which may have scenarios of "logical coupling" or "component coupling". At the same time, many related businesses are in the process of requirement iteration, and the migration of the system cannot hinder the requirement iteration, so the "gradual migration" scheme of old business is very necessary.

5.1 Logic coupling

If there are logical dependencies between two or more plugins, the user must load the latest plugin at the same time. Considering the possibility of hot update failure, logical coupling is that multiple plugins hide a constraint relationship. For example, there is a certain logical coupling relationship between the order business and the purchase business. It is impossible for the person in charge of publishing to publish plugins one by one for super apps with huge traffic. In an extreme state, the user may load plugin A first, and the new version of plugin A and the old version of plugin B are incompatible, which will bring serious consequences. In this case, there are two solutions:

• Option 1 : Logical decoupling between plugins to ensure the independence of each plugin.
• Option 2 : The system provides joint release, which ensures that multiple plugins can be loaded to the latest at the same time on the Native side.

Option 1 is the most ideal state, but in the case of subdivided business scenarios, it is difficult for the project structure to be absolutely independent.

Option 2 can be considered for the old business. The system provides the concept of module. One module corresponds to more than two plugins. They have a bound relationship. In the same download task, the client SDK uses the "transaction" form to ensure that multiple plugins can be downloaded and put into use at the same time. Jointly releasing this capability at the system level effectively avoids the possibility of such errors.

5.2 Component Coupling

If joint release is a compatible solution for "logical coupling" in the plugin dimension, "component coupling" is a more fine-grained component-level coupling relationship. That is to say, there are multiple components in a page from different teams. For example, pages such as product detail pages have evaluation function components. This kind of "a page exists JSContext nested with each other" situation exists in the e-commerce business.

For this "component coupling" situation, there are two solutions:

• Option 1 : The nested components are extracted into an independent repository for use by third-party plugins.
• Solution 2 : Use the ability of "same screen rendering" to realize "multi-Context nesting".

Option 1 is the most ideal solution. But considering the migration cost, we also provide solution 2 (a "same-screen rendering" nested component) to support this scenario, which is similar to a native component. In the case of multiple JSContexts, nest the desired content into another page by plugin name and page name.

As shown in the figure below, plugin A will nest the content of plugin B, and A and B can also be rendered on the same screen. From the perspective of the Web, this situation is a bit like the "iframe" scenario, which supports the nesting of multiple pages. It's very easy to understand for RN developers, and the client SDK can dynamically load the target bundle and render it in place.

5.3 Incremental Migration

For existing apps, because the business cannot suspend iteration, it is difficult for us to complete the overall migration at one time. Therefore, we offer an "incremental migration" scenario. Taking into account the historical background, this solution will not migrate all plugins at one time, but will gradually split them up and migrate to the new release system.

The migration steps are shown in the following figure:

• Migrate independent businesses to Code Push Platform first, and they enjoy an independent JSContext;
• All "codes to be split" share a separate JSContext;
• Continue to split the "code to be split" into several independent plugins, use JSContext independently, and keep the state of step 2 for other content.

As the version iterates, repeat the second and third steps until the historical business is all split. In this way, we can achieve an optimal goal, that is, "independent build" and "independent release" in the true sense.

6. Summary

The goal of the system is to meet the multi-team R&D collaboration efficiency of all apps. The decentralized RN release model takes into account the four aspects of "independent runtime", "independent development", "independent construction" and "independent release", ensuring that The independence of each plugin to run. The ultimate goal is to support Shopee's multiple RN teams to publish freely and operate efficiently on different App platforms at their own pace.

The system design involves "multi-team authority control", "client version control", "grayscale and rollback", "incremental difference", "multi-entry package volume optimization", "one-stop multi-environment integration" to accelerate The entire R&D process has truly achieved both "flexibility" and "stability".