The evolution of the Shopee Games game engine

The author of this article : Shopee Games front-end team.

Summary

The Shopee Games team is committed to enriching the interactivity and entertainment in Shopee e-commerce, allowing users to enjoy more pleasure after shopping, and games can also bring continuous active users and more coupon distribution channels to Shopee. In this context, from the very beginning of the game's birth, we hope that the game is lightweight enough and can iterate quickly, so as to continue to provide users with a variety of game experiences, while not having a big impact on the size of the Shopee App. Therefore, we need to choose the right game engine and build a toolchain suitable for Shopee Games.

This article will introduce how the Shopee Games team chooses a game engine, how to expand the game engine to improve production efficiency, and how to combine the game development process with a mature front-end engineering system to achieve game standardization and improve R&D quality. Shopee Games is a game embedded in the Shopee App, so for business teams that have the same needs for embedded games, the experience summarized in this article will be of some reference.

1. Game engine selection

Shopee Games currently focuses on casual games. In order to reduce the impact on the size of the Shopee App, the technical selection will be biased towards H5 games. And how to choose the H5 game engine, we mainly consider the following factors:

• 2D or 3D games?
• Is it developer friendly? Including whether TypeScript is supported, whether the documentation is complete, and whether the R&D process is suitable for development-led.
• How about performance and compatibility?
• Is the official toolchain perfect?
• Is it open source?
• Are there any successful game cases?
• Is there official customer support?
• Are there any official updates?

First of all, our casual games are mainly 2D games, so we will focus on game engines designed for 2D first. Although 3D engines can achieve 2D effects through orthogonal perspectives, the rendering performance and light weight are not as good as specialized 2D engines, so we first exclude 3D-based engines, such as Unity3D, LayaBox, Three.js, Babylon.js . As for 2D engines, there are mainly old lufylegend.js, Cocos2d-JS and continuously updated Egret and Cocos Creator in China, and Phaser/Pixi and CreateJS in foreign countries.

Then, from the perspective of sustainability and performance, the older lufylegend.js and Cocos2d-JS can be excluded first. And CreateJS is not actually a complete game engine, it is closer to a streamlined rendering engine, lacks the overall tool support, it is difficult to support large-scale games, and it is also excluded.

So, finally, we focus on comparing Egret, Cocos Creator and Phaser. Phaser's rendering engine is Pixi, and Phaser is used to represent both.

The above three game engines all support TypeScript and WebGL, with little difference in performance. For the Shopee Games team, Egret has big advantages:

• Egret supports canvas mode, so some low-end mobile phone users in the Southeast Asian market can also run our games;
• Egret's philosophy is for developers, and our team has strong R&D capabilities, and being developer-oriented can make the entire game perform better;
• On the tool chain, Egret has a self-developed keel animation and editor, which is very suitable for our game development.

Therefore, in summary, we choose Egret as the mainstream engine, and on the basis of Egret's ecology, we will continue to optimize and build a tool chain that can improve the efficiency of game development.

2. Egret engine optimization and public libraries

2.1 Egret engine optimization

Egret Engine is an open source game engine developed by Egret Times that follows the HTML5 standard, including 2D/3D rendering core, EUI system, audio management, resource management and other common modules of game engines.

At present, we have used Egret Engine to develop four games, Shopee Candy, Shopee Pet, Shopee Fruit, and Shopee Link. During the process of project development and iteration, we found that the official engine had some problems and could not fully meet business needs and performance standards. Therefore, we customized the development of the Egret engine, hereinafter referred to as "customized engine".

2.1.1 Performance report

Report some data indicators of the game, such as FPS, DrawCall, First Paint, GPU Size and other performance indicators.

The reporting process is as follows:

By analyzing the reported game performance data, we can better analyze performance bottlenecks, so as to focus on improving game performance. Among them, the detailed performance indicators involved are as follows:

2.1.2 Performance optimization

We have made some performance optimizations based on the official engine to help developers improve game performance.

static composite

In the development process, the scattered images are synthesized into a large image atlas to achieve the purpose of reducing DrawCall.

dynamic combination

Dynamically merge textures into one large texture when the project is running. When rendering a texture, the dynamic combination system will automatically detect whether the texture has been merged into the atlas (image collection). If not, and the map is eligible for dynamic binning, the map will be merged into the atlas.

The dynamic combination of images selects which textures to merge into a large image according to the rendering order, so as to ensure that adjacent DrawCalls can be merged into one DrawCall.

Like the previous static image combination principle, the combined image texture is used to replace the fragmented image texture, thereby reducing DrawCall. The biggest advantage of dynamic image combination is that it improves some scenarios that cannot be statically combined in advance, such as the user's dress up.

Node order adjustment

The performance optimization at the bottom of the engine is to ensure the rendering order of the same texture. For example, when addChild at the same level, if the original order is img1>text>img1 , the engine can automatically optimize to img1>img1>text and reduce DrawCall.

DrawCall optimization tool on

The game Main function opens Benchmark.init(null,null,true) ; or Benchmark.optimizeDc set to true.

2.1.3 Engine downsizing

The official engine includes all modules by default, some of which are not used in our actual project. Therefore, in order to reduce the size of the engine package, it is necessary to remove the unused code, such as:

• Native code;
• Runtime code;
• Compatible codes for small game terminals such as WX;
• KTX texture related code;
• ETC Loader code.

Comparison before and after modification:

In the end, the game front-end JS load was reduced by 16KB, or about 7%. Although this volume may seem small, a small amount of volume optimization is valuable for some areas with poor network.

2.1.4 Bug fixes

For some engine-level bugs encountered in the project, because the engine may be updated and repaired in a timely manner, we often need to fix it ourselves. For example: iOS 14/15 rendering problems, keel library rendering problems, network and sound effects, etc. Among them, after we solved the iOS 14/15 lag problem, we are honored contributed code to help the official Egret team solve this problem .

2.1.5 API enhancements

Some usages of the official engine are too cumbersome and not friendly enough, such as setting the node width and height. Therefore, based on the official engine, we have extended a convenient and fast API for everyone to use.

2.2 Common library

In order to improve the development efficiency and prevent everyone from reinventing the wheel, based on the optimized Egret engine, we have developed a common library, encapsulating common tool classes, common modules, common UI components, and so on.

2.2.1 Tool library

We encapsulate some tool libraries commonly used in games:

• SoundUtil : Music playback tool class, supports playback/pause/multiple playback of sound effects/background music, etc.;
• DragonUtil : keel tool class, responsible for the creation/destruction of keel animation, hides the details of keel creation, and simplifies the use of keel animation;
• ResUtil : Game resource management class, which is convenient for developers to load/release game resources;
• SmartEvent : Encapsulated message notification class library, convenient for everyone to use, easy to decouple between modules, including monitoring and removal of custom events/UI events;

Encapsulating the tool library is to reduce the difficulty of development and avoid repeated wheel building by different teams. At present, it has been used in four Egret games of Shopee Games, saving labor for more than 2 weeks on average.

2.2.2 Basic UI Components

We have extended the basic components of Egret, and provided a series of hook functions such as the life cycle to reduce development difficulty and improve development efficiency; at the same time, we provide some common components for each project, such as: sharing interface/friend interface/little monster bomb Common UI components such as windows.

Egret base component extension

We provide some life cycle hook functions for UI components to facilitate the use of game business, and developers do not need to implement event monitoring and removal separately when implementing each UI class. At the same time, the built-in event management also avoids memory leaks that developers may miss due to development omissions.

The specific hook function is as follows:

2.3 Customized engine synchronization update

With more and more modifications to the customized engine, the following question is: if the official engine is updated, how can we quickly merge the official engine version?

The solution adopted here is the git dual remote solution. The flow chart is as follows:

The detailed steps are as follows:

• For convenience, we define the Egret engine open source library as A, and our own customized engine warehouse as B;
• Pull and modify project B through git clone B;
• Through git remote add A repository, and git fetch A, add A remote and get A's warehouse information;
• Assuming that the development branch of B is dev, switch to this branch;
  Suppose we need to merge a tag of A, such as v5.4.0, use git merge v5.4.0 --allow-unrelated-histories to force the merge.

Due to the need to synchronize the source framework project code, our changes will be limited, otherwise there will be repeated workloads for each merge:

• Try not to rename or delete the original file, or change the function and variable names in the code;
• If you need to expand the function of a class, try to use the form of prototype chain expansion;
• Custom internal tool classes can be defined internally, as long as they do not have the same name;
• The inline code should use the added mode as much as possible, and try not to change the original code;
• Some library code will add a lot of channel-compatible code, we can reduce it appropriately, the merge will be based on the mechanism of analyzing changes from the common ancestor, so it will not be diff every time.

Through the above solutions, we can achieve rapid synchronization between the official warehouse and the customized engine.

3. Game R&D Engineering

Although the Egret engine can meet the basic business needs of Shopee Games, the official also provides a series of tools to meet the development needs of developers. But in the process of using the Egret engine, we still encountered the following pain points:

• lacks the concept of modules : The default TypeScript method is used to compile, and the top-level import and export of files are not supported. The contents of all compiled files are regarded as globally visible, which is likely to cause variable pollution and security problems;
• cannot use npm : The package.json file is not supported in the root directory of the business project, and modular third-party libraries are not supported;
• lacks engineering solutions : There are no engineering-related solutions, such as code review, unit testing, etc., and the project cannot easily access conventional web front-end engineering solutions;
• deployment process is complicated : The code compilation tool depends on the official tool, and the command line version is not provided, so it cannot be deployed separately on the server.

Obviously the Egret project could not meet our engineering needs. Even though the current Web front-end engineering technology is very mature, we are still in the stone age, so we decided to implement the front-end engineering of the Egret project.

3.1 Egret front-end engineering

3.1.1 Support root directory package.json

The package.json file can be said to be a necessary file for front-end projects at present. The Egret engine started relatively early, and the front-end engineering was not so mature at that time. The construction of the Egret engine was a set of build systems written by the official itself.

The package.json file in the root directory is not supported, and many things are difficult to execute. Fortunately, the build tool code of the Egret engine is also written in JS, and it is as open source as the engine code.

Through source code breakpoint debugging, we found that the reason why the Egret project does not support package.json in the root directory is: when Egret is built, it distinguishes engineering projects and library projects by judging whether there is package.json in the root directory, so as to use different builds process to build different products.

In order to minimize the changes and also support the existence of package.json in the root directory of the project, we modified the judgment of building the project to judge the value of the custom field in the package.json to distinguish whether it is a project or not.

It supports the existence of package.json in the root directory, and it is easier to carry out some subsequent engineering transformations.

3.1.2 Engine npm package

The official build relies on the build tool on the local machine. Each deployment release needs to be uploaded to the server after the local build is completed, which is not in line with the deployment specifications and process of the Shopee business, and seriously hinders the rapid iteration of the project. Rhythm.

In order to enable the build to support separate deployment on the server, we transform and encapsulate the code of the customized engine, and publish it in the form of an npm package. The project dependencies are changed from a local build tool to an npm package.

  "dependencies": {
    "@egret-engine/egret-core": "1.6.2-alpha.1",
  }

The npm package mainly consists of two parts:

• build directory: engine-related library files;
• tools directory: Build and compile related tools.

Publishing to npm not only makes the compilation and running of the project out of the local environment, but also enables better version management of the project. However, it is not enough to publish it as an npm package. It also needs to be packaged and built with the following Webpack to achieve our purpose.

3.1.3 Webpack packaging and building

In order to support modular compilation and separate deployment on the server, we chose a mature Webpack build solution to integrate into the Egret project.

Before rebuilding the Egret project, you first need to analyze the dependencies and build products of the Egret project:

• *.js : Code build artifacts.
• *.ts : TypeScript business code file.
• res : Project resource file. For example: images, audio, JSON files, etc.
• egret libs : The Egret project depends on the module, that is, the related JS library file.
• *.exml : Egret-specific tag language file type, used as UI layout, which can be compiled into JS files and JSON files.

The official build is similar to gulp, executing each task in a certain order. Although the official also provides a way to customize the task plug-in, allowing developers to customize the construction process, but this requires developers to redevelop, which is more labor-intensive.

The exml file type is a file type unique to the Egret engine. At present, there is no relevant parsing and compilation tools in the front-end ecosystem; there is no need to reinvent the wheel for res file processing, so we continue to use the official tool and package it on @egret-egine/egret-core/tools as a dependency of the build tool.

For egret libs dependency processing and *.ts code compilation, we can all find better solutions in the front-end ecosystem, which can be used as needed.

The Egret project is packaged through Webpack, and the build dependencies come from npm, so that the build can be deployed directly on the server without the local environment. And the product is also consistent with the official packaged product, so that it is well compatible.

3.1.4 Engineering configuration

After the above transformation, in fact, the Egret project is not much different from the ordinary web front-end project. The mature web front-end engineering solution can be well practiced in our project. It can not only be deployed separately on the server, but also easily Access quality control tools, such as eslint, jest, etc., to improve code quality.

3.2 Egret-Webpack-CLI implementation

At the beginning of the project, we mainly built project scaffolding templates based on Egret and Webpack according to business and engineering needs. However, when creating a new project and creating a demo project, it is still necessary to clone the template warehouse from the warehouse, and perform certain manual configuration according to the project. In the current use, the problem is not big, but it is still cumbersome, and some configurations may be missed, and new projects cannot be used out of the box. Therefore, the development of scaffolding tools can quickly generate corresponding template projects.

3.2.1 CLI

General scaffolding tools are mainly divided into two parts: CLI and Template. The content of the scaffolding template is not placed in the same repository with the CLI, but in different repositories for management and iteration. Through separation, it can be ensured that the two parts are maintained independently and will not interfere with each other; template configuration or dependency update only needs to update the project template, without affecting the CLI part, resulting in re-package.

Referring to other scaffolding ideas, the template is released to the remote warehouse as an independent resource, and then downloaded through the CLI tool when running. After the interactive information of the CLI, the project template is rendered as the interactive input meta-information.

After the terminal executes egret-cli, the corresponding Egret project can be generated according to the interactive information.

3.2.2 Template

Since we need to respond to different needs and there are many business-related configurations, the template business configuration is quite different, and a unified template cannot be achieved for the time being. At the same time, in order to ensure that there is no redundant configuration in a template, we have made a distinction, mainly providing four project templates of base, standard, Shopee, and native.

When you need to do a small demo, you can directly use the base template, which is relatively simple; if you need to study business-related functions, you can choose the Shopee template; if it is a new project, you can directly use the native template.

3.3 Final Egret game development process

The Egret project is in line with the conventional web front-end engineering, which not only solves the development pain points and meets the engineering needs, but also allows us to officially enter the industrial age from the stone age. From development to deployment, there are good tools to assist execution and improve code quality. And development efficiency, new students can also start the project very well.

Mature web front-end engineering is not only beneficial to our business expansion, but also gives the project more possibilities. Some functions that are easy to implement in the front end but difficult to implement in the original game engine, such as dynamic logic code loading, multi-page applications, Egret+React mixed pages, etc., have also been well practiced in our project.

4. More R&D issues

4.1 Background of iOS Review Issues

In the early days of Shopee Games' launch, the number of users and visits was not large, and Apple did not focus on the HTML5 version of Shopee Games. However, with the continuous development of Shopee's business, the number of users and visits continued to increase. In early 2021, Apple raised an opinion on the HTML5 games embedded in Shopee, arguing that the Shopee App violated Clause 4.7 of the " App Store Review Guidelines ".

Apple's review clause reads as follows:

4.7.1 Software offered under this rule must:
- be free or purchased using in-app purchase;
- only use capabilities available in a standard WebKit view (e.g. it must open and run natively in Safari without modifications or additional software); and use WebKit and JavaScript Core to run third-party software and should not attempt to extend or expose native platform APIs to third-party software;
- be offered by developers that have joined the Apple Developer Program and signed the Apple Developer Program License Agreement;
- not provide access to real money gaming, lotteries, or charitable donations;
- adhere to the terms of these App Store Review Guidelines (e.g. do not include objectionable content); and
- not offer digital goods or services for sale.

In summary, it includes the following requirements:

• All free H5 games or use Apple Pay;
• Only use the native functions of WebKit, no extensions are allowed, such as JSBridge;
• H5 developers need to join the Apple Developer Program;
• Monetary gambling is not allowed, and content is subject to the general limitations of other censorship terms.

While Shopee Games is embedded in the Shopee App, it naturally needs to be deeply integrated with Shopee, which must involve JSBridge (such as logging in and jumping to e-commerce stores). In addition, Shopee Games also uses Shopee Coins as the currency in the game, and Shopee Coins are not obtained through Apple Pay, but are accumulated by users in e-commerce shopping. From these two perspectives, Shopee Games' use of HTML5 technology must violate the above Apple censorship provisions.

We refer to the official recommendation of the Egret engine, and change the technical architecture of Shopee Games on the iOS platform, from the original HTML5 to Native, using the Egret Native Runtime . Egret's official tool supports one-click generation of iOS projects and the release of the corresponding App installation package, which can meet the needs of independent games. However, Shopee Games needs to be embedded in the Shopee App. It is not an independent game App, so it is impossible to directly use the official one-click publishing mechanism. We need to further study the principle of Egret Native, so as to integrate with the Shopee App.

4.2 Egret Native Principle

Analyzing the iOS template project created by the Egret Native tool, you can find:

• Egret Native Runtime relies on a series of system libraries and the independently packaged libEgretNativeIOS.a. The main core logic and network functions are in this library. Since Egret Native is not open source, the specific implementation here cannot be known from the template project;
  
• The assets folder in the root directory of the App is fixed as the resource directory. The generated files of the H5 version need to be fixed here, and only one game is supported here;
• Through the libEgretNativeIOS library, the corresponding EgretNativeIOS instance and the corresponding view can be created.

Although Egret Native is not open source, according to the official documentation, combined with the debugging analysis of simulator breakpoints, we can still have further discoveries about Egret Native.

The Egret Native game architecture includes three layers: the front-end game layer, the Egret Native Runtime and the iOS Native layer.

• The front-end game layer is kept consistent with the file content of the H5 version;
• Egret Native Runtime is the core adaptation layer. It uses JSCore to parse the JS files in the game package, and builds JSBridge to realize the communication between JS and Native. The way of rendering at runtime is different from that of the Web. Egret Native has a JS Polyfill and JS Engine supplement for Native, which does not implement all browser functions in the Runtime layer;
• The iOS Native layer mainly manages the Egret Native Runtime life cycle and management views.

When starting the Egret Native game, first initialize the Egret Native instance from the iOS Native layer, create the corresponding game view, and mount it on the main interface. Then, Egret Native initializes the JS engine, binds JSBridge, reads the game resources of the front-end game layer, parses HTML and JS, calls the OpenGL interface, and finally displays the game screen.

4.3 Combining Egret Native and Shopee App

From the template project point of view, the way to embed the Egret Native game into the Shopee App is relatively clear. Add libEgretNativeIOS.a and other necessary dependency libraries to the Shopee App project project, and then store the game package in the assets directory. Finally, bind the necessary Shopee JSBridge for the game logic to implement operations such as login, payment, and jump, and the whole combination is complete.

However, in the final implementation process, some deep-seated problems are found, which need to be solved or avoided by means of the business side. These questions include:

1) Template project does not support multiple games

Shopee Games contains multiple games, and the above assets directory only supports storing assets of one game. And Egret Native Runtime, that is, the above-mentioned libEgretNativeIOS.a is not open source and cannot be modified.

Finally, in the official " Hot Update Solution Description ", we found that Egret Native can set the preload directory path, and the resources in the preload directory have higher priority than the assets directory.

Therefore, we can build multiple games in the Shopee App and store them in a separate directory. Before starting Egret Native, set the preload path to the path of the corresponding game. After Egret Native starts, it will first read the resources in the preload directory to start the game, ignoring the content of subsequent assets.

2) The network cache file increases infinitely

Egret Native Runtime caches network requests, but does not have a perfect cleanup mechanism, which causes the local cache directory to increase infinitely with the increase of game download resources. This part needs to be completed at the logical level of Shopee App iOS Native.

3) Some third-party libraries have naming conflicts

libEgretNativeIOS.a comes with some third-party libraries, such as SocketRocket, and Shopee App also introduces this library. Neither side has aliased this library, resulting in a naming conflict compilation failure. Since libEgretNativeIOS.a cannot be modified, SocketRocket can only be modified at the business level of Shopee App.

4) Egret Native has a memory leak

Since the Egret Native Runtime needs to be started and destroyed repeatedly in the Shopee App, as long as the Egret Native Runtime handles objects and textures slightly, it will cause memory leaks. For the Egret Native independent game, this problem does not exist, because every time the game is closed, the entire App is destroyed. This issue has been reported to the Egret official, but since the official team will not specifically deal with our usage scenarios, there is no progress at present. Fortunately, this part of the memory leak is small and does not pose a big problem for the time being.

By solving the above series of problems, we finally realized the combination of Egret Native and Shopee App, and can run multiple self-developed games on Shopee App. From the perspective of Apple's review, our method is separated from Webkit, and all code logic is built into the installation package submitted for review, which fully meets the requirements of the review terms. Therefore, after the program was launched, Shopee App and Shopee Games successfully passed the App Store review.

4.4 Future Planning

Although Egret Native has been integrated into the Shopee App, after six months of continuous operation, we found that Egret Native still has some problems:

• Runtime is not open source, there are unsolvable problems;
• Runtime only supports Egret games and cannot support other game engines. In particular, the Egret engine is not yet the best choice for 3D game development.

Therefore, we are planning a longer-term solution - the self-developed Native Runtime, which can support multiple H5 game engines at the same time.

This solution adopts an overall architecture similar to WeChat mini-games, but will be further encapsulated at the JS level to facilitate quick access by various game teams in Shopee. The technical summary is that Native exposes the WebGL standard-aligned interface and the necessary BOM interface to the JS side based on the JS engine, so that the ordinary H5 game engine can run seamlessly on the browser and our self-developed Runtime.

The benefits of this program are:

• Good compatibility, can support multiple game engines at the same time;
• Facilitate the transfer of stock H5 games;
• Can use the mature development tool chain of each engine.

However, there are corresponding disadvantages:

• Versions after iOS 12 are gradually dropping support for OpenGL;
• Due to compatibility with multiple engines, it is difficult to use new graphics and image standards such as Metal, WebGL2, Vulkan.

At present, iOS has been iterated to 15, and OpenGL and WebGL can still run smoothly. To support multiple H5 game engines, the WebGL standard cannot be bypassed, so these disadvantages have to be accepted for the time being. In addition, WeChat mini-games will also face the same problem. I believe that there will be an official Apple migration plan in the future, which may encapsulate an OpenGL-like interface in the upper layer of Metal.

At present, this scheme is still in pre-research and development, and it is expected to be implemented and fully used in 2022. After that, not only Shopee Games, but we hope to reuse this 3D/GPU rendering solution in more e-commerce scenarios to create more colorful interactive gameplay for users.

4. Summary

By comparing multiple H5 game engines, Shopee Games chose the Egret engine that is more suitable for business characteristics and team talent characteristics.

In the long-term business development and operation, in order to better support business needs, we have customized the Egret engine, including bug fixes and public library modifications.

While optimizing the engine, in order to keep pace with the official repository, we took advantage of the features of git's multiple remote repositories to achieve code merging in dual repositories.

Going a step further, in order to reuse the mature front-end Webpack construction system and CI/CD process, we developed Egret-Webpack-CLI, and changed the Egret game from the original stand-alone local packaging mode to the server Webpack packaging, which facilitates the reuse of a large number of An excellent front-end npm library for . The above innovations have brought significant improvements to the research and development of Shopee Games.

We encountered some difficulties in the iOS audit, and finally through the deep integration of Egret Native and Shopee App, we successfully realized the Nativeization of the game and passed the App Store audit.