Game On Serverless: SAE helps Guangzhou Xiaomai to improve the performance of microservices research and development

Author: Luo Hao

Xiaomai was established in January 2015. It is a mobile Internet technology company dedicated to achieving high-quality self-growth of its business by taking digital leadership as its advantage. Always adhere to the user value as the center, data-driven, to develop a wealth of tool applications, casual games, puzzles, sports and other series of mobile applications for users. More than 400 products have been developed in total, and the cumulative number of downloads and installations by users has exceeded 700 million. In the next three years, Xiaomai takes the vision and mission of becoming the world's leading developer growth service platform, and hopes to provide developers with full-link solutions through standardized products and services, and to escort all-round technology + services. , Combustion products continue to grow, helping developers of tools and casual games to increase the success rate of their products.

In Xiaomai, a flat management style is implemented, and each business team can independently choose to adopt a more suitable technology stack and infrastructure. Therefore, there are three different computing platforms: ECS, K8s, and SAE (Serverless Application Engine). The situation of coexistence, and they are all running micro-service architecture. Different computing platforms have their own unique advantages and values, but also face their own challenges. At present, it is mainly the game team that uses the SAE platform.

Why choose SAE

For most casual games, first of all, the game itself has its own life cycle, and during the life cycle, the game itself will have very large crests and valleys. For example, the traffic during the day is much larger than that at night, and the traffic during the day is concentrated at several points in time, and 8 o'clock in the evening is the peak of the business, and there is almost no traffic from 2 to 6 in the morning, but the service cannot be stopped. In addition, when the game was first launched, every operation activity would attract a large influx of new customers, and back-end services were required to quickly respond to changes in traffic. Therefore, the business side expected an automatic and elastic computing platform. Secondly, most casual games are stateless, and can be split into different service modules to improve service performance and quality, such as chat, red envelopes, backpacks, upgrades, user data acquisition, video processing, advertising, etc. It can be deployed using a microservice architecture. Finally, during the game's launch period, many new functional modules will be added iteratively, requiring frequent releases and upgrades. Therefore, when selecting models, the business side will comprehensively consider:

1. System stability and disaster tolerance
2. The platform's automatic elastic scalability
3. Support for microservice architecture
4. Convenient release and rollback capabilities, even non-stop server upgrades

These capabilities can actually be achieved through ECS or K8s self-built, but it will bring a lot of operation and maintenance costs to the business team, and it is difficult to balance the cost of investment. Especially in terms of flexibility, the efficiency of self-built elasticity is difficult to meet the rapid changes in traffic, and a lot of redundant resources are often required. And SAE can meet the above 4 requirements very well. In fact, Xiaomai’s game team started to pay attention to the trial of SAE as early as during the SAE public beta. Up to now, more than 50 services and applications have been deployed on SAE, involving more than a dozen games, such as falling in love with guessing idioms, idioms are the strongest answering people, I find faults fast, answer a lot, happy to find faults, how short Videos, etc. If you are interested, you can download the APP and try it out.

SAE landing practice

The serverless application engine SAE is positioned as a one-stop application hosting platform on top of containers. Its core value is to provide users with a K8s operating environment with full application lifecycle management, microservice governance, and flexible operation and maintenance-free. In essence, the user's code will eventually run in the container, but the container does not need to be maintained and managed. Therefore, for existing game services, it can be directly migrated and deployed to SAE without modification. Moreover, SAE also provides a direct deployment mode of JAR packages for JAVA applications, which eliminates the step of mirroring by Xiaomai. It is very close to the original ECS mode, but the experience will be simpler. The approximate comparison is as follows:

The core capabilities of SAE are high availability and automatic flexibility. For Xiaomai's game team, when deploying JAR packages, you can check multi-availability zones to achieve cross-availability zone disaster recovery. The bottom layer of SAE actually provides multiple K8s clusters distributed in different availability zones, and container instances carrying services can be automatically scheduled in multiple availability zones. The flexible configuration is also very simple. It can be set based on CPU, memory, QPS, RT and other indicators. For Xiaomai's online games, the expansion and contraction are mainly triggered by the CPU and memory usage, and can also be specified. The maximum number of instances and the minimum number of instances are very convenient. Moreover, the current timing flexibility and monitoring index flexibility can also be mixed, so when there are operational activities, the two flexible methods can be used together to ensure resource flexibility. But what needs to be noted here is that the threshold of the monitoring index needs to be configured according to the actual situation of the business. It is recommended to clarify through pressure testing before going online.

In addition, through application monitoring, it is also very convenient to view the invocation of the service interface. These capabilities have been integrated into the SAE platform by default, which is very helpful for business troubleshooting.

Finally, in Xiaomai’s game team, Spring Cloud and Dubbo technology stacks are mainly used, so support for microservice governance capabilities is also very necessary. At present, on the SAE console, you can directly configure microservice health checks, elegant offline scripts, configuration management, grayscale release of microservices, and one-click rollback. But in the process of actual use, I have stepped on some pits. For example, when the service is released, the health check sometimes times out and causes the instance to restart continuously, because sometimes the service loads a large amount of data and class libraries, which takes time to start. . Increasing the timeout period of the health check can reduce the probability of occurrence, but the release time will be lengthened. And when the service just started, the initial response was relatively slow. In fact, the service is not fully ready. Here we rely on SAE's ability to provide microservices to go online gracefully, which can ensure the normal online service of the service. In addition, for the distribution department, in order to avoid the load traffic suddenly hitting the new instance, it is more recommended to use the gray scale capability of the microservice traffic percentage. After a period of practice, the final business structure is roughly as follows:

Xiaomai’s game team basically only pays attention to business logic, and the resource level is hosted on the SAE platform, which greatly simplifies the complexity of operation and maintenance. In addition, in order to cope with the rapid iteration of the business, Xiaomai also used Jenkins to encapsulate the API interface of SAE, realized the CI/CD capability, and greatly accelerated the online speed of the service. Compared with the original elastic efficiency and deployment efficiency, the overall R&D efficiency has been greatly improved. The elastic speed has been shortened from minute to minute, and the launch speed of new projects has been shortened from day to minute.

Summary and outlook

1. SAE provides a serverless operating platform in the field of microservices, providing users with new options for reducing costs and increasing efficiency. In addition, the bottom layer of SAE uses a hosted K8s cluster, which also provides the easiest way for users to transform into containers.

2. The addition of SAE in application management and microservice governance makes SAE a one-stop application PAAS platform different from container services, allowing users to focus on business iteration.

3. For application management, SAE also provides the environment "one-click start and stop" function. For example, for the development and test environment, you can set the timing to close and open to optimize the resource occupation of the non-online environment, which can help Xiaomai further optimize the cost.

4. For JAVA applications, SAE provides the DragonWell JDK version, which can accelerate the startup speed of JAVA applications and the consumption of thread resources, and the startup speed can save about 40% of the time consumption.

5. In the future, SAE will continue to improve elastic efficiency and strengthen functional iterations at the application management level, hoping to bring more value-added experience to users. For example, the newly released ZIP package deployment capabilities that support PHP can simplify the complexity of web applications on the cloud. Spend.

For students interested Serverless, you can also click here View more case studies and articles.