What is the back-end architecture of the 131 million monthly active game?

131 million people, what is this concept?

If it is a country, then the "country" happens to have the tenth largest population in the world.

What about the number of players in a game? That's it: Minecraft!

As an explosive game that has been popular all over the world for many years and has 131 million monthly active users, “Minecraft” not only can be played on a single player, but also contains a wealth of online elements. How powerful is the backend of this game to provide online services for such a large player base?

In fact, this is also a feat that can only be achieved by constantly exploring and chasing the limit.

Azure Migration Tour for Minecraft Realms

Minecraft "Minecraft" is a game that I believe everyone is familiar with. As a construction-themed sandbox game, players can use a large number of "blocks" with different appearances and functions as materials to build wonderful and interesting worlds, and even use survival, creation, adventure, extreme, spectator and other modes on this basis. Have more fun.

Minecraft Realms is an online platform provided by the game, where players can create personal servers and invite friends to play together. It can be said that most of the online online experience of the game "Minecraft" is achieved through Minecraft Realms.

As an explosive game with a total global sales of more than 200 million copies and 131 million monthly active users, its back-end system can be described as huge. The entire Realms system consists of tens of thousands of cloud-hosted multiplayer game servers, computing devices running service code, subscription databases and game world state data storage containers, all originally hosted on AWS.

In order to obtain more cost-effective , and Microsoft ecosystem to achieve better interoperability , and further reduce the delay to improve the game play experience , game development team Mojang plans to migrate the whole background to Azure. The entire migration process is divided into two stages.

Phase 1: Migrating cloud-hosted multi-player servers

Minecraft Realms hosts tens of thousands of multiplayer servers in the cloud. These servers are dynamically allocated when users connect to their Realm. At a given time, each Realm runs on a virtual server that is allocated and reclaimed as demand increases and decreases.

In Phase 1, Realm needs to be migrated from AWS to the Azure PlayFab multi-player hosting service. Azure PlayFab provides the support needed to build and operate online games at scale. In particular, this is a game-tailored service that provides global low-latency multi-player hosting services and remote content management capabilities (for updating game assets, managing online payments, and enabling LiveOps), allowing game Runs as a service and performs real-time analysis of game performance. After successfully migrating to PlayFab Multiplayer Servers, the Realms team hardly needs to focus on the management of virtual machines, and can spend more time improving the gaming experience provided by their products to players.

At this time, the multi-player server part of the entire background has been running in Azure, while the rest of the storage and database systems are still in AWS. The general architecture is shown in the following figure:

Phase Two: Migrating Workloads

Subsequently, Mojang smoothly migrated the database, storage and computing equipment from AWS to Azure as planned.

database

The Minecraft Realms service involves a lot of business-critical data, including game world state information, subscription metadata, and telemetry data. This aspect involves a total of three pre-production environments and two production environments, each of which contains a maximum of 3 databases, and this migration needs to be distributed in 13 databases with a total of 1TB of data (the largest database contains 450GB of data ), move from AWS to Azure Database for MySQL in a reliable manner. This mission-critical operation is further complicated by the fact that about 6 thousand service requests per second (RPS) are generated during the migration.

A range of utilities provided by Azure, such as Azure Database Migration Service (DMS), and expert support from the Azure FastTrack team are critical to a successful migration. Mojang also uses the Azure Database Migration Service tool, which simplifies the process of migrating data, schemas and objects from source locations to Azure. The Azure FastTrack team has tailored guides for Mojang to further accelerate, simplify and optimize Azure deployments.

The migration copy of the database lasted about a week. The largest database table (over 1 billion rows!) took only 2 days to load the initial data. Even though the service needs to handle 6k RPS, none of the replication latency is more than a few seconds. During this process, Mojang migrates a new environment from AWS to Azure MySQL every week, and schedules a small downtime period (about 20 minutes for most environments), during which the Azure DMS migration is successfully completed. After the deployment is complete and the environment is back online, the database is successfully hosted on Azure!

After all databases are successfully migrated, the general architecture of the back-end system is shown in the following figure:

storage

The essence of Minecraft Realms' rich gameplay lies in the construction of beautiful virtual worlds where players can gather together in a private space to play together. Realms has saved about 20PB of virtual world game data.

Mojang plans to migrate all world data to Azure Blob storage with as little disruption to gameplay as possible. This process will cause gameplay to experience downtime, as players will not be able to access their Realm while data is being moved. To avoid downtime, Mojang employs a strategy of reading existing world data from AWS when an existing active Realm world is accessed for the first time, then writing all updates to Azure Blob storage. When accessing this data in the future, it will only be accessed through Azure Blob.

So all active Realms will be migrated while they are running. Players will experience imperceptible (millisecond) delays only when launching their Realm for the first time through Azure. In the end, you only need to batch copy the remaining inactive world data (referring to the data that has not been activated in the last year, about 300TB).

The general process of storage migration is shown in the following figure:

calculate

The Realms service code handles all Realms requests from Minecraft clients by performing critical tasks such as logging metadata updates, handling purchases within Realms, provisioning/revoking game servers. The entire Realms service code was initially hosted on AWS, hosting most of the game's core services through more than 60 virtual machines.

To ensure players can update seamlessly, Mojang employs a blue-green deployment strategy (pictured below). One version will be deployed to both Azure and AWS, and user traffic will be switched between the two locations by means of DNS updates. This allows each environment to be fully created and tested before directing test and customer traffic entirely to Azure. Since deployments on AWS and Azure run concurrently, any issues encountered with user traffic can be quickly reversed and fixed. This enables fast and safe rollbacks, minimizes downtime, and enables robust validation.

Summarize

So far, Mojang has successfully migrated the entire Realms service, including front-end servers (computing), virtual world data (storage), and management of all databases and the game server itself, to Azure. This not only helps them improve the efficiency of Realms' core business, but also improves the gameplay experience for Realms players.

With the powerful Azure toolset, Mojang also reduces service maintenance costs - smaller code size, simpler deployment, and higher levels of abstraction. At the same time, the game experience is further improved by utilizing the global coverage of Azure services.

Although large-scale migration of business-critical systems might seem difficult, but the end goal, the establishment of reliable migration planning, and related components disassembled , doing so can greatly simplify the whole process. At the same time, Azure will also provide the necessary technical support and a series of tools and service support for related migration projects.

Do you want to experience building a game backend system based on Azure, or migrate existing systems on other platforms to Azure? Welcome to scan the QR code , sign up for an Azure trial account immediately, and experience an unusual game operation and maintenance experience.