Realtime DB technology detailed

1. Overview of Realtime DB

1.1 Introduction to Realtime DB

Realtime DB is a database hosted in the cloud. Data is stored in JSON format and synchronized to each client connected in real time. Has the following characteristics:

• Instead of using a common HTTP request, it uses a data synchronization mechanism. Whenever the data changes, any connected device will receive the update in real time
• Provide a flexible expression-based rule language, user-defined data structure and when data can be read or written
• The NoSQL database based on MongoDB, therefore, has an optimization direction and functional characteristics that are different from relational databases. The design of the server-side API only supports operations that can be executed quickly, so users need to carefully consider the stored data structure.

1.2 Background and application scenarios of Realtime DB

The development of a cloud native APP based on BaaS requires a set of database middleware for information storage, instant synchronization, atomic modification, and offline caching, to remotely modify the serverless database to achieve the purpose of breaking away from the server interface. In order to meet the above requirements, a middleware such as Realtime DB is designed and implemented. Usually used in the following scenarios:

• instant messaging
• State synchronization
• Real-time dynamic
• team cooperation
• other

2. Development History of Realtime DB Technology

• Local DB: Data local persistence Advantages: stable and reliable, local persistence Disadvantages: data cannot be shared across multiple terminals, low playability
• Server Interface: Data persistence through server interface communication Advantages: data can be shared across multiple terminals, cloud controllable Disadvantages: interfaces need to be customized, usually requiring client + server + operation and maintenance intervention, unable to automatically scale and expand, more dependent Network availability, etc.
• Remote DB: remote real-time database, data cloud persistence Advantages: multi-terminal sharing of data, real-time synchronization, high playability, serverless, free customization of the data structure on the terminal side, no need to deploy and maintain the server Disadvantages: more dependent on network availability

3. Realtime DB research status

3.1 Industry analysis

Firebase is a real-time back-end database startup that was acquired by Google in 2014. Users can use Firebase more conveniently while combining Google's cloud services. FireBase currently proposes two cloud database solutions that support real-time data synchronization and can be accessed through the client, the real-time database and Cloud Firestore.

product	principle	advantage	Disadvantage
Real-time database	Store data as a large JSON tree	• Simple data is easy to store • Support online status detection	• Complicated hierarchical data is difficult to organize on a large scale • Query and sort functions are weak
Cloud Firestore	Store data as a collection of documents	• Simple data is easily stored in a document, very similar to JSON • By using sub-collections in the document, complex hierarchical data is easier to organize on a large scale • There is less need for denormalization and data flattening	More complicated to use

3.2 Functional requirements and challenges

Realtime DB is initially planned to meet the following requirements,

• Support for adding, deleting, modifying, and checking sentences
• Multi-terminal real-time data synchronization support
• Public cloud or private cloud implementation
• Offline cache support
• Data auto-scaling and expansion, serverless
• Data security means
• Custom add, delete, check and modify protocol, expandable functions later

Preliminary selection of technical solutions according to requirements,
1. Cloud database selection to achieve Serverless
Relational database VS non-relational database, based on the company's self-developed serverless-level cloud MongoDB, to achieve server-side cloud database storage

2. Multi-terminal real-time synchronization solution
NoSQL database is used, and the data is stored in a multi-fork tree (KV) format, that is, JSON. There is no need to worry about node data merging. The server can overwrite data according to the operation time

3. Data security means
Data nodes define read and write permissions, cloud empowerment [According to account], generate permission configuration table

4. Transaction support and atomic operations
Customized remote database operation protocol, pre-reserved operator ".", used for later implementation of atomic operations and other instructions

5. Database real-time guarantee
Adopt long connection, based on MDP self-developed long connection SDK-TapConnect, realize stable sending and receiving, millisecond response

3.3 Solution

The client-side architecture design is roughly divided into the following 4 layers

Flatten data and simplify JSON

Basic data type of leaf node: Boolean/String/Long/Double
Key value restriction: must not contain:'/''.''#''$''['']', reserved related operators, used to achieve atomic operations, etc.
Value limit: cannot be:'NaN''Inf''-Inf'

Client-Server communication model

Multi-terminal synchronization scheme

Multi-terminal adaptation scheme
The development of Realtime DB needs to be adapted to Android, iOS, Flutter and other platforms

Option 1: Develop Realtime DB SDK separately for Android, iOS, flutter, etc. Advantages: native-level performance
Disadvantages: high maintenance costs

Option 2 Develop a Native version, provide so, and use it on each side. Advantages: Develop a set of code that can be used by multiple terminals
Disadvantages: The performance problem is more obvious, there is an additional layer of native communication, and it is inconvenient to use, and each end needs to be API adapted

Solution 3 Use the Kotlin Multiplatform framework for development. Advantages: a set of code, run multi-terminal, and the code is compiled and processed by the framework, automatically compile multi-terminal products, naturally supports native-level performance, adopts Kotlin, and is easy to write
Disadvantages: The current framework is an alpha version, and there are unknown risks

Write once, adapt to multiple ends
Common Code: Use Kotlin to implement common logic and rely on the official existing multiplatform support library to implement network, io, database and other operations

iOS Code: You can rely on the existing iOS native libraries through CocoaPods. The calling principle: During the framework compilation, the corresponding Kt class functions are automatically generated according to the external interface of Objective-C/Swift, and external calls are provided.

Android Code: can rely on the existing jar/aar library through gradle, calling principle: Kotlin and java interoperability

If there is platform-differentiated code that cannot be directly implemented in Common Code, you can define expect interface class A in Common Code, and implement actual class A in iOS Code and Android Code respectively.

3.4 Future Outlook of Realtime DB

Build the Realtime DB SDK into a middleware covering iOS, Android, Flutter, and even JS, and use the Kotlin Multiplatform Mobile framework to achieve a set of codes that can be run everywhere, bringing continuous gains in the later stage.

4. Summary

Realtime DB allows direct and secure access to the database from the client code. Thanks to this, it is possible to build a feature-rich collaborative application. The data will be permanently retained on the client. Even if it is offline, real-time events will continue to be triggered. Provide end users with a good instant experience. Realtime DB is still in the R&D stage. As for the shortcomings of the solution selection and implementation, please communicate or put forward optimization opinions or suggestions.

About the Author

Yijun OPPO Application Engineer
Focus on mobile platform development, real-time database, Severless, Quic and other technologies.

For more exciting content, please scan the QR code to follow the [OPPO Internet Technology] Official Account