DingTalk Collaborative Engine and Application Scenario Technology Exploration

Author: Sun Ran (cooked shrimp)

The collaborative engine is DingTalk's runtime and development framework for cross-platform terminals and for solving distributed data consistency.

It mainly solves three types of scene problems:

• Multiplayer real-time collaboration
• Cross-end synchronization and application state relay
• Available Offline Local First

At the same time, it is also open oriented. One-party products can be used, and third-party developers (mini programs, H5 applications) can also be used.

Collaborative Engine Demo Show

Multi-person real-time collaboration: label together

Starting from DingTalk client 6.0.0, the "mark together" function has been added to the pictures in the message conversation. Group members can sketch a picture at the same time, and everyone can see other people's handwriting in real time:

Here, the collaboration engine SDK provides a set of CoCanvas (collaborative drawing board) components, and picture messages quickly assemble multi-person collaboration capabilities by referencing the CoCanvas components. In addition to solving the collaboration problem of multi-person real-time data (handwriting data), the collaboration engine also provides the information of the currently collaborating participants, which is used for the business to realize the function of displaying information such as "how many people are annotating".

Sync Across: Chat Drafts

Have you ever encountered such a scene: you have edited half of the things on your mobile phone, and you want to continue editing when you go back to the computer? Or, take a screenshot on your phone and continue editing on your computer. Now it seems that you have to send the message or picture to yourself on your phone first, and then download it on your computer to continue. The whole process will be interrupted by the cross-end transfer process.

If you use a collaborative engine, the whole process may become like this - you can achieve end-to-end input synchronization:

Or copy on one end and paste directly on the other:

Realize the seamless connection of a series of coherent operations on the mobile phone and the computer!

In the above example, the collaborative engine provides basic data structures such as CoString, CoMap, and CoList, as well as components such as CoImage and CoClipBoard, so that upper-layer business sharing data is as simple as reading and writing local variables. The business only needs to care about what data the business itself needs to process, and does not need to care about what additional cross-end needs to do. In the future, the collaborative engine can also continue to provide high-level components such as CoCamera and CoAlbum, so that desktop applications can realize more advanced functions such as pulling up the mobile phone album to select pictures, and pulling up the mobile phone camera to take pictures.

Cross-end synchronization is to make multiple physical devices of the same user into a unified logical device, sharing data, sharing system devices, and sharing application status. Since the collaborative engine itself already has the most basic data synchronization function, it can easily support these demand scenarios.

Facing Openness: How Three-Party Applications Achieve Cross-Terminal Relay

As a cross-platform underlying engine, the collaborative engine also empowers third-party applications to help achieve:

• One-time development and multi-terminal operation: as a carrier for cross-terminal collaboration of three-party applications, develop a set of small programs with code that runs on multiple terminals (mobile/desktop).
• Desktop-side large-screen productivity: The desktop-side applet is automatically adapted to the large-screen mode, giving full play to the differentiated large-screen productivity advantages of the desktop side
• Multi-person collaboration: based on the collaboration engine, realize multi-person task collaboration
• Cross-terminal relay: Based on the collaborative engine, cross-terminal application relay is realized: the application operation status on the mobile phone is automatically relayed to the desktop large screen to open
• Available offline: Local First. Based on the collaborative engine, the data is local, and the task can be submitted offline. The collaborative engine will ensure the final success of the task
• Backend As A Service: All data (synchronization, collaboration, relay) is based on the collaborative engine, developers only need to write front-end code, no need to develop server-side

What does the collaborative engine do here?

• As the applet data layer, Backend As A Service
• Real-time collaboration of data for multi-person editing operations
• Cross-end data synchronization, application state migration relay

Introduction to Collaborative Engine

The collaborative engine is [for cross-platform terminals] [solving distributed data consistency] [Runtime and development framework].

solved problem

The collaborative engine mainly solves three types of scene problems:

• Multiplayer real-time collaboration
• Cross-end synchronization and application state relay
• Available Offline Local First

The above three types of problems, even if the collaborative engine is not used, there are other implementations. However, for the business side, these implementations have to deal with many underlying engineering problems and even algorithmic problems in the "collaboration" field, such as network processing, local data storage, long-term offline processing, multi-person real-time operation merging, rollback and conflict resolution, etc. .

As a development framework, the collaborative engine provides general collaborative capabilities and solves the above problems by accessing collaborative variables/components:

• Use data declaratively. The business side directly uses collaborative variables, just like using local variables, and does not need to pay attention to the underlying operation merging, rollback, conflict resolution and other issues.
• Cross-platform implementation, all platforms are available. Let all applications (Android/iOS/Windows/Mac/external web) and various forms (Native/H5/applet) use the collaboration capability.

API and basic concepts

The collaborative engine provides a set of collaborative data structures and APIs in the upper-level development framework. They are similar to normal container variables, but come with their own synergy capabilities. For example, a cloud-coordinated counter can be resolved through the CoCounter co-variable like this:

 Container container = Loader.getContainer(url); CoCounter counter = container.getCoCounter("my_counter"); // 点击+1按钮时counter.add(); // 监听数据变化counter.addEventListener(new CoCounterValueChangedListener() { @Override public void onValueChanged(int newValue) { // 更新UI updateView(newValue); } });

Here, we propose several basic concepts

CDS

Collaborative Data Structure, collaborative data structure. A CDS object is a collaborative data variable.

CDS also has different types, such as CoString, CoBoolean, CoMap, and CoList. As the name implies, they correspond to the basic data types and collection types commonly used in programming languages.

Of course, there are more "complex" data types, such as CoText (text type that supports OT), CoPixel (graphic annotation type).

Container

A collection of CDS objects. CDS objects in the same Container can refer to each other, for example, an element in CoList is another CoString object in Container.

Developers only need to develop business with collaborative variables just like using their familiar data structures, and then they can easily realize data collaboration.

Internal Architecture 1 Client Collaboration Engine The client is mainly divided into the following layers from top to bottom: Platform: Each platform provides an API layer for developers API: Cross-platform glue layer Loader: A singleton module responsible for loading Containers Container: Collaboration engine The core of the cross-platform module, responsible for CDS management and the necessary processes when each engine is running. Service: The platform layer capability interface that the collaborative engine depends on Provider: The capability of each platform to implement the Service layer interface 2 Server\
The responsibilities of the collaborative engine server are relatively small, mainly responsible for connection and session management, message broadcasting, sequencing and storage, and there are also operation changes.

More use cases

In addition to the application scenarios in the Demo Show, there are many more possibilities...

More "Together XXX"

In addition to the existing "marking together", there are many more scenes of "XXX together" such as "playing together", "listening together", "watching together" and so on. In these scenarios, the problems related to data collaboration encountered by developers will have many intersections, and they can all be solved by a set of collaboration engines.

Weak network offline optimization

This topic seems to have little to do with collaboration, but the Local First attribute that comes with the collaboration engine has guaranteed usability in weak networks and offline. Many of the things that need to be done for offline optimization of weak nets, the collaborative engine has already done.

So how does the collaborative engine make the weak network available offline?

Offline availability

The collaborative engine contains a cache queue - PendingQueue, which is responsible for caching all unconfirmed local operations (ops) on the server side. When a business operates a collaborative data structure, an operation will be generated, and the collaborative engine will first cache it in the PendingQueue and store it. The operations in the PendingQueue will be sent in a first-come-first-served order, and the next one will not be sent until the operation is confirmed by the server. The PendingQueue is also restored from the local database each time the co-engine is initialized. Therefore, in the collaborative engine, the generation and recovery of data all go through the process of first storing the data locally, then sending it and waiting for the server to confirm it. This is the so-called Local First, which guarantees the offline availability of the collaborative engine.

Weak network availability

First, the collaboration engine is aware of network connectivity. When it is clear that the network is not connected, the sending operation of the PendingQueue will be suspended; after the network is connected, the sending mechanism will be automatically enabled.

Secondly, when the network is connected, for whatever reason, as long as the sent operation does not receive confirmation from the server within a certain period of time (such as within 5 seconds), the collaborative engine will resend the operation until the server confirms it. This ensures the availability of the collaborative engine under weak networks.

Compared

The following table compares the work related to offline optimization of weak network with some existing mechanisms in the collaborative engine:

It can be seen that most of the things that need to be done in the offline optimization of weak networks, the collaborative engine already has the corresponding capabilities. The initial design of the collaborative engine has met the main requirements for offline availability of weak networks, and ensures that operations will never be lost, and can also be used in related work of offline optimization of weak networks.

Summarize

This article introduces the collaborative engine framework, which is dedicated to solving the problem of multi-person and multi-terminal data collaboration, allowing businesses to easily realize the function of multi-person collaboration. We listed several intuitive demos, and then discussed some other application scenarios. In addition to being applicable to various "together XXX" services, it can also be used to solve cross-end data relay problems and offline optimization of weak networks.

Pay attention to [Alibaba Mobile Technology] WeChat public account, 3 mobile technology practices & dry goods every week for you to think about!