Crossing Type Boundaries: Building a Unified Type System Between TS, JSON Schema and JS Runtime

Recently we open sourced Sunmao (mortise and tenon) , a framework for developing low-code tools. In Sunmao, in order to improve the development and use experience in multiple scenarios, we designed a type system that runs through TS (Typescript), JSON schema and JS (Javascript) runtime.

Why Sunmao needs a type system

First of all, let's introduce the two core designs in Sunmao:

• role division
• Scalability

Role division means that Sunmao divides users into two roles: component developers and application builders.

Component developers pay more attention to parts such as code quality, performance, and user experience, and use this as a standard to create reusable components. When component developers develop a new component in their own way, they can encapsulate the component as a Sunmao component and register it in the component library.

Application builders use existing components and implement application-related business logic. Combining components with Sunmao's platform features, application builders can do this more efficiently.

The roles are divided because applications are being developed all the time, but components are iterated much less frequently . Therefore, with the help of Sunmao, users can hand over the task of component development to a small number of senior front-end engineers or based on open source projects, while handing over the work of application building to junior front-end engineers, back-end engineers, and even people without code development experience Finish.

Extensibility means that most of the Sunmao component code is not maintained within Sunmao, but dynamically registered. This also requires that Sunmao's GUI editor can perceive the configurable content of each component and present a reasonable editor UI.

application metadata

It is not difficult to see that in order to meet the requirements of role division and scalability, we need to maintain a metadata between different roles for both parties to collaborate, and finally render the metadata as an application.

After the component developer implements the component code, the configurable part is defined as the format of the metadata, and the application builder configures the specific metadata according to the scenario.

Responsive State Management

On the other hand, in order to reduce the development difficulty of application builders, Sunmao has designed an efficient responsive state management mechanism , and we will share its design details in a separate article.

It can be simply understood that Sunmao allows each component to expose its own state to the outside world, and any other components can access the state and establish dependencies, and automatically re-render when the state changes.

For example, an input box with an id of demo_input exposes the state of 当前输入内容 , and another text component with an id of demo_text responsively displays the length of the current input content.

Type and development experience

Therefore, in order to improve the development experience of different roles in Sunmao, we have the following types of requirements:

• Application metadata is typed.

  • GUI editors can render a reasonable editor UI based on the metadata type.
  • Based on the metadata type, the specific values configured by the application builder can be verified.

• The state exposed by a component is typed.

  • App builders can get features like editor completion when using these states.

• The Sunmao Component SDK is typed.

  • After the component developer defines the metadata type, the type protection should be obtained when the component is actually developed using the SDK, so as to reduce the cost of connecting the component to Sunmao.

Considering portability, serialization capability and ecology, we finally chose to use JSON schema to describe metadata types.

A simplified input box component metadata is defined as follows:

 {
  "version": "demo/v1",
  "metadata": {
    "name": "input"
  },
  "spec": {
    "properties": {
      "type": "object",
      "properties": {
        "defaultValue": {
          "type": "string"
        },
        "size": {
          "type": "string",
          "enum": ["sm", "md", "lg"]
        }
      }
    },
    "state": {
      "type": "object",
      "properties": {
        "value": {
          "type": "string"
        }
      }
    }
  }
}

This piece of metadata describes:

• The input box accepts defaultValue and size two configurations, which are used to specify the initial value and size of the input box.
• The input box will expose the value state to the outside world, which is used to access the current input content of the input box.

The metadata definition based on JSON schema is enough for GUI editor to render UI according to it. The next goal is to reuse the JSON schema type definitions into the runtime of the Sunmao component SDK and editor.

Connect TS and JSON schema

In order to provide the best type experience, Sunmao's component SDK is developed based on TS and currently uses React as the UI framework.

In the metadata spec.properties that is the part configured by the application builder, it will be passed in in the form of React component props parameters to implement the logic of the component.

Usually, we will use TS to define the type of props, also taking the input box component as an example, the type of props is defined as follows.

 type InputProps = {
  defaultValue: string;
  size: "sm" | "md" | "lg";
};

function Input(props: InputProps) {
  // implement the component
}

At this time, the problem arises. As a component developer, you need to define both the JSON schema type and the TS type. Whether it is the initial definition or subsequent maintenance, it is an additional burden.

So we are looking for a way to enable component developers to define both JSON schema and TS types at the same time.

First, we use TS to implement a simple JSON schema builder, which only supports the construction of number type schemas:

 class TypeBuilder {
  public Number() {
    return this.Create({ type: "number" });
  }

  protected Create<T>(schema: T): T {
    return schema;
  }
}

const builder = new TypeBuilder();
const numberSchema = builder.Number(); // -> { "type": "number" }

As the name suggests, JSON schema exists in JSON form and is part of the runtime, while the TS type only exists in the compilation phase.

In this simple TypeBuilder we build the runtime object numberSchema with value { type: "number" } . The next goal is how to numberSchema this runtime object with the number type in TS.

Following the idea of establishing an association, we define a TS type TNumber :

 type TNumber = {
  static: number;
  type: "number";
};

In TNumber , there is a JSON schema structure of number type, and there is a static field that points to the number type in TS.

On this basis, optimize our TypeBuilder:

 class TypeBuilder {
  public Number(): TNumber {
    return this.Create({ type: "number" });
  }

  protected Create<T>(schema: Omit<T, "static">): T {
    return schema as any;
  }
}

const builder = new TypeBuilder();
const numberSchema = builder.Number(); // typeof numberSchema -> TNumber

The key trick here is the handling of return schema as any . In the call to this.Create , the static field is not actually passed in. But when calling Number , the generic type expected this.Create returns TNumber type, which contains the static field.

Normally, this.Create fails the type check, and the assertion of as any can fool the compiler into thinking that we returned a --- containing static TNumber type, but doesn't really introduce the extra static field at runtime.

At this point, the TS type of the runtime object numberSchema already points to TNumber, and TNumber['static'] points to the final expected number type.

So far, we have connected TS and JSON schema.

In order to simplify the use in the code, we can also implement a generic Static for obtaining the type of the runtime object constructed by TypeBuilder:

 type Static<T extends { static: unknown }> = T["static"];

type MySchema = Static<typeof numberSchema>; // -> number

Extending this technique to a JSON schema of type string also works:

 type TNumber = {
  static: number;
  type: "number";
};

+type TString = {
+  static: string;
+  type: "string";
+};

class TypeBuilder {
  public Number(): TNumber {
    return this.Create({ type: "number" });
  }

+ public String(): TString {
+   return this.Create({ type: "string" });
+ }

  protected Create<T>(schema: Omit<T, "static">): T {
    return schema as T;
  }
}

Of course, there are many details in the actual use process, for example, JSON schema supports configuration of many other additional information in addition to basic type information; and how the more complex JSON schema types AnyOf, OneOf, etc. are combined with TS types.

So in Sunmao, we finally use the more complete open source project typebox to implement TypeBuilder.

A more complex schema example:

 const inputSchema = Type.Object({
  defaultValue: Type.String(),
  size: Type.StringEnum(["sm", "md", "lg"]),
});
/* JSON schema
{
  "type": "object",
  "properties": {
    "defaultValue": {
      "type": "string"
    },
    "size": {
      "type": "string",
      "enum": ["sm", "md", "lg"]
    }
  }
}
*/

type InputProps = Static<typeof inputSchema>;
/* TS type
{
  defaultValue: string;
  size: "sm" | "md" | "lg";
};
*/

Infer types in the JS runtime

After implementing the combination of JSON schema and TS, we further thought about how to infer types in the editor's JS runtime to provide features such as auto-completion for application builders.

In the Sunmao editor, writing JS code is supported through a feature named 表达式 and has access to the reactive state of all components in the application.

The flexibility of expressions is that they support any valid JS syntax, such as writing more complex multi-line expressions:

<!-- prettier-ignore -->

 {{(() => {
  function response(value) {
    if (value === 'hello') {
      return 'world'
    }
    return value
  }
  const res = response(demo_input.value);
  return String(res);
})()}}

Before analyzing the JS runtime type inference method, let's show the application of type inference in expressions:

It can be clearly seen from the demo that for the variable returned by the function response res we have accurately inferred its type, thus further completing the method of the corresponding type variable.

It is worth noting that when response is passed in a variable of type string, the type of res is also inferred as string, and when the incoming value becomes a number, the inferred result of the return value is Also changed to number. This is consistent with the internal implementation logic of the response function.

But what the expression contains is just regular JS syntax, not TS code with the type, how does Sunmao infer the type from it? Actually we use the JS code analysis engine tern to achieve this.

Origin of Tern

The author of Tern, Marijn Haverbeke, is also the author of the widely used open source projects CodeMirror , Acorn and other projects in the front-end field.

Marijn created the need for "code completion" in the process of developing CodeMirror, a code editor based on the Web, and developed Tern to analyze JS code and infer types in the code, and finally implement Code completion.

In the process of developing Tern, Marijn also found that in the editor scene, the code is usually incomplete and the syntax is illegal, so he developed a JS parser: Acorn that can parse "illegal JS".

It is worth mentioning that the type inference algorithm implemented in Tern mainly refers to the paper "Fast and Precise Hybrid Type Inference for JavaScript" , the author of this paper is Brian Hackett, an engineer who was in charge of developing the Firefox browser JS engine SpiderMonkey at Mozilla at that time and Shu-yu Guo, the paper describes the type inference algorithm used by SpiderMonkey.

However, Marijn also introduced in his blog that Tern's scene is not the same as SpiderMonkey. Starting from the editor completion scenario, Tern can be more aggressive, using more approximations, sacrificing certain accuracy to provide better inference results or less performance overhead.

Type inference algorithm for Tern.

Tern builds the type graph corresponding to the code through static analysis of the code. Each node of Graph is a variable or expression in the program, and the currently inferred type; each edge is the propagation relationship between variables.

First understand type graph and propagation from a simple piece of code.

 const x = Math.E;
const y = x;

For this code, tern will build the type graph as shown below:

Math.E As a JS standard variable, it has been pre-defined as number type in tern, and the assignment of variables x and y generates the edge graph in the type graph , the type of Math.E is also propagated along the edge, and the type of x and y is propagated to number, completing the type inference.

If you modify the code slightly, the inference of tern may surprise you:

 const x = Math.E;
const y = x;
x = "hello";

When x is assigned to the string type again, the result of the variable y will not change accordingly (number is a basic type in JS, and there is no reference relationship). But in the type graph of tern, the action of assigning to x will add the string type to it, and propagate to y along the edge. Under the type inference of tern, x and y both have both string and number types .

This is obviously inconsistent with the actual code result ( x for string, y for number ), but this is tern to reduce type graph construction cost and algorithm An approximation of what logic does: ignore control flow, and assume that all actions in the program happen at the same point in time. And usually such an approximate inference method does not have much adverse effect on code completion scenarios.

There are also more complex type propagation scenarios in the code, which are more typical of function calls. Take another piece of code as an example:

 function foo(x, y) {
  return x + y;
}
function bar(a, b) {
  return foo(b, a);
}
const quux = bar("goodbye", "hello");

It can be seen that the type graph constructed according to tern can still be deduced that the type of quxx is string after multiple function calls.

For more complex scenarios, such as reverse inference, inheritance, type graph construction techniques for generic functions, etc., you can refer to the blog link above.

Using tern in Sunmao

Based on the type inference capability provided by tern, the code completion requirements of regular JS in Sunmao expressions can be solved. But the Sunmao expression mentioned above has access to the reactive state of all components. These states are automatically injected into the JS scope and do not exist in the code of the expression, so tern is not aware of their existence and type.

However, tern provides a definition mechanism to which variables and types that already exist in the environment can be declared. In Sunmao, the component defines the externally exposed state type through JSON schema, so we can automatically provide this part of the type declaration for tern through a conversion function between JSON schema and tern definition:

 function generateTypeDefFromJSONSchema(schema: JSONSchema7) {
  switch (schema.type) {
    case "array": {
      const arrayType = `[${Types.ARRAY}]`;
      return arrayType;
    }
    case "object": {
      const objType: Record<string, string | Record<string, unknown>> = {};
      const properties = schema.properties || {};
      Object.keys(properties).forEach((k) => {
        if (k in properties) {
          const nestSchema = properties[k];
          if (typeof nestSchema !== "boolean") {
            objType[k] = generateTypeDefFromJSONSchema(nestSchema);
          }
        }
      });
      return objType;
    }
    case "string":
      return "string";
    case "number":
    case "integer":
      return "number";
    case "boolean":
      return "bool";
    default:
      return "?";
  }
}

In some scenarios, the state JSON schema of the component is relatively loose, so we will also slightly modify the above method, read the type from the actual value of the state at runtime, dynamically generate the tern definition, and provide more type declaration information.

summary

Through the method in this article, we realize that only one type definition is maintained, and a unified type system is automatically built between TS, JSON schema and JS runtime, and the development experience of different roles in Sunmao is improved.

In the future, we will also introduce the related Sunmao function design, including

• How reactive state enables on-demand rendering
• How to develop an editor that supports hybrid highlighting and code completion after type inference
• ...

If you are interested, you can follow and participate in the Sunmao project in the open source community, and you are also welcome to send us your resume.