Automatic code generation scheme based on AST

Recently, a request was received, and the corresponding JS SDK file needs to be defined through the d.ts file provided by a third party. The form is as follows:

The d.ts file provided by the third party:

export class SDK {
  start(account: string);
  close();
  init(id: string): Promise<{ result: number; }>
}

The defined JS SDK file:

 // 初始化 wrapper 对象，省略了细节
const wrapper = (wrap) => wrap;

// 定义 JS SDK
const SDK = {
  async start({ account }) {
    return await wrapper.start(account)
  },
  async close() {
    return await wrapper.close(account)
  },
  async init({ id}) {
    return await wrapper.init(id)
  },
}

export default SDK;

At the beginning of the project, we manually wrote the JS SDK based on the d.ts file provided by the third party. Because this d.ts changes frequently, we need to keep synchronizing the JS SDK; at the same time, because our project is maintained by multiple people, the handwritten JS SDK will inevitably have many conflicts. These problems are not good for the efficiency of research and development. of.

By analyzing d.ts and its corresponding JS SDK, it can be seen that their format is basically fixed, and there is also a very clear correspondence between the two. So we can think about whether we can generate the corresponding JS SDK directly from d.ts through an automated method?

The relatively simple idea is to analyze the d.ts code line by line, and use regular methods to match keywords to obtain key information. This method is simple and rude but not elegant enough. It requires very complex matching rules to meet the needs. Once the d.ts format changes, the original matching rules may be directly unusable and the maintenance cost is too high.

To avoid a series of problems caused by format changes, "abstraction" can be said to be a relatively more appropriate solution. The AST of the code is an abstract way, which can effectively avoid the impact of changes in format and writing, and convert the source code into a tree structure data that can be easily read by scripts to facilitate subsequent operations.

AST analysis of d.ts

Since d.ts is also a typescript file, we can use the official API provided by typescript to generate the corresponding AST:

// https://ts-ast-viewer.com/
const dTsFile = fs.readFileSync(resolve(__dirname, filePath), 'utf-8')

const sourceFile= ts.createSourceFile(
  'sdk.ts',                       // 自定义一个文件名
  dTsFile,                        // 源码
  ts.ScriptTarget.Latest          // 编译的版本
)

We can also use https://ts-ast-viewer.com to check whether the generated sourceFile (AST) meets expectations:

With AST, the next step is to analyze what information we need in it. From the previous example of d.ts to JS SDK, we can see that the most important thing is to know two things in d.ts:

1. What methods are defined;
2. What parameters are passed in the method.

According to AST, ClassDeclaration located under MethodDeclaration is a series of methods defined by the d.ts; and MethodDeclaration in Parameter defines the parameters of the method.

Next, is it necessary to read the node information of the AST, and then directly generate the JS SDK? the answer is negative. The reason is that if the logic of "Analyze AST" and "Generate JS SDK" are coupled together, due to the large number and rich types of AST nodes, a large number of conditional judgments may be required, and the final logic will be very confusing. This feeling of "seeing a little and doing a little" is no different from the idea of reading d.ts line by line and then generating JS SDK.

In order to avoid the difficult maintenance problem caused by this kind of excessive coupling, we can introduce a "domain-specific language (DSL)".

Use DSL to generate JS SDK

For the definition of DSL, please refer to this article "Domain Specific Languages (DSL) What Developers Need to Know" . The definition of DSL sounds like a powerful one, but to put it bluntly, it means to define a transitional format that can link the previous and the next.

In our scenario, we can define a DSL in JSON format to record the key information extracted from the AST, and then generate the required JS SDK files from this DSL. This method seems to be an extra step and increase the workload, but in actual use, you will find that it is very helpful for logical decoupling, and it is also a great benefit for subsequent maintenance.

For our example:

export class SDK {
  start(account: string);
  close();
  init(id: string): Promise<{ result: number; }>
}

By analyzing its AST, it can be sorted into such a DSL:

const DSL = [{
  name: 'start',
  parameters: [{
    name: 'account',
    type: 'string'
  }]
}, {
  name: 'close',
  parameters: []
}, {
  name: 'init',
  parameters: [{
    name: 'id',
    type: 'string'
  }]
}]

The name and parameters of the method are clearly recorded in the DSL. If necessary, you can easily add more information, such as the type of return value, and so on.

The next step is to analyze the format of the JS SDK:

const wrapper = (wrap) => wrap;

// 定义 JS SDK
const SDK = {
  async start({ account }) {
    return await wrapper.start(account)
  },
  async close() {
    return await wrapper.close(account)
  },
  async init({ id}) {
    return await wrapper.init(id)
  },
}

export default SDK;

Since the format is also fixed, you only need to prepare a string template, then traverse the DSL, and fill in the organized method into the template:

const apiArrStr = DSL.map(api => {
  // 伪代码，省略了信息提取的步骤
  return `
  async ${name}(${params}) { return await wrapper.${name}(${params}) }
  `
})

const template = `
const SDK = {
  ${apiArrStr}
}

export default SDK;
`

return template;

summary

This article introduces the method of analyzing d.ts code through AST, and then automatically generating the corresponding JS SDK. At the same time, it introduces the concept of DSL to further solve the problem of logical coupling, hoping to inspire readers.