Visual low-code system practice based on design draft identification

本文首发于「Shopee技术团队」公众号，探索更多Shopee技术实践

This article is based on a practical summary of the Shopee Supply Chain WMS (Warehouse Management System, Warehouse Management System) team using the front-end low-code system to reduce costs and increase efficiency.

 目录
1. 项目背景
    当前开发模式下的痛点
2. 思考破局
    业界方案对比
3. ASLINE 方案设计
    3.1 核心设计思想
    3.2 核心架构
4. 核心难点及解决方案
    4.1 设计稿组件智能识别
    4.2 布局转换
    4.3 接口字段与组件字段的匹配
5. 系统展示
6. 落地情况及后续计划

1. Project Background

Within the Shopee Supply Chain WMS team, after several years of precipitation, a set of effective collaboration methods have been formed among our design team, front-end team, and back-end team.

We have the following basic resources and specifications:

• Front-end basic component library;
• Front-end business component library;
• A set of design specifications that front-end and design abide by, design UI based on component library, and output design draft through Figma;
• The current collaborative development and joint debugging method of the front and back ends: through the YAPI online platform, before the interface is completed, the interface document is given in advance.

Pain points in the current development model

However, although these existing components, specifications and processes can help us to produce pages faster to a certain extent, the degree of improvement in development efficiency is limited.

As a warehouse management system mainly based on ToB business, there are a large number of pages related to the management background in the front-end page. The ones that appear more frequently are the list query page and content details page, which are mainly based on form query display, form input, and information display. As follows:

Moreover, after one or two years of precipitation, our page business has a high degree of componentization, and most pages can be split into business component modules one by one.

We have a self-developed basic component library similar to Element UI. The business component is based on multiple basic components coupled with certain business attributes and business logic, and further encapsulated, thus gradually forming a set of business component library.

At the same time, in the joint debugging of front-end and back-end interfaces, facing these pages, the front-end needs to spend a certain amount of time to match each display field and interface field, and the workload of joint debugging is large.

To sum up, the main pain points here are:

• Business pages are highly similar and in high demand, and there are a large number of list query pages and content detail pages;
• The workload of joint debugging of front-end and back-end interfaces is large.

Based on the existing component library resources, design specifications, interface documentation platform resources and the current pain points, we are thinking, can we effectively connect the existing tools in a better way? What means can be used to better improve development efficiency?

2. Think outside the box

In response to the above pain points, our core questions are:

• How to quickly produce front-end pages without business logic? Can the design draft be directly converted into page code?
• How to effectively reduce the front-end and back-end joint debugging time?

To this end, we naturally thought of the concept of the current relatively popular low-code platform. Based on pain points and demands, we have pre-researched some excellent similar low-code platform solutions in the industry.

Comparison of industry solutions

The following is a comparison of several popular low-code platforms in the industry: imgCook, CodeFun, Feibing, H5-dooring, etc.

Program	imgcook	CodeFun	flying ice	H5-dooring
end support	PC, mobile	mobile based	PC, mobile	PC, mobile
For the crowd	development, operation	Mainly for development	development, operation	development, operation
Framework support	15 development specifications including Vue, React, uni-app, React Native, H5, etc.	Vue, WeChat applet	React, Vue	React
Secondary development experience	1) Support visual editing<br/>2) Support schema source code development<br/>3) Support style configuration, attribute configuration, data source processing	1) Support simple behavior increase<br/> 2) Support simple style name modification	1) Provide block code template<br/>2) The code only has component examples, which need to be modified a lot<br/>3) Automatically generate code environment	1) Supports visual editing<br/>2) It is scalable, but basically cannot support secondary editing of pages with high complexity
Design draft support	Sketch, PSD, File Image, Figma	Sketch, Photoshop (internal beta)	not support	not support
Generate code quality	1) High code quality<br/> 2) High maintainability and iterability	1) Reasonable page structure<br/>2) Simpler code	1) Reasonable page structure<br/>2) UI component sample code<br/>3) Basic components will be generated repeatedly	1) Absolute positioning layout that is not easy to maintain<br/> 2) There is some code redundancy
Drag and drop	√	×	×	√
interface connection	√	×	×	×
open source	×	×	√	√

👈左滑查看完整表格

In general, different low-code platforms have their own characteristics and advantages, but none of them fully meet our needs. The most important thing is that it cannot be connected with our existing workflow and existing basic components and business component library resources.

Our appeal is:

• Need to support the identification of Figma design drafts;
• The system can be combined with our self-developed component basic library and business component library, that is, the generated code is based on our own basic component library and business component library;
• Can effectively connect with API, speed up front-end and back-end joint debugging;
• The exported code can be edited twice, with enough flexibility to ensure future optimization and new requirements.

Based on this, we decided to develop a visual low-code system based on UI design draft recognition and automatic API interface docking.

Of course, there is one point that needs to be emphasized here: what we do is low code instead of zero code . Although our business pages are highly similar, the business logic in them is different and very complex. It can be understood as "semi-finished code", its purpose is only to help complete the joint debugging and docking of front-end page layout and some interfaces, and we need to get this middle code , secondary development, and supplement and improve the remaining business logic.

We designed this low-code system to balance efficiency and flexibility.

3. ASLINE Schematic Design

To this end, we developed the ASLINE low-code system by ourselves.

ASLINE, which means Assemble Line, we hope that for some high-frequency, large-volume, and repetitive pages, the system can quickly produce corresponding high-quality code like an assembly line.

3.1 Core Design Ideas

The core design idea of the system is to make full use of existing resources, namely:

Complete component library + online design draft resources + online interface documentation .

The core process is as follows:

1. By identifying the design draft, identifying the components used in the design draft, combining them into pages, and automatically generating flex-based smart layout UI code;
2. Through the interface document, the relevant code of the front-end network request is automatically generated, and the fields in the component are automatically matched and replaced;
3. Based on the above two processes, most of the framework-related code (Vue) is generated by completion, and the complete and runnable structured page code is exported.

The core capabilities provided by the system include:

• Intelligent identification of design draft to page layout and components;
• Canvas supports component dragging and smart snapping;
• Support visual configuration adjustment;
• Component and API interface fields are automatically matched;
• API interface part code is automatically generated;
• The code is automatically generated, merged and exported for download.

3.2 Core Architecture

Let's take a look at our abstract design of the entire system architecture:

The interaction of the entire system is based on the low-code platform , and the core processes are connected in series through three core algorithms: design draft recognition algorithm, intelligent layout conversion algorithm, and API and component field matching algorithm.

4. Core difficulties and solutions

There are many difficulties in this system. Here we will talk about the following three points around the core algorithm:

• How to identify the design draft?
• Layout question: How does the identified component implement the transition from absolute canvas-based positioning to streaming layout on export?
• How to achieve the matching of interface fields and component fields? How to improve matching accuracy?

4.1 Intelligent recognition of design draft components

The ASLINE low-code system supports design draft identification and can intelligently identify the corresponding components used in it.

There is a misunderstanding here, we are not directly comparing the similarity between images, but by abstracting and normalizing the images into a numerical matrix. The final comparison is actually the similarity of the two standardized digital matrices.

In the field of component recognition, you may think of image recognition. The mainstream solution used in the industry is Mask R-CNN's method of relying on machine learning, but we feel that this project does not need to be so complicated and does not need to be used. A complex algorithm even requires the use of machine learning.

At the same time, there is also a scheme that does not require identification, and directly indicates the name of the component through the design draft mark. In the end, this solution was not selected mainly because it did not have good versatility (requires the intervention of the design team, and is not conducive to promotion to other teams).

Therefore, we choose the method described below as a compromise, which is simple in implementation, fast in processing and high in accuracy .

We can obtain a JSON file describing all elements of the design draft through Figma's OpenAPI. Through this JSON file, we can know the text information, CSS information, size and position of various elements. After getting the data, you can extract some effective features, convert these features into a digital matrix, and identify the corresponding components through data comparison.

4.1.1 Data acquisition

We can get the JSON data of any design draft through Figma's OpenAPI. The interface is https://api.figma.com/v1/files/:file_key/nodes , and the format of the returned data is roughly:

 {  
  name: 'Component Name',  
  lastModified: '2021-11-29T08:57:41Z',  
  version: '1334322378',  
  role: 'viewer',  
  nodes: {  
    id: '6:188',  
    name: 'Component Name/View',  
    type: 'FRAME',  
    children: [{  
      "id": "6:202",  
       "name": "bg",  
       "type": "RECTANGLE",  
       "blendMode": "PASS_THROUGH",  
       "absoluteBoundingBox": {  
          "x":1792.0,  
          "y":-762.0,  
          "width":114.0,  
          "height":32.0  
        },  
        "fills":[  
          {  
            "blendMode": "NORMAL",  
            "type":"SOLID",  
            "color":{"r":1.0,"g":1.0,"b":1.0,"a":1.0}  
          }  
        ],  
        "strokes":[  
          {  
            "blendMode":"NORMAL",  
            "visible":false,  
            "type":"SOLID",  
            "color": {  
              "r":0.85882353782653809,  
              "g":0.85882353782653809,  
              "b":0.85882353782653809,  
              "a":1.0   
            }  
          }  
        ]  
      }  
    // ... 还有 N 多个类似的节点  
    ]  
  }  
}

The nodes will contain all kinds of information about each element. The amount of information is too much and too much, and it contains a lot of redundant information, so we need to clean the data in the next step.

4.1.2 Data cleaning

After obtaining all the data of the design draft, we need to clean the data. This is because when the designer draws the design draft, some text or element information may use the hide attribute to hide it instead of deleting the element. Sometimes there may be more layers that do not display effects (similar to an empty div element), or some remarks are marked in the design draft. These are all noises for the identification of the design draft, so the first The second step is to remove these noises by setting some special rules (such as regular matching and identifying some special parameter names).

The main removals include:

• Remove elements whose visible attribute is false;
• remove the remarks;
• Eliminate elements that are not meaningful for display.

4.1.3 Feature data extraction

For the understanding of this step, imagine that when we see such a component in the design draft:

Next, we went to the documentation of the component to find which component should be used, and we found this component:

Although the text information in the two components is different, the length and width are different, but we can know that they are actually the same component.

This is because our brain automatically abstracts the components. We automatically extract the similar features of the two images, standardize them, and then match them: these two images are actually the same thing.

Therefore, our component matching actually mimics this process. Here we only use the extracted text as a feature to explain the process:

• First, use depth-first traversal to extract the information of all text elements;
• Then, generate a two-dimensional array, the number of columns of the two-dimensional array is the width of the Table component (that is, the actual pixel width of the component in the design draft), and the number of rows is the height of the Table component;
• Next, the position of each text element in this two-dimensional array is set to 1, and the other elements of the two-dimensional array that do not have the coordinates of the text element are set to 0.

Let's simulate it with a picture. The black part is the place marked 1 in the two-dimensional array, and the white part is the place marked 0 in the two-dimensional array:

Next, we need to do standardization, that is, to standardize the size and number of text elements, as well as the width and height of this component, according to a certain proportion, into the following style:

The left side is the component after normalization, and the real two-dimensional array is what the right side looks like, a two-dimensional matrix. It actually looks like this (the image size is limited, the actual 0 and 1 will be more):

4.1.4 Data Compression

After getting the above two-dimensional matrix, if you want to save it, it is very space-consuming, so the data should be compressed. While compressing the data, it is also necessary to directly compare the data, that is, a special lossless compression algorithm is required.

It can be found that most of the two-dimensional array of this Table component is 0, and the part of 1 is a small part, so we first turn this two-dimensional array into a one-dimensional array, and then divide it into one part every 32 bits, you can find that each The value in the share is a 32-bit binary.

We convert this binary to decimal number, if the value is 0, this one is automatically skipped. If the value is not 0, an object is used to save it, the key value is set to the index of this piece of data, and the value is set to the value converted from binary to decimal. This way we compress an otherwise very large matrix object into a much smaller object.

Compressed data format:

 {  
  "Table": {  
    "matrix": {  
      "1": 127,  
      "2": 478,  
      “10”: 1229332,  
      // ...  
      "528": 127,  
      "529": 2147483392,  
      "530": 8388607,  
      // ...  
    }  
  }  
}

Among them, "1", "2", "528" represent the standardized coordinates, the information representing this area is not all 0, and the corresponding number behind represents the information of the area, we will change the original 01010101 display to For decimal display.

4.1.5 Component comparison

After the data is compressed, the next step is to compare the data. In the real system, the database will save a compressed data of the standard Table component. When we start the comparison, we first convert a component in the design draft into compressed data through the above transformation, and then convert the data into compressed data. This data is compared with the standard component information library stored in the database.

The comparison process is as follows. After the above calculation, the data of the components in the design draft can be obtained:

 {  
  "Table": {  
    "matrix": {  
      "528": 127,  
      "529": 2147483392,  
      "530": 8388607,  
      // ...  
    }  
  }  
}

Data of a component in the standard component repository:

 {  
  "Table": {  
    "matrix": {  
      "528":300,  
      "529":2147483392,  
      "530":8388607,  
      // ...  
    }  
  }  
}

It can be seen that the key value of the data of the components in the design draft is 528. At this time, I will check whether the components in the standard library have 528. It is found that the index on both sides is 528, and the values are 127 and 300, which means that their Some of the binaries are different. At this time, you can use bit operations to know which bits are the same.

For example, after using the AND operation, we get 127 & 300 = 44 , 44 is the value of a binary number converted to decimal, and the number of 1 in 44 represents the similarity of the two segments, Then how can we know how many digits are 1 in the binary of 44?

We can choose to convert 44 to 32-bit binary, and then traverse it to get how many bits are 1 in total. But a quicker method is that in the 32-bit binary of 44, the number of digits that are 1 must be less than 32, then we use a special bit operation method i & (i-1) , using this bit operation, we can A binary rightmost 1 is removed.

Why i & (i-1) can clear the rightmost 1? Because from a binary point of view, n is equivalent to adding 1 to the least significant bit of n-1. The binary of 44 is 101100, the binary of 43 is 101011, 44 & (44-1) = 40 , and the binary of 40 is 101000, which is one less than the binary of 44. So you can use a loop to subtract 1 from the previous number and know that this number is 0.

Using this method, the number of comparisons must be less than 32, and we can optimize the algorithm based on this. After looping through all the components through this method, select the one with the highest similarity to get the comparison result.

The above is the whole process of identifying a component based on image information, which is roughly divided into 5 steps:

1. data collection;
2. data cleaning;
3. Feature data extraction;
4. data compression;
5. Data comparison.

Through the above 5 steps, the component with the highest similarity can be found. The advantage of this scheme is that it does not require a particularly large sample size. Similar to the function of the model after deep learning training, we found a rough version of the function. The recognition speed of this solution is fast, and the extraction of text information and component recognition are obtained together, and do not need to be divided into two parts, which further improves the efficiency.

However, this solution will sometimes have inaccurate recognition, and this solution is some feature data that is manually defined (for example, in the above example, we set the feature data as text information, and it can also be information such as borders) , which cannot be covered too comprehensively. Therefore, we also have a complete set of degradation and degradation schemes based on identification:

degree of recognition	User action
Components are all identified and identified correctly	Second confirmation
Components are all recognized but some are wrong	Secondary confirmation, identifying wrong components can be manually replaced
Components are identified correctly	Unrecognized parts can be dragged and combined on the canvas
component 0 identification	Completely free drag and drop combinations

Of course, there are still some issues that need to be explained here, such as why not directly use an ID to clearly identify which component is in the design draft? Why do we have to use algorithms to identify? Here are a few reasons:

• Considering the permission assignment of Figma accounts and corresponding design resources within the team, if designers are required to clearly identify each component and bring a specific ID, it will increase the design workload, and it is not conducive to promoting this tool to other teams;
• Based on the existing identification method, assuming that a component is added, there is no need to synchronize the new ID on the UI side.

Based on the above status quo, we have designed such an identification method. And it's generic and can be done based on any component.

4.2 Layout transition

When many low-code platforms are displayed on the canvas, they are based on absolute positioning due to the convenience of calculation and dragging. Most of the layout methods used by the final page code are also absolute positioning.

But in real business code, HTML code needs to have a hierarchical structure, and for now, most of them use flex for flexible layout. So we need an algorithm to convert absolutely positioned components to flex layout.

Our approach is:

• Through the calculation of the position of each component, the HTML code with the hierarchical structure is automatically generated;
• According to the positional relationship between components, the first is divided into "row" structure, and the second is calculated with "column" structure, and the HTML/CSS code based on flex layout is generated.

How to know the row and column information of the layout? We use the projection method , taking such a layout as an example:

It consists of 3 business components , each of which is a flex item. So how do you know its nested structure during the transformation - knowing that it has 2 rows, where the first row has 1 element and the second row has 2 elements side by side?

First look from the left to the right, assuming a beam of horizontal light hits it, the result projected on the wall - the number of discontinuous black blocks, is the number of lines that can be separated:

Based on the above results, the obtained rows are separated and projected from top to bottom to get how many columns are in each row.

Ultimately, we will convert the original absolute positioning based layout to a Flex based layout, similar to this:

 <div class="container">  
  <div class="row-1">  
    <PageHeader />  
  </div>  
  <div class="row-2">  
    <div class="column-3">  
      <ProTable />  
    </div>  
    <div class="column-4">  
      <TrackingHistory />  
    </div>  
  </div>  
</div>

 .container {  
  display: flex;  
  flex-direction: column;  
}  
.row-2 {  
  margin-top: 12px;  
  display: flex;  
}  
.column-4 {  
  margin-left: 20px;  
}

For the entire container, we will set it to a vertical flex layout, set display: flex and flex-direction: column .

Under the container container, the row information (row information) is set according to the above-mentioned projection analysis results, and flex-direction: row is set in each row container, and then according to the vertical projection analysis in each row, the row information is obtained. Column distribution, and at the same time, according to the actual physical arrangement, the spacing information between each column is obtained, thereby summarizing the overall flex layout code.

4.3 Matching between interface fields and component fields

After identifying the corresponding business components in the design draft and rendering them on the page, the ASLINE low-code system also provides the ability to associate and match each business component with an interface, by entering the associated interface document URL, automatically generate front-end network request related code, and automatically match and replace the fields in the component.

Here we mainly do a few things:

First, when the user provides the URL of the interface document corresponding to the business component, the system can help generate the code related to the request, saving part of the workload.

Second, the matching between the front-end page display and the interface field is completed. What does it mean? For example, in a form component, a field is displayed as start time on the front end, and the fields returned to the front end by the back-end interface may be expressed in the form of start_time , begin_time , etc. . Here, what the algorithm needs to do is to match the fields displayed by the front end with the fields returned by the back end and associate them correctly.

Here's the overall process:

1. After the component is selected, the field data of the component can be obtained;
2. Enter the interface document request parameters, and pass the document parameters and component field data to our API processing service;
3. The API processing service obtains the interface document request parameters, and pulls the interface document data entered by the user;
4. The component field data is matched with the interface document data, and the interface field with the highest matching degree is calculated by the similarity algorithm;
5. Replace the component field with the field value after matching the interface document;
6. Return the component data and API interface data after matching the interface document;
7. After the system receives the response data, it updates the matched component data to the page;
8. Users can manually confirm and adjust the matched fields.

Its form is roughly as follows, for our business component ProHeader:

The data panel on the right is as follows:

By entering the URL of the interface document corresponding to the interface, we can pull the interface-related information. The Label column represents the field copy displayed on the front end, and the Prop column shows the matching result between the field and the interface field after intelligent matching.

For such a PageHeader component, after identification and interface matching, such a piece of code will be automatically generated when exporting. Taking the Vue project as an example, the approximate pseudocode is as follows:

 <template>  
  <SPageHeader  
    // 一些组件的 props 入参  
    :infoSchemas="schemaProps.infoSchemas"  
    :infoValues="infoValues"  
    :titleInfo="titleInfo"  
  >  
  </SPageHeader>  
</template>  
 
<script>  
import { Vue, Component } from 'vue-property-decorator';  
import { SPageHeader } from '@/ssc-vue-ui';  
import request from '@/utils/request';  
  
// 通过分析接口文档生成的请求入参 interface  
interface InterfaceGetCcOrderDetail {  
    // ...  
}  
  
// 自动生成请求接口相关代码  
export const getCcOrderDetail = async (params: InterfaceGetCcOrderDetail) => {  
  const res = await request.get(  
    '/api/test/xxx/xxx/get_xx_order_detail',  
    {  
      params: {  
        ...params,  
      },  
    },  
  );  
  return res;  
};  
  
@Component  
export default class SPageHeaderComponent extends Vue {  
  titleInfo = {  
    // 通过识别设计稿得到的一些组件默认文案  
  };  
  
  // basicInfo label 列表，根据设计稿和接口文档，完成组件的实际展示参数与接口文档字段的匹配，及一些业务信息  
  infoSchemas = [  
    { label: 'Source From', key: 'source_from', enums: 'CycleCountSourceFrom' },  
    { label: 'Source ID', key: 'source_id' },  
    { label: 'Type', key: 'cc_type', enums: 'CycleCountType' },  
    {  
      label: 'Operate Function',  
      key: 'cc_operate_function',  
      enums: 'CycleCountSkuCCOperateFunction',  
    },  
    { label: 'DynamicCycle Count', key: 'is_dynamic_count', enums: 'YesNo' },  
    { label: 'Blind Count', key: 'is_blind_count', enums: 'YesNo' },  
    { label: 'One By One', key: 'is_one_by_one', enums: 'YesNo' },  
    { label: 'Create Time', key: 'create_time' },  
    { label: 'Creator', key: 'creator' },  
    { label: 'Cycle Count Method', key: 'cc_method', enums: 'Express' },  
  ];  
  // 简单的初始化逻辑，发送请求，渲染组件  
  mounted() {  
    this.getDetailInfos();  
  }  
  async getDetailInfos() {  
    try {  
      const res = await getCcOrderDetail();  
      console.log(res);  
    } catch (error) {  
      console.log(error);  
    }  
  }  
}  
</script>  
 
<style lang="scss">  
// CSS 相关代码  
</style>

Contains all parts of the entire Vue file, component input parameters, interface requests, component rendering, etc.

Of course, this is just a business component. For a complete page, we will also base on a single component, and another one index.vue to aggregate all the components on the page, and complete the typesetting of related content based on flex layout.

5. System display

Next, we will briefly show the whole system and its core steps.

At the beginning, there is only one input box to enter the homepage, just fill in the corresponding Figma URL.

On the corresponding Figma page, select a common list page in our business and fill in its URL into the input box of the system.

After the identification is completed, the system will give a list of the currently identified business components, and there is a process for the user to confirm the adjustment twice:

Then click Next to enter the main editing area. Get the components we identified on the artboard:

It is divided into:

• Draggable component area: You can add content to the canvas, and some unrecognized components can be dragged into the canvas from here;
• Operation panel area: some general toolbars for canvas operations, such as undo, rollback, auxiliary lines, etc.;
• Canvas area: You can adjust the layout and delete components;
• Secondary editing panel: Perform in-depth adjustments to each component, associating interfaces and other operations.

Finally, after the adjustment is completed, click the export code in the upper right corner to get the separate code of each business component and the complete code of this page.

6. Implementation and follow-up plans

It is necessary to emphasize again here: this platform is only a low-code platform, not a zero-code platform. What the system does for us is only some tedious and repetitive tasks such as page layout and interface joint debugging. After getting the code exported by the system, we still need to complete some complex business logic based on this part of the middle code.

This is a set of system tools that we have realized from our own business scenarios, considering the pain points in actual daily work, and effectively reducing some repetitive mechanical labor.

At present, this system has been connected to multiple sub-projects based on the Vue framework under the Shopee supply chain:

• Applicable scope: At present, the full identification plus the semi-recognition that can be used accounts for about 30%-35% of the design draft, mainly distributed in the list page and details page;
• For these list pages and detail pages, the recognition success rate of business components can reach 92% from the current access and use situation, and only a few cases need to be manually modified or dragged directly on the page;
• For applicable pages, an average of 80% development time savings (from 0.5 - 1 business day to 1 hour for a single listing page).

Of course, ASLINE can significantly improve development efficiency and reduce development costs, but it also has certain limitations and requires various specifications/component libraries as support. And the workload of early development is not low.

Regarding the follow-up prospect, we have begun to adapt to different component libraries and development frameworks (React), and introduced intelligent learning to improve the recognition accuracy and the success rate of interface field matching.

At the same time, we are also continuously optimizing the ease of use and experience interaction of the canvas, and continuously increasing the number of supported components. We expect to support quick drag-and-drop assembly of common pages without input of design resources, serve some non-front-end students, and generate greater business value.

author of this article

Coco, front-end engineer, from the Shopee Supply Chain team.