Realization of Spine-based 2D image changeover action

The 2D mall reloading business is different from common business scenarios. There are a large number of reloading and switching actions. However, the existing runtime libraries in the industry lack corresponding API implementation and support. Therefore, this course will lead you to go deep into the underlying principles of Spine 2D rendering, go into the Spine source code runtime library, analyze the call and processing flow of its core modules and rendering layers, and based on this, introduce how to implement reloading and changing actions based on PIXI-spine. And the difficulties in the process of function realization.

1. Introduction to the basic concepts and principles of Spine

For the more important basic concepts in Spine, you can refer to an article shared by the author before, including the conceptual understanding of skeletons, bones, accessories, slots, and skins.

One more concept needs to be emphasized here: The relationship and difference between the data object and the instance object 160b9f2ab6ea82.

Data objects are stateless and can be shared among any number of skeleton instances. The data object class name with corresponding instance data ends with "Data", and the data object without corresponding instance data has no suffix, such as attachments, skins, and animations.

Instance objects have many properties the same as data objects. The attributes in the data object represent the assembly posture and usually do not change. The same property in the instance object represents the current pose of the instance when the animation is played. Each instance object maintains a reference to its data object, which is used to reset the instance object back to the assembly pose.

For example, SkeletonData is a data object, and Skeleton is an instance object. Similarly, Bone instance objects will have corresponding BoneData, Slot instance objects will have corresponding SlotData, etc.

Second, the overall flow chart of Spine rendering

Three, design ideas

Business background:

The company's internal business has a large number of changes and switching actions, so the materials exported by the Spine editor also need to be split, and "headwear", "hairstyle", "tops", etc., are classified into dresses one by one; Similarly, due to the numerous and changeable actions, the actions need to be split into individual actions, and there may be dress substitutions within the actions. Therefore, the separated action material may contain both bone information and dress information.

Take the following action of swinging the sickle as an example, it will be divided into the following parts:

plug-in ideas:

Based on the previous split, in order to allow the characters to easily replace the dress and switch the action, we need to design the dress and action into a "pluggable" form, which is essentially based on the initial basic bones, and then continue. Add costume accessories to the skeleton, or add bone information, and these newly added bones and costumes can also be "removed" from the current skeleton in the future. Realize the replacement of dress or switching of actions.

rendering library selection:

Rendering layer\Comparison item	compatibility	Degree of encapsulation	Scalability
canvas	Does not support grid attachments and coloring	low	low
webgl	√	low	low
threejs	Does not support two color shading and blending modes	high	general
pixijs	√	high	high

In the comparison of the technical solutions adopted for the rendering layer, canvas and threejs still have some compatibility issues; because canvas and webgl both use the browser's original canvas for rendering, the degree of encapsulation is low. If you want to invest in business, you need to do Secondary packaging; Considering the scalability of 2D animation rendering and future business functions, pixijs is relatively more advantageous. The packaging based on pixijs allows us to easily manage instances.

Finally, the pixijs + pixi-spine plug-in was used as the 2D rendering technology solution of cmxiu.

overall layered design:

Regardless of the rendering layer scheme, the function of changing up and changing actions is currently not implemented, and only supports the consumption of one piece of material. Due to the needs and particularities of the business itself, we need to extend this function by ourselves. The overall design is as follows:
!

The top business call layer: exposed to business use, by creating a Role instance, you can easily perform operations such as addDress, removeDress, addAction and removeAction, and the processing logic of the lower layer is transparent to the upper layer.

Business adaptation layer: According to the adaptation logic under the business scenario of cmxiu, load the material resources of cmxiu, decompress and parse the resources, and at the same time call the method provided by the extension plug-in at this layer to modify the data on the rendering instance, including Bones, slots, accessories, etc. to achieve switching dressing and switching actions.

Replace extension plug-ins: expand on the basis of the pixi-spine plug-in. Due to the lack of new and modified accessories on pixi-spine, the new bone sockets and other APIs, it is necessary to extend the underlying methods to provide upper-level calls.

pixi-spine and pixijs: The lowermost rendering library provides rendering support.

4. Realization of the change-up function

According to the author's previous article, every time a slot is rendered, it will get the corresponding attachment from the current skin according to the attachmentName of the current slot.

The skin is the mapping table for accessory query, so you only need to go to the current skin (CM Show only has the default default skin) to update the corresponding accessory, which can realize the dressing function.

As shown below:

As described in the previous rendering process, the rendering layer traverses the slots to render one by one. Essentially, it reads the attachment instance mounted on the slot. Therefore, we need to understand the construction process of the attachment instance and how to update these instances.

Slot1 comes from slotData1. When initializing, it will read the attachmentName in slotData1 and find the corresponding attachment instance in skin.attachments. Here, the index of each item is one-to-one, and the first item in the skin.attachments array corresponds to slotData1, after retrieving the corresponding attachment instance, assign it to the corresponding slot instance, waiting to be rendered by the rendering layer.

From here we can know that what we need to update is the corresponding attachment in the slot, and the attachment retrieval comes from the skin, so what we actually need to update is the corresponding attachment query table on the skin, and update the attachment instance of the corresponding level to a new one An example of dressing up.

Next, the second question that needs to be clarified is how do we generate the corresponding consumer material resources and generate the corresponding attachment instances. SkeletonJson is a parser defined by the spine core library to parse JSON and generate the corresponding skeletonData. This process Include the structure skin.

Therefore, we need to define the loader to load the cmxiu material resources and process them into the corresponding resource format. After the textureAtlas process, the AtlasAttachmentLoader is constructed to be called by skeletonJson. With the loader, json is provided. At this time, skeletonJson can parse and construct the corresponding annex.

Here we need to do the following things:

1. The custom loader loads the material resources, and processes them into the corresponding resource format for lower-level consumption;

2. Imitate the pixi-spine processing flow and construct AtlasAttachmentLoader to call skeletonJson;

3. Extend the skeletonJson bottom prototype chain method, generate attachment instances, and update the new attachment instances to the corresponding hierarchical position of the current skin.

Custom loader processing and skeletonJson calls are as follows:

    // 资源加载 解析 预处理
    const filesParsing = await parsingFiles(this.src);
    const result = await loadAndDealDressFiles(this.dressId, filesParsing);
    result.json = JSON.parse(result.json);
    result.png = {
      [result.pngid]: result.pngContent,
    };
    const renderResource = await getRenderRes(result);
    this.renderResource = renderResource;

    ...

    // 资源消费 构造AtlasAttachmentLoader 调用扩展API updateAttachment
    const { renderResource } = this;
    const that = this;
    const adapter = PIXI.spine.staticImageLoader(renderResource.metadata.images);
    new PIXI.spine.core.TextureAtlas(renderResource.metadata.atlasRawData, adapter, spineAtlas => {
      let attachmentLoader;
      if (spineAtlas) {
        attachmentLoader = new PIXI.spine.core.AtlasAttachmentLoader(spineAtlas);
      }
      const skeletonJsonParser = new PIXI.spine.core.SkeletonJson(attachmentLoader);
      const updateSlotList = skeletonJsonParser.updateAttachment(
        that.sprite.spineData,
        renderResource.data.attachments,
      );
      ...
      that.sprite.skeleton.setToSetupPose();
    });

The core logic of extending skeletonJson's underlying prototype chain method is as follows:

 core.SkeletonJson.prototype.updateAttachment = function(
    skeletonData,
    skinMap,
    skinName = 'default',
  ) {
    ...
    Object.keys(skinMap).forEach(slotName => {
      const slotIndex = skeletonData.findSlotIndex(slotName);
      if (slotIndex === -1) throw new PluginError(`Slot not found: ${slotName}`);
      const slotMap = skinMap[slotName];
      Object.keys(slotMap).forEach(entryName => {
        ...
        const attachment = this.readAttachment(
          slotMap[entryName],
          skin,
          slotIndex,
          entryName,
          skeletonData,
        );

        if (attachment !== null) {
          skin.addAttachment(slotIndex, entryName, attachment);
        }
      });
      
    });
    ...
    return updateSlotList;
  };

Fifth, the realization of the function of changing the action

The processing of actions is more complicated than dressing up, because actions contain more information, including bone information, slot information, attachment information, and animation information.

Before starting the implementation, there is a question that needs to be considered: Does the data object need to be updated? Is it feasible to update the instance object directly?

The answer is no. The actual rendering of the instance object originally comes from the data object, but the actual rendering is the instance object. Why do you need to maintain the update of the data object?

There are two considerations here. One is that the information on the data object is still needed when creating the attachment, and the other is to keep the data relationship between the data object and the instance object synchronized to prevent the two from being separated and not conducive to subsequent maintenance.

For the action, first we need to update the bone information:

1. Update boneData and update bone.

As shown in the figure above, first we need to add the newly added boneData to skeletonData, and then use boneData to construct a new bone instance and add it to skeleton bones. Since the order of bone is not strictly required, only the parent bone is required. The child bones can be parsed before, so the newly added bone can also be pushed directly behind the array.

2. Update slot information and related information:

The update of slot information is more complicated than that of the bone. In addition to the information of the slot itself, there is also information related to the slot. Because the order of the slots is strictly limited, the update of each information must be based on the location of the slot. index to insert.

As shown in the above figure, we need to insert the newly added slotData at the index position corresponding to the array in skeletonData, and at the same time, create the slot instance and insert it into the correct position in the skeleton instance. Since it is currently in the new slot phase, the attachment is null, and slot The final rendering is also to retrieve the skin, so you need to create an empty object in the corresponding position to insert in the skin. Because the index information is recorded in the slotData itself, the new slotData will cause this information to change. Take the figure as an example, in newSlotData4 The index of slotData4 is 4, the index of slotData4 is updated to 5, and so on.

However, in addition to the above information, slot also affects two places, drawOrder and container:

When the drawOrder is initialized, it is a shallow copy of the slots. When there is an animation that controls the change of the slot level, the order in the drawOrder will be adjusted and the current rendering level will be changed. Therefore, we need to reinitialize the shallow copy of the drawOrder to ensure the slot data Unanimous.

The container is the container for pixijs to render the sprite objects in each slot on the screen. The essence of updating the container container object is to be used for on-screen rendering. This mapping relationship is also one-to-one and in the order of index, so it needs to be inserted in the corresponding position in the container array. The new container object.

3. Update the attachment instance mapping on the skin:

Similarly, we need to update the attachment instance mapping on the skin and retrieve the corresponding attachment update on the skin. Based on the previous two steps, we have created a new bone slot and other information and inserted it in the correct position. Next, we need to register a new attachment instance. In the skin, this step is actually similar to the principle of switching dress up and the processing process, so I won't repeat it here.

4. Update the animation object information:

In the last step, we need to update the animation object information, we need to create a new animation state object, and update the binding relationship with the original skeleton instance.

Extend the update method at the bottom layer and pass in the processed data object data externally.

The core logic is as follows:

   ...
    const skeletonData = skeletonJsonParser.updateAnimation(
      this.sprite.spineData,
      renderResource.data.animations,
    );
    this.sprite.updateAnimationState(skeletonData);
    this.sprite.actionNames = Object.keys(renderResource.data.animations); 

Spine.prototype.updateAnimationState = function(skeletonData) {
    this.stateData = new core.AnimationStateData(skeletonData);
    this.state = new core.AnimationState(this.stateData);
    return this;
};

Six, the pits encountered

1. Clear the cache when rendering data goes to the cache

Every time the slot is rendered, the attachment will be checked, and every time it will be rendered, it will be judged whether the attachmentName has changed, and the attachment cache hash will be retrieved. However, we need to update the same-named outfit of the same slot, so we need to manually clear the cache To trigger a rendering update.

     ...
      updateSlotList.forEach(({ slotIndex, attachmentName }) => {
        that.sprite.skeleton.slots[slotIndex].data.attachmentName = attachmentName;
        // 重新设置为空触发更新
        that.sprite.skeleton.slots[slotIndex].currentSpriteName = '';
        that.sprite.skeleton.slots[slotIndex].sprites = {};
        that.sprite.skeleton.slots[slotIndex].currentMeshName = '';
        that.sprite.skeleton.slots[slotIndex].meshes = '';
      });
      that.sprite.skeleton.setToSetupPose();
      ...

2. Slot index affects multiple places and needs to be synchronized in multiple places

As mentioned in the fifth point, in the update process, because the slot information is associated with multiple information, we need to synchronize the update, and the index position is handled strictly in accordance with the position relationship and cannot be disturbed. Therefore, we need to update slotData, index, slots, drawOrder, query mapped skin and container in slotData.

3. DrawOrder is a new array object and cannot directly reuse slots

From the previous introduction, we can see that drawOrder is a shallow copy of slots. Therefore, we cannot simply and crudely directly perform assignment operations, but honestly copy the slots array.

this.sprite.skeleton.drawOrder = this.sprite.skeleton.slots.map(slot => slot); 

4. When pulling out the skins, start with the big index

Since skins did not have corresponding retrieval attachment objects at first, we created a new empty object, but when pulling out, in order to ensure that the order is not cross-affected, so when pulling out the objects in the skin, we need to start from the back Pull forward.

5. Incompatibility of flipX and flipY, mesh compatibility issues
6. By default, the first attachment is taken as the default attachment rendering

Seven, summary

This article summarizes the overall process of Spine rendering, and based on the current Spine runtime library, it implements the reloading and changing action function in a targeted manner, and expands on the original pixi-spine to meet business needs. At the same time, it deeply analyzes the reloading and changing action function. The concrete realization and the pits in the realization process.

Thanks for watching~