Using IndexedDB cache to speed up WebGL 3D programs

foreword

When using webgl to develop 3D applications, it is often found that the loading of 3D scenes is slow, and it often takes a long time to wait, which makes the user experience very unfriendly. The main reason for the slow loading is that the 3D application involves a lot of resource files. These resource files are mainly models and their pictures, and these models and pictures are often relatively large.

In order to speed up the 3D scene, IndexedDB can be used for resource caching on the client side. IndexedDB, the client-side persistent database! Using this caching technology, after the initial access, the file-level data in the 3D scene will be written to the local cache database of the access device, achieving a permanent life cycle on the client side, and clearing the browser cache will not affect the cached 3D model files.

Introduction to IndexedDB

IndexedDB is a front-end data persistence solution (ie, front-end cache), implemented by the browser.
IndexedDB has the following characteristics

• File-based storage. Means its capacity can reach the maximum free space on the hard disk
• Non-relational database. It means that expanding or shrinking fields generally does not require modifying the database and table structure (unless new fields are used as indexes)
• Key-value store. Means access without string conversion process
• Rich storage types. It means that the browser cache is no longer only able to store strings
• Asynchronous: means that all operations are performed in callbacks

The local browser has three persistent data storage technologies, namely Web Storage, IndexedDB, and Web SQL. IndexedDB has the technical characteristics of high query efficiency, large storage space and asynchronous operation, which has huge advantages.

a large storage space of . The storage space of IndexedDB is much larger than that of LocalStorage, generally not less than 250MB, and there is even no upper limit. In HTML5 local storage, IndexedDB stores the most data.

query efficient . IndexedDB is a lightweight NOSQL database that comes with the browser. More efficient than Web Sql, including indexing, transaction processing and query functions.

Asynchronous operation . IndexedDB does not lock the browser when operating, and users can still perform other operations, which is in contrast to LocalStorage, which operates synchronously. Asynchronous design is to prevent the reading and writing of large amounts of data, which slows down the performance of web pages.

At the same time, IndexedDB internally uses an object warehouse to store data. All types of data can be directly stored, including JavaScript objects, to meet the storage needs of 3D scenes.

Therefore, using IndexedDB caching is one of the most excellent front-end caching solutions. Like Babylon.js, it already supports IndexedDB caching at the engine level. You can refer to the following documents:
https://doc.babylonjs.com/divingDeeper/scene/optimizeCached。

Three.js's idea of using IndexedDB

There are many materials on how to use IndexedDB, so this article will not repeat them.

To use IndexedDB to cache model resources, you first need to obtain model-related resources. These model resources include model files and related image files. For example, for the GLTF model, its resources include the .gltf model main file, the .bin format file, the texture map file, and so on. When loading a model for the first time, it must be loading resource files on the network. Through the LoadingMananger of threejs, various resource files of a gltf model can be collected. code show as below:

    const resourceCollector = [];
    const loadingManager = new LoadingManager();
    loadingManager.setURLModifier( (url,path) => {
      console.log(url);
      if(url.startsWith("data:") || url.startsWith("blob:")) {
        return url;
      }
      resourceCollector.push(url);
      return url;
    });

The above code resourceCollector collects the addresses of all model resources in the process of loading the model. Store all resources in IndexedDB after collection:

saveGltfModel:async function(options,resourceCollector){
    const gltfUrl = options.gltfPath;
    const blobs = {};
    for(let i = 0;i < resourceCollector.length;i ++) {
      let url = resourceCollector[i];
      let blob = await loadAsBlob(url);
      blobs[url] = blob;
      await addToDatabase("model",{key:url,blob})
    }
    await addToDatabase("model_info",{key:gltfUrl,content:resourceCollector});
  },

Where loadAsBlob is to load a resource into a blob object, the code is as follows:

  const xhr = new XMLHttpRequest();
  xhr.open("GET", url);
  xhr.responseType = "blob";
  xhr.onerror = function() {reject("Network error.")};
  xhr.onload = function() {
      if (xhr.status === 200) {resolve(xhr.response)}
      else {reject("Loading error:" + xhr.statusText)}
  };
  xhr.send();

The addToDatabase method adds the resource to the IndexedDB database.

function addToDatabase(storename, data) {
  const promise = new Promise( (resolve,reject) => {
    let store = database.transaction(storename, 'readwrite').objectStore(storename);
    let countReq = store.count(data.key);
    countReq.onsuccess = function(event) {
      console.log("count:",event.target.result);
      let count = event.target.result;
      if(count ==  0) {
        let request = store.add(data);
        request.onerror = function (event) {
          console.error('add添加数据库中已有该数据')
          reject(event);
        };
        request.onsuccess = function (event) {
          console.log('add添加数据已存入数据库')
          resolve(event);
        };
      }
    };
  });
}

The next time you get a model, you can first determine whether it is stored locally. If it has been stored locally, you can directly obtain model resources from the local:

 if(this.indexDbCache && indexedDB) {
      if(database == null) {
        database = await initialDB();
      }
      const storeObject = await findInDatabase("model_info",key);
      if(storeObject) {
        return this.loadGltfInDb(options);
      }
    }

Cache effect evaluation

Through testing, it can be found that for relatively large scenes, the model loading speed can be increased several times, ten times or even dozens of times. It can be seen that the IndexedDB cache effect is obvious.

If you are interested in visualization, you can communicate with me on WeChat 541002349.

Pay attention to the public account "ITMan Biao Shu" to receive more valuable articles in time.