Front-end embedded point sdk package

introduction

The front-end sdk solution is very mature. Previously, the company's internal unified burying products were used, and the entire link was opened from the front-end burying and the visual query after data reporting. However, I encountered a problem in a recent privatization project, because the services are all on the server that the customer applied for. It is necessary to store the buried point data in its own database. At the same time, the front-end buried point function is simple and does not require too much What a fancy thing. The company’s internal embedded point products are not applicable, and some very mature external point embedded products are too bloated, so I started to seal a simple embedded point sdk based on the open source package, and briefly talk about some of its functions and solutions. .

Features

For products, the primary concern is the pv and uv of the page, followed by the frequency of some important operations (mainly click events). For some pages with high exposure, you may also pay attention to the heat map of the page. effect. On the basis of meeting these key functions, some general user environment parameters (device parameters, time parameters, and regional parameters) are also carried, and the request is sent to the specified back-end service interface, which basically satisfies a buried point of skd Features.

And the SDK that I encapsulated this time probably has the following functions:

1. Automatically report pv and uv after the page is loaded
2. Support users to manually report buried points
3. Common parameters such as time and equipment are carried by default when reporting
4. Support user-defined buried point parameter reporting
5. Support user identification settings
6. Support automatic start of heat map burying point (any click on the page will be automatically reported)
7. Support dom element configuration click event reporting
8. Support user-defined buried point reporting interface configuration

How to use

Step 1: Introduce in front-end engineering

Put the packaged embedded point sdk file on the CDN, and the front-end project will be imported into the page by CDN method

Step 2: Initialize the buried point configuration

const tracker = new Tracker({
                    appid: 'default', // 应用标识，用来区分埋点数据中的应用
                    uuid: '', // 设备标识，自动生成并存在浏览器中,
                    extra: {}, // 用户自定义上传字段对象
                    enableHeatMapTracker: false, // 是否开启热力图自动上报
                    enableLoadTracker: false, // 是否开启页面加载自动上报，适合多页面应用的pv上报
                    enableHistoryTracker: false, // 是否开启页面history变化自动上报，适合单页面应用的history路由
                    enableHashTracker: false, // 是否开启页面hash变化自动上报，适合单页面应用的hash路由
                    requestUrl: 'http://localhost:3000' // 埋点请求后端接口
        })

The third step: use custom burial point reporting method

// 设置用户标识，在用户登录后使用
tracker.setUserId('9527')

// 埋点发送方法，3个参数分别是：事件类型，事件标识，上报数据
tracker.sendTracker('click', 'module1', {a:1, b:2, c:'ccc'})

Design

After understanding the function and usage, let’s talk about some specific design ideas and implementation schemes in the function.

Buried point field design

The buried point field refers to the parameters that need to be carried when the buried point request is reported, and it is also the field to be used in the final analysis of the buried point data. It usually includes two parts: business fields and general fields, and is designed according to specific needs. Business fields tend to be standardized and concise, while general fields tend to be complete and practical. It's not that the more fields to report, the better, whether it is the front-end request itself or the back-end data storage is a burden. The buried point fields I designed for the needs are as follows:

Field	meaning
appid	Application ID
uuid	Device id
userId	User id
browserType	Browser type
browserVersion	Browser version
browserEngine	Browser engine
language	Language
osType	Equipment type
osVersion	Device version number
eventTime	Buried point reporting time
title	page title
url	Page address
domPath	Event triggered dom
offsetX	The x coordinate of the dom triggered by the event
offsetY	The y coordinate of the dom triggered by the event
eventId	Event ID
eventType	Event type
extra	User Defined Field Object

pv statistics

There are two methods for PV statistics according to the needs of the business side. The first method is completely controlled by the business side. The general method of burying points is called to report when the page is loaded or changed. The second type is to enable automatic PV statistics through initial configuration, and the sdk will complete the reporting of the buried points of this part. The first method is very easy to understand, so I won’t expand it in detail. Here are some implementation principles of sdk automatic burying statistics:

For multi-page applications, every time you enter a page is a pv visit, so after configuring addEventListener = true, the sdk will listen to the load event of the browser. When the page is loaded, it will be buried and reported, so it is essentially browsing Monitor and process the load event of the device.

For single-page applications, the load event will only be triggered when the page is loaded for the first time, and subsequent routing changes will not be triggered. Therefore, in addition to monitoring the load event, you also need to monitor the corresponding event according to the routing change. There are two routing modes for single-page applications: hash mode and history mode. The processing methods of the two are different:

• Hash mode, the implementation principle of hash routing for single-page applications is to achieve no page refresh by changing the hash value of the url. The change of the hash will trigger the browser's hashchange event, so only the hashchange event needs to be monitored in the embedded SDK. The buried point can be reported when the event is triggered.
• In the history mode, the history routing of a single-page application is realized by manipulating the native history object of the browser. The history object records the history of the browser session and provides some methods to manage the session stack. Such as:

history.go(): 
history.forward():
history.back():
history.pushState():
history.replaceState():

Different from the hash mode, the above three methods of history.go, history.forward and history.back will trigger the popstate event of the browser, but the two methods history.pushState and history.replaceState will not trigger the popstate of the browser event. However, mainstream front-end frameworks such as react and vue's single-page application history mode routing are based on the underlying implementation of history.pushState and history.replaceState. Therefore, there is no native event that can be used to monitor the trigger point. In order to solve this problem, the new event can be triggered by rewriting these two events of history:

const createHistoryEvent = function(type) {
    var origin = history[type];
    return function() {
        var res = origin.apply(this, arguments);
        var e = new Event(type);
        e.arguments = arguments;
        window.dispatchEvent(e);
        return res;
    };
};

history['pushState'] = createHistoryEvent('pushState');
history['replaceState'] = createHistoryEvent('replaceState');

After the rewriting, as long as the pushState and replaceState events are monitored in the buried point SDK, the buried point report of the routing changes in the history mode can be realized.

uv statistics

The embedding point is essential for pv support. The SDK will provide a method setUserId to set the user uid and expose it to business use. When the service platform obtains the information of the logged-in user, call this method, and the subsequent embedding point will be The uid is always included in the request, and finally the uv statistics are performed with this field during the burying point analysis. But such uv statistics are inaccurate, because the uv value calculated is less than the actual value because the user is not logged in. Therefore, it is necessary to give a distinguishing mark even when the user is not logged in. There are several common ways of this kind of identification:

• User ip address
• When the user visits for the first time, store a randomly generated uuid in cookie or localStorage
• Browser fingerprint tracking technology, by obtaining the recognizable information of the browser and performing some calculations to obtain a value, then this value is the browser fingerprint. The recognizable information can be UA, time zone, geographic location or your language and many more

Each of these methods has its own drawbacks. The accuracy of the ip address is not enough. For example, sharing an ip, proxy, and dynamic ip in the same local area network will cause data statistics to be wrong. The flaw of both cookie and localStorage is that users can take the initiative to clear them. The application of browser fingerprint tracking technology is currently not very mature.

After comprehensive consideration, the localStorage technology is adopted in the SDK. When the user visits for the first time, a random uuid will be automatically generated and stored. Subsequent reporting will carry this uuid to identify the user information. At the same time, if the business platform calls the setUserId method, the user id will be stored in the uid field. In the final statistics of uv, refer to the uid or uuid field according to the actual situation. The accurate uv data should be a value between uid and uuid.

Heat map report

The point of heat map burying means: listen to user click events at any position on the page, record the clicked elements and positions, and finally get the click distribution heat map on the page according to the number of clicks. The implementation principle of this block is relatively simple. You only need to enable the monitoring of click events for all elements in the buried point sdk. The more important point is to calculate the x and y position coordinates of the mouse click, and you can also set the current click The name or class of the element is also reported together for more refined data analysis.

dom click to report

dom click to report is to achieve the function of automatically reporting buried point data by adding specified attributes on the dom element. Specifically, configure an attribute of tracker-key ='xxx' in the dom element of the page, which means that the element needs to be clicked and reported. It is suitable for reporting general buried point data (without custom buried point data), but There is no need to report the degree of heat map. This configuration method is to save the steps of actively calling the reporting method, but if there are custom data fields in the buried point, you should still call the SDK's buried point reporting method in the code. The implementation method is also very simple. By monitoring the click event on the body globally, when the event is triggered, it is judged whether the getAttribute('tracker-key') value of the current event exists. If it exists, it means that the embedded point event needs to be reported, and the embedded point is called. Just click the reporting method.

Reporting method

The most common way to report the buried points is through the img tag. The img tag is easy to use and is not affected by cross-domain browsers. However, there is a problem that the length of the URL will be limited by the browser. If the length limit is exceeded, it will be automatically truncated. Different browsers have different size limits. In order to be compatible with the IE browser with the most stringent length limit, the character length cannot exceed 2083.
In order to solve the problem of the character length limit reported by img, you can use the beacon request that comes with the browser to report the buried point. The method of use is:

navigator.sendBeacon(url, data);

This method of burying point reporting uses the post method, so the data length is not limited, and the data can be sent to the server asynchronously, and it can ensure that the request is sent before the page is unloaded, that is, the reporting of the burying point is not accidental on the page The impact of uninstallation solves the problem that the ajax page uninstallation will terminate the request. But there are also two disadvantages:

1. There is browser compatibility, most of the mainstream browsers can support, ie does not support.
2. Cross-domain configuration on the server side is required

Therefore, these two methods can be combined and packaged into a unified method to report the buried points. Use the img tag first. When the character length exceeds 2083, use the beacon request instead. If the browser does not support the beacon request, it is best to replace it with a native ajax request. (However, if you don’t consider the IE browser, the img reporting method is actually sufficient and the most suitable method)

const reportTracker = function (url, data) {
    const reportData = stringify(data);
    let urlLength = (url + (url.indexOf('?') < 0 ? '?' : '&') + reportData).length;
    if (urlLength < 2083) {
      imgReport(url, data);
    } else if (navigator.sendBeacon){
      sendBeacon(url, data);
    } else {
      xmlHttpRequest(url, data);
    }
}

About the acquisition of common parameters

The reason why I want to talk about this part is because when I get the device parameters at the beginning, I write the corresponding method by myself, but because of the incomplete compatibility, some devices are not supported. Later, they were replaced with special open source packages to handle these parameters. For example, the platform package specializes in processing the current device's osType, browser engine, etc.; the uuid package specializes in generating time data. So when developing, we still have to make good use of the power of the community. It is definitely faster and better to find a mature solution than to write it yourself.

Key code appendix

This article probably ends here, and finally attaches the sdk core code:

// tracker.js
import extend from 'extend';
import {
    getEvent,
    getEventListenerMethod,
    getBoundingClientRect,
    getDomPath,
    getAppInfo,
    createUuid,
    reportTracker,
    createHistoryEvent
} from './utils';

const defaultOptions = {
    useClass: false, // 是否用当前dom元素中的类名标识当前元素
    appid: 'default', // 应用标识，用来区分埋点数据中的应用
    uuid: '', // 设备标识，自动生成并存在浏览器中,
    extra: {}, // 用户自定义上传字段对象
    enableTrackerKey: false, // 是否开启约定拥有属性值为'tracker-key'的dom的点击事件自动上报
    enableHeatMapTracker: false, // 是否开启热力图自动上报
    enableLoadTracker: false, // 是否开启页面加载自动上报，适合多页面应用的pv上报
    enableHistoryTracker: false, // 是否开启页面history变化自动上报，适合单页面应用的history路由
    enableHashTracker: false, // 是否开启页面hash变化自动上报，适合单页面应用的hash路由
    requestUrl: 'http://localhost:3000' // 埋点请求后端接口
};

const MouseEventList = ['click', 'dblclick', 'contextmenu', 'mousedown', 'mouseup', 'mouseenter', 'mouseout', 'mouseover'];

class Tracker {
    constructor(options) {
        this._isInstall = false;
        this._options = {};
        this._init(options)
    }

    /**
     * 初始化
     * @param {*} options 用户参数
     */
    _init(options = {}) {
        this._setConfig(options);
        this._setUuid();
        this._installInnerTrack();
    }

    /**
     * 用户参数合并
     * @param {*} options 用户参数
     */
    _setConfig(options) {
        options = extend(true, {}, defaultOptions, options);
        this._options = options;
    }

    /**
     * 设置当前设备uuid标识
     */
    _setUuid() {
        const uuid = createUuid();
        this._options.uuid = uuid;
    }

    /**
     * 设置当前用户标识
     * @param {*} userId 用户标识
     */
    setUserId(userId) {
        this._options.userId = userId;
    }

    /**
     * 设置埋点上报额外数据
     * @param {*} extraObj 需要加到埋点上报中的额外数据
     */
    setExtra(extraObj) {
        this._options.extra = extraObj;
    }

    /**
     * 约定拥有属性值为'tracker-key'的dom点击事件上报函数
     */
    _trackerKeyReport() {
        const that = this;
        const eventMethodObj = getEventListenerMethod();
        const eventName = 'click'
        window[eventMethodObj.addMethod](eventMethodObj.prefix + eventName, function (event) {
            const eventFix = getEvent(event);
            const trackerValue = eventFix.target.getAttribute('tracker-key');
            if (trackerValue) {
                that.sendTracker('click', trackerValue, {});
            }
        }, false)
    }

    /**
     * 通用事件处理函数
     * @param {*} eventList 事件类型数组
     * @param {*} trackKey 埋点key
     */
    _captureEvents(eventList, trackKey) {
        const that = this;
        const eventMethodObj = getEventListenerMethod();
        for (let i = 0, j = eventList.length; i < j; i++) {
            let eventName = eventList[i];
            window[eventMethodObj.addMethod](eventMethodObj.prefix + eventName, function (event) {
                const eventFix = getEvent(event);
                if (!eventFix) {
                    return;
                }
                if (MouseEventList.indexOf(eventName) > -1) {
                    const domData = that._getDomAndOffset(eventFix);
                    that.sendTracker(eventFix.type, trackKey, domData);
                } else {
                    that.sendTracker(eventFix.type, trackKey, {});
                }
            }, false)
        }
    }

    /**
     * 获取触发事件的dom元素和位置信息
     * @param {*} event 事件类型
     * @returns 
     */
    _getDomAndOffset(event) {
        const domPath = getDomPath(event.target, this._options.useClass);
        const rect = getBoundingClientRect(event.target);
        if (rect.width == 0 || rect.height == 0) {
            return;
        }
        let t = document.documentElement || document.body.parentNode;
        const scrollX = (t && typeof t.scrollLeft == 'number' ? t : document.body).scrollLeft;
        const scrollY = (t && typeof t.scrollTop == 'number' ? t : document.body).scrollTop;
        const pageX = event.pageX || event.clientX + scrollX;
        const pageY = event.pageY || event.clientY + scrollY;
        const data = {
            domPath: encodeURIComponent(domPath),
            offsetX: ((pageX - rect.left - scrollX) / rect.width).toFixed(6),
            offsetY: ((pageY - rect.top - scrollY) / rect.height).toFixed(6),
        };
        return data;
    }

    /**
     * 埋点上报
     * @param {*} eventType 事件类型
     * @param {*} eventId  事件key
     * @param {*} data 埋点数据
     */
    sendTracker(eventType, eventId, data = {}) {
        const defaultData = {
            userId: this._options.userId,
            appid: this._options.appid,
            uuid: this._options.uuid,
            eventType: eventType,
            eventId: eventId,
            ...getAppInfo(),
            ...this._options.extra,
        };
        const sendData = extend(true, {}, defaultData, data);
        console.log('sendData', sendData);

        const requestUrl = this._options.requestUrl
        reportTracker(requestUrl, sendData);
    }

    /**
     * 装载sdk内部自动埋点
     * @returns 
     */
    _installInnerTrack() {
        if (this._isInstall) {
            return this;
        }
        if (this._options.enableTrackerKey) {
            this._trackerKeyReport();
        }
        // 热力图埋点
        if (this._options.enableHeatMapTracker) {
            this._openInnerTrack(['click'], 'innerHeatMap');
        }
        // 页面load埋点
        if (this._options.enableLoadTracker) {
            this._openInnerTrack(['load'], 'innerPageLoad');
        }
        // 页面history变化埋点
        if (this._options.enableHistoryTracker) {
            // 首先监听页面第一次加载的load事件
            this._openInnerTrack(['load'], 'innerPageLoad');
            // 对浏览器history对象对方法进行改写，实现对单页面应用history路由变化的监听
            history['pushState'] = createHistoryEvent('pushState');
            history['replaceState'] = createHistoryEvent('replaceState');
            this._openInnerTrack(['pushState'], 'innerHistoryChange');
            this._openInnerTrack(['replaceState'], 'innerHistoryChange');
        }
        // 页面hash变化埋点
        if (this._options.enableHashTracker) {
            // 首先监听页面第一次加载的load事件
            this._openInnerTrack(['load'], 'innerPageLoad');
            // 同时监听hashchange事件
            this._openInnerTrack(['hashchange'], 'innerHashChange');
        }

        this._isInstall = true;
        return this;
    }

    /**
     * 开启内部埋点
     * @param {*} event 监听事件类型
     * @param {*} trackKey 埋点key
     * @returns 
     */
    _openInnerTrack(event, trackKey) {
        return this._captureEvents(event, trackKey);
    }

}

export default Tracker;

//utils.js
import extend from 'extend';
import platform from 'platform';
import uuidv1 from 'uuid/dist/esm-browser/v1';

const getEvent = (event) => {
    event = event || window.event;
    if (!event) {
        return event;
    }
    if (!event.target) {
        event.target = event.srcElement;
    }
    if (!event.currentTarget) {
        event.currentTarget = event.srcElement;
    }
    return event;
}

const getEventListenerMethod = () => {
    let addMethod = 'addEventListener', removeMethod = 'removeEventListener', prefix = '';
    if (!window.addEventListener) {
        addMethod = 'attachEvent';
        removeMethod = 'detachEvent';
        prefix = 'on';
    }
    return {
        addMethod,
        removeMethod,
        prefix,
    }
}

const getBoundingClientRect = (element) => {
    const rect = element.getBoundingClientRect();
    const width = rect.width || rect.right - rect.left;
    const heigth = rect.heigth || rect.bottom - rect.top;
    return extend({}, rect, {
        width,
        heigth,
    });
}

const stringify = (obj) => {
    let params = [];
    for (let key in obj) {
        params.push(`${key}=${obj[key]}`);
    }
    return params.join('&');
}

const getDomPath = (element, useClass = false) => {
    if (!(element instanceof HTMLElement)) {
        console.warn('input is not a HTML element!');
        return '';
    }
    let domPath = [];
    let elem = element;
    while (elem) {
        let domDesc = getDomDesc(elem, useClass);
        if (!domDesc) {
            break;
        }
        domPath.unshift(domDesc);
        if (querySelector(domPath.join('>')) === element || domDesc.indexOf('body') >= 0) {
            break;
        }
        domPath.shift();
        const children = elem.parentNode.children;
        if (children.length > 1) {
            for (let i = 0; i < children.length; i++) {
                if (children[i] === elem) {
                    domDesc += `:nth-child(${i + 1})`;
                    break;
                }
            }
        }
        domPath.unshift(domDesc);
        if (querySelector(domPath.join('>')) === element) {
            break;
        }
        elem = elem.parentNode;
    }
    return domPath.join('>');
}

const getDomDesc = (element, useClass = false) => {
    const domDesc = [];
    if (!element || !element.tagName) {
        return '';
    }
    if (element.id) {
        return `#${element.id}`;
    }
    domDesc.push(element.tagName.toLowerCase());
    if (useClass) {
        const className = element.className;
        if (className && typeof className === 'string') {
            const classes = className.split(/\s+/);
            domDesc.push(`.${classes.join('.')}`);
        }
    }
    if (element.name) {
        domDesc.push(`[name=${element.name}]`);
    }
    return domDesc.join('');
}

const querySelector = function(queryString) {
    return document.getElementById(queryString) || document.getElementsByName(queryString)[0] || document.querySelector(queryString);
}

const getAppInfo = function() {
    let data = {};
    // title
    data.title = document.title;
    // url
    data.url = window.location.href;
    // eventTime
    data.eventTime = (new Date()).getTime();
    // browserType
    data.browserType = platform.name;
    // browserVersion
    data.browserVersion = platform.version;
    // browserEngine
    data.browserEngine = platform.layout;
    // osType
    data.osType = platform.os.family;
    // osVersion
    data.osVersion = platform.os.version;
    // languages
    data.language = getBrowserLang();
    return data;
}

const getBrowserLang = function() {
    var currentLang = navigator.language;
    if (!currentLang) {
      currentLang = navigator.browserLanguage;
    }
    return currentLang;
}

const createUuid = function() {
    const key = 'VLAB_TRACKER_UUID';
    let curUuid = localStorage.getItem(key);
    if (!curUuid) {
        curUuid = uuidv1();
        localStorage.setItem(key, curUuid);
    }
    return curUuid
}

const reportTracker = function (url, data) {
    const reportData = stringify(data);
    let urlLength = (url + (url.indexOf('?') < 0 ? '?' : '&') + reportData).length;
    if (urlLength < 2083) {
      imgReport(url, data);
    } else if (navigator.sendBeacon){
      sendBeacon(url, data);
    } else {
        xmlHttpRequest(url, data);
    }
}

const imgReport = function (url, data) {
    const image = new Image(1, 1);
    image.onload = function() {
        image = null;
    };
    image.src = `${url}?${stringify(data)}`;
}

const sendBeacon = function (url, data) {
    //判断支不支持navigator.sendBeacon
    let headers = {
      type: 'application/x-www-form-urlencoded'
    };
    let blob = new Blob([JSON.stringify(data)], headers);
    navigator.sendBeacon(url, blob);
}

const xmlHttpRequest = function (url, data) {
    const client = new XMLHttpRequest();
    client.open("POST", url, false);
    client.setRequestHeader("Content-Type", "application/json; charset=utf-8");
    client.send(JSON.stringify(data));
}

const createHistoryEvent = function(type) {
    var origin = history[type];
    return function() {
        var res = origin.apply(this, arguments);
        var e = new Event(type);
        e.arguments = arguments;
        window.dispatchEvent(e);
        return res;
    };
};

export {
    getEvent,
    getEventListenerMethod,
    getBoundingClientRect,
    stringify,
    getDomPath,
    getDomDesc,
    querySelector,
    getAppInfo,
    getBrowserLang,
    createUuid,
    reportTracker,
    createHistoryEvent
}