How do we know if users like our new features?

DevUI is a team with both design and engineering perspectives. It serves the Huawei Cloud DevCloud platform and Huawei's internal mid- and back-end systems, as well as designers and front-end engineers.
Official website: devui.design
Ng component library: ng-devui (Welcome to Star)
Official communication: add DevUI-official assistant
DevUIHelper plug-in: DevUIHelper-LSP (Welcome to Star)

introduction

Does the launch of a new version really improve the product experience?

Are users really willing to switch to the new version?

Do users really like our products?

The father of modern management: Peter F. Drucker has a famous saying:

If you can’t measure it, you can’t manage it.

Therefore, we have to measure the experience of the new version's online process, and use data to prove it.

I was fortunate to participate in a closed development project for a new version of a company’s internal product last year. This article focuses on how to better formulate internal and public beta plans, how to use data to prove users’ preferences for the new and old versions, and the conversion rate of users. For each dimension, I will share some of my own experience on the setting of data collection indicators and the angle of data analysis.

begin

The overall online process of our new version in the closed phase is shown in the following figure:

Our first version went online after a lot of problem analysis, discussion and design, and then our new version went through internal testing and public testing, constantly collecting new feedback, and then constantly revising our business flow, and finally on the live network The environment has received unanimous praise from users.

Below I am mainly based on the data analysis methods we do in the online process, which data can be collected to measure our product experience.

In addition, during the launch of the new version, there are actually additional gains. Our user portraits are clearer, the consistency specifications of business scenarios are becoming more and more clear, and some basic components of the front-end page have been precipitated.

1. High-fidelity visual draft output

Before developing the new version, we did a lot of problem analysis and summary on the current old version, and then went deep into the customer site, through real face-to-face interviews and research with users, and sorted out a large number of user demands and slots.

Then the product manager and designer will classify and sort out the problems, and then output the low-fidelity design draft.

Then senior front-end and back-end developers, operators, designers, product managers, etc., participated in the discussion of the low-fidelity design draft, and constantly revised the details of the design draft:

• The participation of developers is mainly to ensure the achievability of details
• The participation of designers, products, etc. is mainly to control the level of business interaction
• Operation and user participation are mainly to determine whether the current pain points are alleviated and resolved

After repeated discussions and revisions to the low-fidelity design draft, everyone finally reached a relatively unanimous conclusion. This process is actually very difficult. The experience itself is subjective, and anyone can publish and hold their own opinions.

At this point, the product manager actually needs to turn the tide, abandon the trivial and unnecessary disputes, and output high fidelity under the premise that the core demands are met. Developers start to implement development, boldly trial and error, and carefully verify.

At that time, our front-end developers, designers, products and others were sitting in a small room for closed development. The most direct benefit of this was the reduction of communication costs.

In addition, according to the past, product persuading the development to implement a demand change, or development persuading the designer to implement a design change, often brings a negative effect: it is not convinced.

When others enter your field of expertise to influence your ideas, they often use dignity and pressure to realize their ideas. This may bring about forced revisions, and you are still not convinced.

When everyone sits together, we can speak directly through the voices of users in the downstream internal and public beta phases. When users jump up, "design, product, development" will consciously identify errors, even without You give an order, he quietly revised it, and went online silently with conviction.

The closure improves the system efficiency of the new version on-line operation to the highest level.

Second, the self-test phase

After the development of the new version is completed, the version at this time must have a large number of defects and high-fidelity reduction problems.

Fully self-tested by developers

At this time, it is recommended to fully mobilize developers to conduct full self-testing. The developers of each business module alternately test each other's modules, comb the core logic of the page, determine whether the basic process is available, and quickly respond to fix the identified problems. You can collaborate through wiki documents , Remark clearly the person who made the defect, the person in charge, the way to reproduce and the severity, etc., and the serious defect should be closed on the same day as much as possible.

Design day review

Secondly, the co-designer conducts a detailed review of the page, mainly to identify and record issues such as high-fidelity reduction and consistency.

Product experience and testing

Finally, product managers should also participate in the self-testing stage as soon as possible, mainly to record and feedback from the perspective of usability issues and ease of use of the main business process, and unified wiki management for designer and product issues and self-proposed issues. Finally, according to the priority of the problem, go to the development side to fix it.

After repeated iterations of discovering and modifying problems in the self-test phase, the number and severity of the bugs gradually converged to a controllable range, and the internal testing phase can now be advanced.

Third, the internal test phase

In the internal test phase, our target test population should switch to our real user group, and more hidden problems will be discovered in real user scenarios.

In addition, we can also combine our 16078cc8214cff heat map and burying event phase to collect user data, perceive the user's operation flow in the new version in advance, and do a risk assessment for the subsequent public test.

3.1 Strategies for soliciting user groups for internal testing

In the collection of users, the strategy we adopted at that time was to insert an advertisement bulletin in the old version to try our best to ensure that the user group for the internal test comes from our real user group and has the habits of using the old version.

In order to better motivate users to be proactive, it is best to set a certain amount of material incentives or bonuses, so as to stimulate users' feedback motivation during use.

There must be a time limit for the entire closed beta activity, usually about 1-2 weeks.

In addition, you can set the best single-day award, and the top 15-20 awards of the entire event. The user's ranking is evaluated based on the quality and quantity of the feedback questions, such as first prize rewards, reward information and ranking information for feedback of 100 questions. It will be posted to the group bulletin every day to let everyone know their ranking and fully motivate everyone to be proactive.

Users understand the main purpose of this event by clicking on the advertisement pop-up box. Willing users will sign up spontaneously, and then the operation staff will organize the group. The internal test user group still needs to set a limit on the scale and feedback of excessive user volume. The problem will cause the internal testing phase to fail to converge, and it must be formulated according to the scale of team manpower support, such as 300, 500, etc.

3.2 Feedback on internal beta issues

In order to better guide users to feedback their own feelings, we use the company's internal tools similar to Github's Issue function to pull closed beta users into the created project in advance, and provide several types of tags by default, as shown in the following figure. Show:

The user should select the label of the question when raising the issue. The labelled question is very convenient for the product manager to classify the collected questions.

Set the deadline for problem feedback every day. After the time is up, the product manager will determine the best of the day and mark the user feedback as valid, bugs and problems that need to be modified, sort out the development side of the distribution as soon as possible, and respond quickly.

Since the user has been guided to make the problem tags in the early stage, we can filter the user’s favorite and disliked feature points according to the tags, sorting and analyzing the problems in the internal test phase can also assist in evaluating the internal test users’ satisfaction with the new version. degree.

3.3 Questionnaire survey

The completion of the internal test questionnaire can be used as a bonus item for the final reward, thereby encouraging internal test users to actively fill in.

The main purpose of the questionnaire survey is to supplement the previous question feedback data and jointly complete the experience measurement of the internal test process.

The problem design can mainly consider the following aspects. Of course, it must be fully designed based on the actual demands of the team to ensure the availability of collected data.

1. User portrait collection, such as the user’s current job type, department information, the type of related tools and platforms used before, whether the old version has been used, the time of use of the old version, etc.;
2. For the evaluation of the new version of the core optimization module, you can set the single evaluation option of the new version of the core optimization point;
3. Whether you will continue to use the new version, and whether you will actively recommend the new version to your friends and other relevant collections of subjective wishes;
4. Scenario points to be improved, collecting business points that will continue to be optimized in the next step;
5. ...

3.4 Closed Beta Summary

In the internal test phase, through Issue feedback and questionnaire data collection, users’ satisfaction with the new version can be measured, and the degree of preference can also be obtained through word cloud analysis to obtain the distribution of hot words on the experience evaluation of the new version, as well as users’ comments on the new version. Measurable data such as the proportion of version usage and recommended subjective willingness.

Using these data can help us better plan the next public beta and optimize the defects of the current version.

Finally, let’s talk about our actual experience of this event. Due to the existence of incentives and our tool itself, everyone will use it every day, so the internal test users are very motivated. There will be about 300 questions feedback a day, and the real user scenario There are indeed a lot of bugs and experience problems, which greatly reduces the impact of the official launch.

Fourth, the public beta stage

4.1 Design of public test buried point scheme

The public beta stage means that our products have met the requirements for official external use. In order to avoid excessive impact on the user habits of the old version, we have added the public beta stage.

At that time, we added a new version of the ad towing pop-up box and quick access to the old version. In the click event of the towing entrance, we made event burying points. Considering that users will share the link access operation with each other, we are in the new The version will also identify whether the user's previous version is new or old, and if there is an old change to the new operation, it will also be reported to the incident.

Of course, it will also provide users with the entrance to the old version from the new version and the corresponding buried events. Changes in the interactive behavior of a Web site will definitely affect the user's usage habits and bring additional learning costs to the users. Therefore, one of the two versions Switching between the two must do a good job of burying the incident.

The main buried point indicators we used in the public beta phase are as follows:

1. New user: the user who visits a site for the first time;
2. Page visits PV: The number of page loads is actually the number of page views;
3. Number of page visitors UV: Generally refers to the number of people who visit a certain site in one day. Multiple visits by the same visitor in one day are only counted as one visitor;
4. New user: a new user who visits a site;
5. User conversion rate: the ratio of users who used the old version to the new version to the total number of users;
6. Retention rate: the percentage of users who are still active after a certain period of time in a specific user group in a certain statistical period of time, showing the daily retention of new users and active users respectively;
7. Bounce rate: the percentage of sessions that left after only visiting one page;
8. Heat map: Observe the user's operation hotspot area;

4.2 Self-verification of buried point plan

Self-verification of the feasibility of the embedding scheme is very important. Once the public beta entry is opened, the user's behavior will be reported. At this time, if there is a problem with the embedding, the most important user data will be lost, the user's natural behavior data on the first day. It is commendable.

The buried points of the above-mentioned schemes should be fully tested in advance to ensure normal data reporting.

It is recommended to conduct exercises for the design, data reporting, and data collection stages of the relevant buried points in a relatively independent self-test environment.

4.3 Public beta data analysis

In the live network environment to visit the official entrance of the new version and the warm-up activities of the operation, a large number of users entered the new version. Due to the adjustment of the internal test phase, the overall user feedback is mainly the impact of interaction habits.

Of course, the most important thing is the data analysis of the buried points. After every day, the product should sort out the buried point data in time for data analysis, and record the data trend of each day in detail.

Taking the UV conversion rate as an example, the user's growth trend can be clearly felt as shown in the figure below, and the data is used to prove the spontaneous attractiveness of the new version to users.

4.4 Public beta data analysis

In the crowd test phase, we combined the buried events to achieve measurable data on the entire new version’s launch. Through the analysis of the aggregated data, we can better formulate the official launch time of the new version, and continue to correct the appearance of the new version in the public testing stage. Experience and defect issues.

Five, summary

The experience of experience is subjective. Many of us can point out a lot of problems with a product experience. The previous WeChat product Zhang Xiaolong did not laugh at himself and 100 million people teach him to make products every day. Our pursuit of experience is endless.

Some of the methods mentioned above are only to assist us in the experience measurement of the existing scheme design. When it comes to the experience of the last product, it depends on whether it is really convenient to help users solve the problem, and whether there is a use scenario that meets the needs of users. Spontaneous attraction.

join us

We are the DevUI team, welcome to come here to build an elegant and efficient man-machine design/development system with us. Recruitment email: [e-mail].

Text/DevUI The Tail of Summer

Recommendations of previous articles

"Let's build a browser engine (recommended collection)"

"Teach you to build a grayscale publishing environment"

"Fire scones under the waterfall: a three-step method to help you quickly locate website performance problems (super detailed)"