A/B testing white paper: Leaders grow 5X revenue than laggards

Forrester Survey Shows 126% ROI for Businesses Using A/B Testing

On April 26, the "White Paper on the Overall Economic Impact of Volcano Engine A/B Testing" was officially released. This white paper, written by market research firm Forrester Research, reveals the important impact of A/B testing on business revenue growth, operating costs, productivity optimization, and more. Based on research on multiple companies, Forrester found that revenue growth of companies is highly correlated with their insights. The revenue growth rate of leading companies can reach 5 times that of laggards. A/B testing is one of the most effective tools for companies to improve their insights. .

In the white paper, Forrester researched a number of companies using Volcano Engine A/B testing and found that these companies have significantly increased their advertising revenue and membership operating income. From a comprehensive financial analysis, the ROI (return on investment) of companies using A/B testing has reached 126%.

Volcano Engine A/B Testing Total Economic Impact White Paper: Insights-Driven Companies with Higher Maturity 5x Revenue Growth than Laggards

A/B testing has become a tool for enterprises to reduce costs and increase efficiency

With the gradual fading of the demographic dividend, all walks of life are facing growth bottlenecks of varying degrees. Enterprises urgently need to transform from the extensive operation model in the past to a refined operation model that puts customer experience first. The white paper shows that: 94% of business decision makers believe that data insights related to customer needs and preferences are critical to improving business revenue; 81% of companies believe that they can use data to gain in-depth understanding of customer needs, and continuously improve and optimize solutions based on data. Enhanced competitive advantage.

As one of the most effective tools for customer insight, A/B testing is being adopted by more and more businesses. A/B testing is essentially a controlled experiment, and its method is to randomly compare different strategies and determine the optimal solution through experimental data. In order to objectively present the real value of A/B testing, Forrester used the Total Economic Impact (TEI) model to conduct research and analysis of companies using Volcano Engine A/B testing products.

Research shows that A/B testing can achieve advertising revenue growth, increase member operating income, save operating costs, and improve data analysis efficiency through optimization to achieve productivity optimization. Quantitative analysis shows that the product launch iteration efficiency of the interviewed companies has increased by 20%, the data analysis efficiency has increased by 60%, the three-year advertising business profit has increased by 2.24 million yuan, and the three-year member operating profit has increased by 1.15 million yuan. From a comprehensive financial analysis, the ROI of enterprise application A/B testing reached 126%.

"Through the diversion model and continuous testing, we have greatly optimized the open-screen advertising, and the advertising revenue has doubled, which was impossible to achieve in the past." One interviewed company said that the effect of A/B testing exceeded expected.

The white paper points out that A/B testing also brings non-quantitative benefits, including reducing the risk of new product releases, improving customer engagement and activity, implementing a data-driven culture, and improving enterprise adaptability, enabling rapid business iteration and continuity. Innovation.

Gu Feng, a senior consultant at Forrester, believes that insight-driven maturity is highly correlated with corporate revenue growth. Insight-driven companies with higher maturity have five times the revenue growth of laggards.

From A/B testing to "Agile Digital Intelligence Engine"

ByteDance is known as an "A/B testing" company, with the catchphrase "Ask AB in everything," and even Douyin's name was tested through A/B testing. As an enterprise-level technical service platform under ByteDance, Volcano Engine also regards A/B testing as an important technical tool.

Guo Dongdong, head of Volcano Engine Data Products, said that based on the internal advanced underlying algorithms, scientific diversion capabilities, and intelligent statistical engines, Volcano Engine A/B tests have covered business scenarios such as recommendation, advertising, search, UI, and product functions, providing experimental design , experiment creation, indicator calculation and statistical analysis, to final evaluation and online services throughout the entire experiment life cycle.

Guo Dongdong pointed out that A/B testing is an important productivity tool, but transforming into an insight-driven organization requires more data-driven capabilities. To this end, based on the data-driven concept of ByteDance, Volcano Engine provides a complete set of data middle-end for enterprise customers, realizing from single-product digital intelligence to global digital intelligence.

Volcano Engine Data Product Panorama

The volcano engine data center is divided into SaaS layer and PaaS layer. The SaaS layer is mainly data application products, and the core is to help businesses use data and promote business growth, including A/B testing product DataTester, growth marketing platform GMP, growth analysis DataFinder, customer data platform CDP, and intelligent data insight ABI. The PaaS layer is mainly a data middle-end product, which improves the efficiency of data development and accumulates data assets, including the lake-warehouse integrated analysis service LAS, the Stateless cloud-native open source big data platform E-MapReduce, the extremely fast OLAP engine ByteHouse, and the big data R&D governance suite DataLeap.

Guo Dongdong introduced that, different from the traditional concept of building a data-driven system, the Volcano Engine data middle-end starts from business scenarios, drives the capacity building of the middle-end according to the actual application of the enterprise, and gradually implements data-driven capabilities from top to bottom to build agile data for enterprise customers. Digital engine.