Knowledge Base

Chapter Summary: The Basics of Testing Hypotheses in Business

What Does a Business Need?

The aim of any business is to generate revenue and, eventually, profit. What metrics are treated as priority depends on the company's strategy and plans.

A company's or department's strategy determines which metrics need to be improved. An analyst must know how to use and identify the strategy to choose important metrics that can be improved.

Leading Metrics. Decomposition

When changes in an indicator have a direct impact on the company's or department's main business targets, it's called a leading metric. Leading metrics help you make predictions; lagging metrics help explain the past.

Since all metrics are interconnected we first need to describe the ways they're connected, we have to decompose the metric.

Take revenue:

revenue=average purchase sizenumber of purchases\text{revenue} = \text{average purchase size} * \text{number of purchases}
number of purchases=number of acquired usersuser-order conversion rate\text{number of purchases} = \text{number of acquired users} * \text{user-order conversion rate}

Or we can say that

revenue=number of acquired usersrevenue per visitor (RPV)revenue per visitor=total revenuenumber of visitors\begin{aligned} \text{revenue} &= \text{number of acquired users} * \text{revenue per visitor (RPV)} \\ &\text{revenue per visitor} = \frac{\text{total revenue}}{\text{number of visitors}} \end{aligned}

The average revenue per acquired user is called revenue per visitor (RPV).

The same metric (revenue) can be decomposed in a variety of ways, all of them correct. This makes it possible to study the funnel from different angles and find ways to improve key business metrics.

Identifying Important Metrics

Here are the most common approaches that can help you identify key business metrics:

  • Stakeholder analysis
  • Profit and loss (P&L) analysis
  • Customer journey analysis
  • User experience (UX) analysis
  • User interviews

It's important to understand that the ultimate goal of these methods is not to identify key metrics, but by applying these methods gives you get an idea of what is important for users, businesses, or owners and reveal leverage points.

Stakeholder analysis

A stakeholder is a person or group with an interest in the company's activity.

Stakeholders, in the broad sense, can be the heads of the company, its employees or partners, and tax authorities.

In the narrow sense, stakeholders are people with financial interests in the company's performance.

When conducting stakeholder analysis, we need to take into account their work and goals, as well as the way the company's activity affects them.

P&L

A profit and loss (P&L) statement is an accounting statement.

P&L can be really useful for the heads of a company, because:

  • It contains data on all of the company's revenue and expenses.
  • The P&L statement presents information for a period, rather than as of a specific date.
  • It has a standard structure. It's easy to find the necessary template for the type of business in question.
  • The statement is already decomposed into expenditures by area of activity and department. This makes it possible to calculate unit economics with minimal extra calculations.

Customer journey analysis

A customer journey map (СJM) is a diagram that reflects a user's interaction with the website (or service or product) starting with their first visit.

A CJM can provide insight into the way users interact with your product and help you understand their goals, needs, and emotions.

UX analysis

The basic idea of user experience (UX) analysis is to study how users interact with the interface with the participation of the users themselves.

Here are some methods employed in UX analysis:

  • Usability testing
  • Eye tracking
  • Interviews

Metrics related to customer satisfaction help you determine whether users enjoy your product and how much revenue it might bring.

Experiment Logs

To avoid testing the same hypothesis multiple times, specialists maintain experiment logs and hypothesis logs where they record information on hypotheses that are tested, the external conditions, and the outcomes.

Experiment logs can be really helpful in terms of sharing knowledge between departments. They're also essential when discussing or planning further actions with stakeholders, since they provide decision makers with reliable information and help them reject bad ideas.

Here are some general principles for creating experiment logs:

  • The reliability of the information you enter must be confirmed not only by its author, but also by another log user.
  • The rules for testing must be clear and stated explicitly.
  • Transparency. The rules for adding, changing, and deleting data from the log must be clear to all users.
  • Standard formatting. All users must understand how to find the information they need.
  • Accessibility to all interested parties within the company.

How to Generate Ideas

Brainstorming

An important part of this method is avoiding criticism or self-limitation.

Instructional materials

Books, courses, video blogs from experts, and podcasts are great sources of ideas. The important thing is to look beyond the different circumstances to find the common features.

Your own experience

Any work experience can help you generate ideas.

Analyzing the market and competitors

Market analysis involves researching customers and competitors: what are they, what do they want, and what are they capable of?

Customer profiles shed light on important product characteristics that target audience segments would like to see.

Competitor analysis allows you to identify valuable features that cause users to buy more, more often, and at a higher price.

User interviews

Interviewing users is another way to find out what consumers need and what's preventing them from buying your product.

Customer feedback

  • Feedback from the sales department.
  • Feedback from the technical support department.
  • Reviews from app stores and aggregators.
  • Professional reviews.

Insights from studying data

You might start by researching possibilities for improving certain business metrics, but then generate hypotheses for entirely different ones.

UX research

This tells you how users perceive your product and interact with the interface. It can serve as a source for ideas for making the user experience better and more convenient.

TRIZ (the Theory of Inventive Problem Solving)

TRIZ (the Theory of Inventive Problem Solving) is a set of methods for solving complex creative tasks. It helps you dive deep into the problem, find its root, detect contradictions, and solve them.

Formulating a Good Hypothesis

A hypothesis is a premise that can be either proven or rejected in an experiment (but has not yet been).

A hypothesis should:

  • Be logical and consistent
  • Be testable
  • Not contradict known facts
  • Be effective in terms of practice or knowledge
  • Be detailed
  • Concern a particular sphere
Send Feedback
close
  • Bug
  • Improvement
  • Feature
Send Feedback
,