Chapter Summary: Prioritizing Hypotheses

How and Why We Prioritize Hypotheses

To maximize the expected improvement of key business metrics, we have to prioritize the hypotheses we formulate.

The Eisenhower Matrix

The Eisenhower Matrix is a way of prioritizing the problem is by assigning two qualities: importance and urgency.

Each task enters one of the four quadrants: important and urgent, not important and urgent, important and not urgent, not important and not urgent. Hypotheses from the A quadrant (important and urgent) must be tested first. Then hypotheses from the B quadrant (important and not urgent). For hypotheses, but not for tasks, those that are not important (the C and D quadrants) aren't tested at all.

WSJF

Weighted shortest job first (WSJF) is a method of prioritizing that allows you to evaluate tasks in more detail.

Formula:

\text{WSJF} = \frac{\text{Cost of delay}}{\text{Job duration (size)}}

The denominator contains job duration, or how long hypothesis testing is expected to take. cost of delay in the numerator is the sum of several parameters.

\overset{\normalsize\text{Cost}}{ \text{of Delay}}\space \raisebox{.5em}{=} \space\overset{\normalsize\text{User-Business}}{\text{Value}}\space \raisebox{.5em}{+} \space\overset{\normalsize\text{Time}}{\text{Criticality}}\space \raisebox{.5em}{+} \space\overset{\normalsize\text{Risk Reduction and/or}}{\text{Opportunity Enablement}}

User-Business Value — how much we'll boost a user or business metric if the hypothesis is true

Time Criticality — how urgently the hypothesis must be tested

Risk Reduction/Opportunity Enablement — whether testing the hypothesis will help avoid serious risks or create new business opportunities

Each parameter is given a rating. You can choose a scale that works for you: for example, ratings from 0 to 10.

ICE and RICE

Impact, confidence, effort/ease (ICE) is one of the most popular ways of prioritizing problems:

\begin{aligned}\text{ICE SCORE} = &\text{ Impact}\times\text{Confidence}\times\text{Ease}\\\\\text{ICE SCORE} &= \frac{\text{Impact}\times\text{Confidence}}{\text{Effort}}\end{aligned}

There's also a modified version, RICE:

\text{RICE SCORE} = \frac{\text{Reach}\times\text{Impact}\times\text{Confidence}}{\text{Effort}}

RICE has four components:

Reach — how many users will be affected by the update you want to introduce

Impact — how strongly this update will affect the users, their experience, and their satisfaction with the product

Confidence — how sure you are that your product will affect them in this way

Effort — how much will it cost to test the hypothesis

Here, too, you can use whatever scale is convenient: ratings from 0 to 10 or the Fibonacci sequence.

The Reach, Impact, Confidence, and Effort Parameters

It's not difficult to find out how many users you reach. You can use data you already have or evaluate competitors or market volumes.

The impact parameter shows how strongly a change impacts users who are reached.

There are different ways of measuring impact:

By the share of total screen space which the modified elements will comprise (%)
By the degree to which user experience will change
By a preliminary estimate of importance for users
By the number of new users whom the update would attract

The confidence parameter reflects how sure you are of your estimates for the other parameters.

For example, the confidence level will be high if

You can find the exact number of users who will be affected by the changes
You have evidence that this change will have exactly the impact you predicted (for example, thanks to your previous experience or competitors' cases)
You've calculated the effort required precisely and there's no risk of it rising, or the risk is very low.

The effort parameter tells you how difficult it is to test a hypothesis.