Using Hypothesis Testing to validate features for a software product
How do you choose the best next feature for your software product ?
When developing a new software product there are often lots of competing factors in choosing the next feature to develop.
When there is an established backlog of features and each feature’s likely quantifiable business benefit is known then Weighed Shortest Job First (WSJF) is a good technique for establishing the priority and roadmap for the product.
But what do you use when there is no roadmap and just ideas of potential features that might provide business benefit, but they’ve not been tested or quantified yet ?
Hypothesis Testing is a technique I use for assisting in this situation. Often performed in conjunction with the Lean Startup method it is a structured way to capture the initial ideas of features, and iteratively develop on those to test that the feature will bring the desired business benefit.
Cyclically forming hypotheses and testing a potential feature through experiments and real user interaction helps shape it, and assists in failing fast if it is quickly established not to be viable or provide the anticipated business benefit.
The starting point is to understand the users and refine ideas, often using Design Thinking, then it is time to form, measure and test the hypotheses.
I thought I’d share two templates that I use during these cycles.
Managing your hypotheses:
The first step is to produce your list of hypotheses; breaking each hypothesis so that it is discrete and quick to test.
A hypothesis may often build upon the successful outcome of another or its lessons learnt so you need to capture these dependencies.
Once you have the start of a list, prioritise it to identify the key tests that must be proven true to continue exploring the feature.
The following is a template for managing your list of hypotheses.
It is a simple table to capture the hypotheses and in particular the prioritisation and dependencies between them.
Measure & Test:
The following the templates of the Test Cards and Learning Cards that assist in providing structured input and output from the hypothesis testing.
These cards facilitate the testing to be planned and documented, and provide a record of decisions made during each cycle.
Test card :
- Hypothesis: Document what you believe the feature will bring to the product
- Test: Describe how you will verify your hypothesis
- Metric: Detail how you will measure the anticipated business benefit of the feature
- Criteria: Outline the success criteria for the hypothesis — eg. what is the target metric for the benefit to be realised (eg. cost or time saving)
Learning card :
- Hypothesis: What was the initial assumption on the test card
- Observation: Document what was observed
- Learnings and insights: Detail the validation of the hypothesis
- Decisions and actions: What actions will be performed next based on the learnings (eg. which hypothesis that is dependent on this one will be tested next, or has sufficient been learnt to prioritise the feature?).
I have used these templates several times to shape a software product. They can be used when only an inkling of an idea for a product is known, but equally can be used on established products to re-shape existing features to better meet user need.
