← Product Management from Scratch
Lesson
Lesson 3.2 — Test the hypothesis cheaply: prototypes and experiments
Choose the cheapest experiment that can validate the key value hypothesis before committing to large development.
A map of experiments: from cheapest to most expensive
A map of experiments: from cheapest to most expensive
Before writing code, a PM should ask: 'What is the cheapest way to learn whether our hypothesis is true?' There is a whole arsenal of tools — from interviews to prototypes — arranged by cost and strength of signal.
Interviews about past behaviour are the cheapest start. You talk to potential users about their real experience, not a hypothetical future. More expensive is the Fake-door ('a door to nowhere'): a button or link to a non-existent feature that counts clicks and reveals interest. Smoke test / landing page: you describe the value proposition and collect emails or pre-orders before development. Concierge: you manually perform the service for the first users, simulating the product without automation. Wizard-of-Oz: the user thinks they are interacting with a system, but behind the screen a human is doing everything by hand. Prototype (a clickable mockup): gives a realistic user experience without working code.
The selection rule: use the least costly experiment that still gives an unambiguous signal on your hypothesis. If an interview is enough, do not build a prototype. If you need a behavioural response (not words), climb the ladder to a Fake-door or a landing page.
Important: define the success criterion before you launch the experiment. For example: 'If 8% of visitors leave an email, the hypothesis is supported; if fewer than 3%, it is rejected.' Without a threshold set in advance, results are easy to interpret in favour of whatever answer you want.
Lesson notes
A map of experiments: from cheapest to most expensive
Before writing code, a PM should ask: 'What is the cheapest way to learn whether our hypothesis is true?' There is a whole arsenal of tools — from interviews to prototypes — arranged by cost and strength of signal.
Interviews about past behaviour are the cheapest start. You talk to potential users about their real experience, not a hypothetical future. More expensive is the Fake-door ('a door to nowhere'): a button or link to a non-existent feature that counts clicks and reveals interest. Smoke test / landing page: you describe the value proposition and collect emails or pre-orders before development. Concierge: you manually perform the service for the first users, simulating the product without automation. Wizard-of-Oz: the user thinks they are interacting with a system, but behind the screen a human is doing everything by hand. Prototype (a clickable mockup): gives a realistic user experience without working code.
The selection rule: use the least costly experiment that still gives an unambiguous signal on your hypothesis. If an interview is enough, do not build a prototype. If you need a behavioural response (not words), climb the ladder to a Fake-door or a landing page.
Important: define the success criterion before you launch the experiment. For example: 'If 8% of visitors leave an email, the hypothesis is supported; if fewer than 3%, it is rejected.' Without a threshold set in advance, results are easy to interpret in favour of whatever answer you want.