+ Introduction

What is Product Market Fit?
Why is it important?
How do I separate product and market?
The PMF Journey
Assessment Exercise

+ Problem Insight

Understand your customers
In person interaction examples
Customer research tips
Personas
Empathy map
Customer journey map
Other resources

+ Value Proposition

Customer profile - pains & gains
Value map - products
Mistakes to avoid
Critical assumptions exercise
Formulate hypotheses
Exercise Sheet - Value prop canvas

+ Problem solution fit

What is an MVP?
Determine what you want to test
Marketing MVP tests
WorldCover Case Study
Product MVP tests
Other resources

+ Launch & Measure

Designing lean experiments
A/B Testing
Lean experiment examples
What metrics should I use?

+ Iterate, pivot, or persevere?

Build -measure-learn
Keep in mind
Destacame Case Study
Where to focus in pivot?
Escala Case Study
Nomanini Case Study

+ Measuring PMF

PMF path
NPS
Must-have score
Lead indicator engagement data
Engagement
Retention
How do I know when I’m at PMF?

Resources

 

We’re now in a place to launch an actual product. Through lean experiments, we’re going to continuously evaluate customer needs and the desirability of our solution. We’ll get the team set up to capture and evaluate data, both quantitative and qualitative, through transactions and use of the product through actual pilots, and start to examine where there might be a viable business model.

Designing Lean Experiments

Identify the key behavior you want your customers to do & identify barriers

This is what you will measure to see the effect of your test -- i.e. signups, referrals, usage.

What are the barriers preventing people from doing the key behavior that you want them to do.  These could be frictions that make it difficult for them to sign up and engage, or that make the key behavior not sufficiently compelling or hide the potential benefits.

Define your expected outcomes & establish a baseline and control group for comparison

A common approach is the parallel split test, also known as an A/B test, where you only expose a sub-group (Treatment) of customers to an experiment who are measured against the rest of your population (Control).

If you don’t yet have a lot of customers or are not running overlapping concurrent experiments, consider using “time-based” batches whose performance you aim to beat in subsequent experiments.

 Change just 1 variable at a time between your Test and Control groups

For example, you could present two versions of a home page for a solar lighting company and measure which one has higher sales conversions.

Control for other factors that could affect your results

Test with a roughly similar population or, ideally, the same people with each iteration.  Pay attention to how your experiments could be affected by things like climate, time of day or seasonality, gender of participants, social class, and other factors.

Prove that customers are willing to hand over some type of “currency” 

Design experiments where your target customers need to hand over actual currency in exchange for use of the product or service you provide. This currency could be monetary, or you could have them demonstrate persistence by investing a lot of time in navigating through a signup process, or entering their email address or contact information to express willingness to pay / commit to your product .

Track actionable metrics

Track data points that clearly help you distinguish what is working from what is not, and make concrete decisions to move forward more effectively.  Measure actions, not words.  
Avoid tracking “Vanity Metrics”, which are data points that sound good, but don’t actually help you make decisions. (Read Vanity vs Actionable Metrics).

Time box your experiments

You may feel the desire to continue running the experiment “just a little while longer” in the hopes of getting better results. The problem here is that when left unchecked, weeks can easily turn into months, which could unnecessarily risk your precious runway
It may be useful to set a time bound target for all of your experiment expected outcomes, i.e.: Writing a blog post will drive > 100 sign-ups in 2 weeks.
 

If you’re thinking about A/B Testing... 

A/B Testing Framework

  • Randomized Experiment with two variants, A & B, which are the control and treatment groups.
  • Tests actual behavior in response to different features or stimuli.
  • Ideally you would run an A/B test for each improvement you want to implement.
  • Use online tools to calculate the statistical confidence level you’ll need for each test.
 

Lean Experiment Examples

How do you test whether your MVP products are working?  How might you validate your hypotheses?  Here are three types of experiments you can design. (For more inspiration and details on the approaches below, see IDEO’s Field Guide to Human Centered Design)

 

Test Perceived Value


Are people intrigued by what you’re offering?
· Pick up” test -- Place several different versions of a product on a table. Do people pick up yours vs. the others? What happens if you tweak one or two features?
· “Click through” test -- If your MVP is digital (i.e. a landing page), how many people click through to read or see more? Offer 2 different versions and see which garners more “hits”. This is a basic A/B test.

Test Willingness to Buy


If it’s available, will people actually part with their money in exchange for your offering?
· “Fake” sales -- Have a signup sheet or a “buy now” button so people can demonstrate intent to pay.
· Crowdfunding -- Set up a Kickstarter campaign and see if people care enough to finance your project.
· High Hurdle -- Make your sign-up process arduous & see who persists. This is a way to find your dedicated customers.

Test Willingness to Use


Will people use the product as intended?
· Walk Away………and then -- Hand over your product & go away. Come back in an hour, a day or a week and see if it was actually used as intended.
· Ask for a self-report -- Ask someone else: Ask a parent, teacher, neighbor, or onlooker about whether people used the product as you desired. Or embed a sensor or meter to track use (getting permission first).

 

Excercises Templates

Use the following templates as a starting point to create your own experiments.

🎯  Formulate a hypothesis

🎯  Design your Lean Experiment

🎯  Experiment Report

 

What metrics should I use? 

This will depend on your organization’s stage and capabilities.  If you’re just starting out, a short list of simple metrics are the best place to start - i.e. one key performance indicator (KPI) per each of the categories below.  As your data and analytical capabilities grow, you might expand what you’re tracking and analyzing as you continue to roll out experiments to validate core hypotheses about the product and market.  The actual categories and target KPIs will vary widely depending on your market and business model, but a few common examples you could consider: 

  • Sales / Uptake: New and repeat customers, Number of customers who engage with sales campaigns, Cost of acquiring a customer (CAC).
  • Customer Lifetime Value: Predicted net profit attributed to the entire future relationship with a customer or group (LTV).
  • Marketing: Customer reach by marketing/communications channel and campaigns, Customers signed up for loyalty programs, Customer engagement with loyalty programs (points accumulation, redemption, etc.).
  • Product: Number of customers with each product/service, Number of customers with multiple products.
  • Customer Care: Number of customer complaints / queries, Average reply and resolution time, Number of customers responded to, Number of customers with complaints resolved or escalated, Number of customers satisfied with the outcome.
  • Churn: Number of customers who have terminated or stopped using your product for X days.
  • Transaction / Activity: Customer activity and dormancy rates, Customer preferred transaction channels, Customers by size and type of transactions.
 
 

Follow through the next chapter


+ Introduction

What is Product Market Fit?
Why is it important?
How do I separate product and market?
The PMF Journey
Assessment Exercise

+ Problem Insight

Understand your customers
In person interaction examples
Customer research tips
Personas
Empathy map
Customer journey map
Other resources

+ Value Proposition

Customer profile - pains & gains
Value map - products
Mistakes to avoid
Critical assumptions exercise
Formulate hypotheses
Exercise Sheet - Value prop canvas

+ Problem solution fit

What is an MVP?
Determine what you want to test
Marketing MVP tests
WorldCover Case Study
Product MVP tests
Other resources

+ Launch & Measure

Designing lean experiments
A/B Testing
Lean experiment examples
What metrics should I use?

+ Iterate, pivot, or persevere?

Build -measure-learn
Keep in mind
Destacame Case Study
Where to focus in pivot?
Escala Case Study
Nomanini Case Study

+ Measuring PMF

PMF path
NPS
Must-have score
Lead indicator engagement data
Engagement
Retention
How do I know when I’m at PMF?

Resources