You are now in a place to launch an actual product. Lean experiments will help you to continuously evaluate customer needs and the desirability of your solution to ensure ongoing improvement. By capturing and evaluating data, both quantitative and qualitative, of transactions and product use through pilot tests, you can uncover a viable business model.

5.1 Designing lean experiments

Identify the key behavior you want from your customers & identify barriers

Find a metric to track the key behavior you desire. Use this metric ( i.e., signups, referrals, usage) to measure the effect of your test. .

Identify the barriers that deter  people from the key behavior that you want.  These could be frictions that make it difficult for them to sign up and engage, that make the key behavior not sufficiently compelling or that hide your product’s  potential benefits.

Define your expected outcomes & establish a baseline and control group for comparison

A common approach is the parallel split test, or A/B test, where you give a sub-group (treatment group) of customers exposure to an experiment or change/adjustment in your product. You then measure their behavior against the rest of your customers (control).

If you don’t yet have a lot of customers or are not running overlapping concurrent experiments, consider using “time-based” batches, aiming to improve your performance in subsequent experiments.

Vary just 1 variable at a time between your treatment and control groups

For example, you could present two versions of a homepage for a solar lighting company and measure which one has higher sales conversions. Keep all other aspects of the product exactly the same while the two homepages are up so as to ensure that any difference in sales can be attributed to the homepage design.

Control for other factors that could affect your results

Test with a roughly similar population or, ideally, the same people with each iteration to ensure it is your product (and not the preferences of the group of customers) that explain differences in sales. Pay attention to how your experiments could be affected by things like climate, time of day or seasonality, gender of participants, social class, and other factors.

Prove that customers are willing to hand over some type of “currency”

Design experiments where your target customers need to hand over actual currency in exchange for use of the product or service you provide. This currency could be monetary, or you could have them demonstrate persistence by requiring a lot of time in navigating a signup process or by entering their email address or contact information to express willingness to pay/commit to your product .

Track actionable metrics

Track data points that clearly help you distinguish what is working from what is not, so you make concrete decisions to move forward.  Measure actions, not words.

Avoid tracking “Vanity Metrics”, which are data points that sound good, but don’t actually help you make decisions. (Read Vanity vs Actionable Metrics).

Time box your experiments

You may feel the desire to continue running the experiment “just a little while longer” in the hopes of getting better results. The problem is that when left unchecked, weeks can easily turn into months, unnecessarily risking your precious time and resources.

It may be useful to set a time limit for the expected outcomes of your experiments, i.e.: writing a blog post will drive at least 100 sign-ups in 2 weeks.
 

5.2 A/B Testing

A/B testing framework

  • Randomized experiment with two variants, A & B, which are the control and treatment groups.

  • Tests actual behavior in response to different features or stimuli.

  • Ideally you would run an A/B test for each improvement you want to implement.

  • Use online tools to calculate the statistical confidence level you’ll need for each test.

 

5.3 Lean experiment examples

Here are three types of experiments you can consider. (For more inspiration and details on the approaches below, see IDEO’s Field Guide to Human-Centered Design)

 

Test perceived value


Are people intrigued by what you’re offering?
· "Pick up” test -- Place several different versions of a product on a table. Do people pick up yours versus the others? What happens if you tweak one or two features?
· “Click-through” test -- If your MVP is digital (i.e. a landing page), how many people click through to read or see more? Offer two different versions and see which garners more “hits”. This is a basic A/B test.

Test willingness to buy


If it’s available, will people actually part with their money in exchange for your offering?
· “Fake” sales -- Have a signup sheet or a “buy now” button so people can demonstrate intent to pay.
· Crowdfunding -- Set up a Kickstarter campaign and see if people care enough to finance your project.
· High hurdle -- Make your signup process arduous & see who persists. This is a way to find your dedicated customers.

Test willingness to use


Will people use the product as intended?
· Walk away………and then -- Hand over your product & go away. Come back in an hour, a day or a week and see if it was actually used as intended.
· Ask for a self-report -- Ask someone else: a parent, teacher, neighbor, or onlooker about whether people used the product as you desired. Embed a sensor or meter to track use (with permission).

 

Excercise Templates

Use the following templates as starting points to create your own experiments.

🎯  Formulate a hypothesis

🎯  Design your lean experiment

🎯  Experiment report 

 

5.4 What metrics should I use?

This will depend on your organization’s stage and capabilities.  If you’re just starting out, a short list of simple metrics are the best place to start - i.e. one key performance indicator (KPI) per each of the categories below.  As your data and analytical capabilities grow, you might expand what you’re tracking and analyzing as you continue to roll out experiments to validate core hypotheses about the product and market.  The actual categories and target KPIs will vary widely depending on your market and business model, but a few common examples you could consider: 

  • Sales / Uptake: New and repeat customers, mumber of customers who engage with sales campaigns, cost of acquiring a customer (CAC).

  • Customer Lifetime Value: Predicted net profit attributed to the entire future relationship with a customer or group (LTV).

  • Marketing: Customer reach by marketing/communications channel and campaigns, Customers signed up for loyalty programs, customer engagement with loyalty programs (points accumulation, redemption, etc.).

  • Product: Number of customers with each product/service, number of customers with multiple products.

  • Customer Care: Number of customer complaints / queries, average reply and resolution time, number of customers responded to, number of customers with complaints resolved or escalated, number of customers satisfied with the outcome.

  • Churn: Number of customers who have terminated or stopped using your product for X days.

  • Transaction / Activity: Customer activity and dormancy rates, customer preferred transaction channels, customers by size and type of transactions.