These are my notes on Continuous Discovery Habits by Teresa Torres.

Very practical advise and tool for Product Managers. To quote Teresa:

Approach to give you confidence to act while preparing you to be wrong.

Interestingly, a lot of the advice it very applicable to software architecture:

• Best path to desired outcome is the “ill-structured problem”:
  • There is no right or wrong, only better or worse.
  • Most of the work is framing the problem itself:
    • Framing impacts how we solve it.
    • Good problem-solvers try many framings.

Key Insights

• Product trio:
  • Product Manager + Designer + SW eng.
  • Product trios should interview together.
• At minimum, weekly touch points with customers by the team building the product where they conduct small research activities in pursuit of a desired outcome.
  • Continuous interviewing is the keystone habit for continuous discovery.
• Opportunity Solution Tree (OST):
  
• Avoid “whether or not” decisions, but a “compare and contrast”.
• Best designers evolve the problem and solution space together.
• When an idea does not work, take the time to reflect on what you learned.
• Josh Seiden, Outcomes Over Outputs:
  • An outcome is a change in human behavior that drives business results.
• Primary research question:
  • What opportunity matter most to this customer?
  • Best is to ask them to share specific stories about their experience:
    • Dig into the details.
    • Don’t allow them to generalize.
• An opportunity should have more than one potential solution, otherwise it is simply a solution in disguise.
• Do not score opportunities:
  • Make a data-informed, subjective comparison.
• Choosing an opportunity is an easily reversible decision, so:
  • Avoid analysis-paralysis:
    • We will learn more by acting.
  • You are just committing to explore it further.
• Research shows:
  • More idea == more diverse and novel.
  • Most original ideas tend to be generated toward the end of the ideation session.
  • Individuals outperformed brainstorming groups.
  • We are better at evaluating ideas as a group.
  • Best:
    1. Generate ideas individually.
    2. Share.
    3. Go to 1.
• Fast discovery iterations only possible if instead of testing ideas, test the assumptions that need to be true in order for the ideas to succeed.
  • Do not test one idea at a time, but assumptions from all ideas at the same time.
  • Assumption tests should simulate an experience, so that the participant behaves, not only says that they will do or not.
  • Define evaluation criteria upfront.
  • You are not trying to prove that an assumption is true, but to reduce the risk.

TOC

• Introduction
• Part I - What is Continuous Delivery?
  • Chapter 1 - The What and Why of Continuous Delivery
  • Chapter 2 - A Common Framework for Continuous Delivery
• Part II - Continuous Discovery Habits
  • Chapter 3 - Focusing on Outcomes over Outputs
  • Chapter 4 - Visualizing What You Know
  • Chapter 5 - Continuous Interviewing
  • Chapter 6 - Mapping the Opportunity Space
  • Chapter 7 - Prioritizing Opportunities, Not Solutions
  • Chapter 8 - Supercharged Ideation
  • Chapter 9 - Identifying Hidden Assumptions
  • Chapter 10 - Testing Assumptions, Not Ideas
  • Chapter 11 - Measure Impact
  • Chapter 12 - Managing The Cycles
  • Chapter 13 - Slow You Work
• Part III - Developing You Continuous Discovery Habits
  • Chapter 14 - Start Small, and Iterate

Introduction

• Focus on customers was a better strategy than obsessing about our competitors.
• Naively thought business would be human centered.

Part I - What is Continuous Delivery?

Chapter 1 - The What and Why of Continuous Delivery

• Questions:
  • Does the customer want it?
  • Are you improving over time?
  • Are you creating value for the customer and your business?
• Approach to give you confidence to act while preparing you to be wrong.
• Product trio:
  • Product Manager + Designer + SW eng.
  • Other roles are welcomed but beware of the cost.
• Mindset:
  1. Outcome-oriented.
  2. Customer-centric.
  3. Collaborative.
  4. Visual:
     • Humans have an immense capacity for spatial reasoning.
  5. Experimental.
  6. Continuous.
• At minimum, weekly touch points with customers by the team building the product where they conduct small research activities in pursuit of a desired outcome.

Chapter 2 - A Common Framework for Continuous Delivery

• The focus on outcomes at the cost of the customer is not uncommon.
• Best path to desired outcome is the “ill-structured problem”:
  • There is no right or wrong, only better or worse.
  • Most of the work is framing the problem itself:
    • Framing impacts how we solve it.
    • Good problem-solvers try many framings.
• Pursue business value by creating customer value.
• Opportunity: customer needs, pain points or desires.
• Opportunity Solution Tree (OST):
  
  • Shared understanding.
  • Split big opportunities into smaller and smaller ones.
• Chip and Dan Heath, Decisive Book:
  • 4 villains of decision-making:
    1. Looking too narrowly at a problem.
    2. Confirmation bias.
    3. Letting short-term emotions affect decisions.
    4. Overconfidence.
  • First book to read after this one.
• Avoid “whether or not” decisions, but a “compare and contrast”.
• Best designers evolve the problem and solution space together.
• When an idea does not work, take the time to reflect on what you learned.

Part II - Continuous Discovery Habits

Chapter 3 - Focusing on Outcomes over Outputs

• Josh Seiden, Outcomes Over Outputs:
  • An outcome is a change in human behavior that drives business results.
• Product teams have to do discovery work to identify the connection between product outcomes (metrics they can influence) and business outcomes (metrics that drive the business).
• Measure What Matters valid criticisms from Felipe Castro.
• Outcomes communicate uncertainty.
• Multiple teams with the same business outcome make it easy to blame other teams for the lack of progress.
• Traction metrics (usage of a particular feature) do not give product teams any latitude:
  • They are outputs.
  • Avoid them except:
    1. Junior product teams.
    2. When optimizing a solution instead of finding a new solution.
• Outcome negotiated between product leader and product trio.
• Stable product trio focused on the same outcome is critical, so that they do not need to learn again how to move the metric:
  • Start with a learning goal, then move to a SMART goal.

Chapter 4 - Visualizing What You Know

• Steps:
  1. Agree on scope, focusing on desired outcome.
  2. Individually create an experience map.
  3. Study each other’s map.
  4. Co-create a map:
     • Include all individual nodes of all maps.
     • Collapse similar nodes.
     • Map more than the happy path.
     • Add context (visually if possible)
• Experience maps are visual.
• Drawing is more specific than writing. Language is vague.
  • I strongly disagree.
• Maps:
  • Nodes: moment in time, action or event.
  • Links.
• Refine the map as you learn.

Chapter 5 - Continuous Interviewing

• Interviewing is not to ask your customers what you should build:
  • It is to discover and explore opportunities.
• Direct (factual) questions require recalling facts without context:
  • Prone to cognitive biases.
  • Answers not accurate.
• Confidence isn’t a good indicator of truth or reality.
• When information is missing, our brains simply fill in details to make the story coherent.
• Primary research question:
  • What opportunity matter most to this customer?
  • Best is to ask them to share specific stories about their experience:
    • Dig into the details.
    • Don’t allow them to generalize.
  • Use the experience map as a guide.
• Research question: what you are trying to learn.
• Interview question: what you ask in your interview.
• Synthesize using an interview snapshot:
  • Quick facts: help identify type of customer.
  • Opportunities: use customer’s words.
  • Top right, experience map for this customer.

• Interview at least one customer every week.
• Source of candidates:
  • While users are using your product.
  • Ads.
  • Support.
  • Marketing.
  • Customer Advisory Board.
• Product trios should interview together.

Chapter 6 - Mapping the Opportunity Space

• Our job is to address customer opportunities that drive our desired outcome.
• Branches in the OST must not have overlaps.
• Look for key moments (or nodes) in:
  1. Experience Map.
  2. From patterns in interview experience maps.
• Avoid:
  • Opportunities framed from your company’s perspective.
  • Vertical opportunities (trees with no branches).
  • Opportunities with two parents.
  • Capturing feelings as opportunities.
• An opportunity should have more than one potential solution, otherwise it is simply a solution in disguise.

Chapter 7 - Prioritizing Opportunities, Not Solutions

• Compare top level opportunities in the tree, and recursively do the same with the children of the chosen one.
• Comparison criteria:
  1. Size: how many customers and how often?
  2. Market: how if affects our position in the market:
     • Differentiators vs table stakes.
  3. Company:
     • Does it support company/business unit/team vision/mission/strategy?
     • Available skills in the team.
  4. Customer:
     • How important it is.
     • How satisfied customers are with the current solution?
• Do not score opportunities:
  • Make a data-informed, subjective comparison.
• Choosing an opportunity is an easily reversible decision, so:
  • Avoid analysis-paralysis:
    • We will learn more by acting.
  • You are just committing to explore it further.

Chapter 8 - Supercharged Ideation

• Research shows:
  • More idea == more diverse and novel.
  • Most original ideas tend to be generated toward the end of the ideation session.
• Brainstorming rules:
  1. Focus on quantity.
  2. Defer judgement.
  3. Combine and improve ideas.
  4. Welcome unusual ideas.
• Research shows that individuals outperformed brainstorming groups due to:
  1. Social loafing:
     • People tend to work harder on their own.
  2. Group conformity:
     • People censored themselves due to concerns about how others would judge their ideas.
  3. Production blocking:
     • You are about to say an idea, somebody else jumps in, and you forget your idea.
  4. Downward not setting:
     • Performance of a group tends to be limited to the lowest-performing member.
• Brainstorming groups are subject to “illusion of group productivity”:
  • Groups overestimate their performance, because “cognitive failures” (getting stuck) happens less often than to individuals.
• Best:
  1. Generate ideas individually.
  2. Share.
  3. Go to 1.
• Tips for ideation:
  • Take frequent breaks.
  • Walk.
  • Change scenery.
  • Try different times of the day.
  • Sleep over it.
  • Look at competitors or other industries for analogous problems.
  • Pretend to have a magic want.
• Research shows that we are better at evaluating ideas as a group.
• Dot-vote ideas as a group:
  • First, remove ideas that do not address the opportunity.
  • Only criteria is how well the idea addresses the target opportunity.
  • Until you have set aside 3 ideas.
  • Each idea should have a strong advocate.
• Include the whole team in ideation:
  • Consider also including key stakeholders.

Chapter 9 - Identifying Hidden Assumptions

• Fast discovery iterations only possible if instead of testing ideas, test the assumptions that need to be true in order for the ideas to succeed.
• Use story mapping to align/understand ideas:
  • Be specific.
  • Story map what would be the best solution based on what we know right now.
  • List assumptions in each step.
• Assumptions == risks (desirability, feasibility, usability) + ethical viability.
• Use pre-mortems:
  • Imagine it is 6 months into the future and the product/initiative launched, and it was a complete failure. What went wrong?
• Walk up the OST and be very specific about why the solution will address the opportunity and the opportunity the outcome.
• Prioritizing assumptions:
  • Place them relative to each other.
  • Pick the 2-3 top right most.
  • More important == less likely to have a workaround.

• Expect to generate ~20 assumptions per idea.
• Phrase your assumptions so that they need to be true:
  • NO: customer will not remember their password.
  • YES: customers will remember their password.

Chapter 10 - Testing Assumptions, Not Ideas

• Do not test one idea at a time, but assumptions from all three ideas at the same time:
  • Avoid confirmation bias and escalation of commitment.
• Assumption tests should simulate an experience, so that the participant behaves, not only says that they will do or not.
• It is common for ideas to share assumptions:
  • Can rule out multiple ideas at once.
• Define evaluation criteria upfront:
  • Be explicit:
    • How many people we’ll test.
    • How many people need to exhibit the behaviour.
• You are not trying to prove that an assumption is true, but to reduce the risk:
  • Start small and iterate to bigger, more reliable tests.
• In the majority of cases, you will learn plenty from small-scale tests.
• Tools:
  1. Unmoderated user testing:
     • Produce a video test you can watch later.
  2. One-question surveys.

Chapter 11 - Measure Impact

• Measure not only what you need to evaluate our assumption tests, but also what we need to measure impact on our outcome.
• Start small and experiment your way to the best instrumentation.

Chapter 12 - Managing The Cycles

• Most of the work in discovery is not following the process; it is managing the cycles.

Chapter 13 - Slow You Work

• Share with stakeholders your whole journey, so they can make their own conclusions and help co-create.
• Use your discovery framework to help stakeholders see where their idea does (or does not) fit.
• You aren’t going to win the ideological war in one conversation (or never).

Part III - Developing You Continuous Discovery Habits

Chapter 14 - Start Small, and Iterate

• Continuous interviewing is the keystone habit for continuous discovery.
• The best time to advocate for discovery is when a feature falls short of expectations.
• Consider what you can do in the unique company’s context and let go of the “That would never work here”.