Megan Scheminske, Senior UX Researcher at Teachable, joined the Rally team on May 11 for an AMA to discuss the pros and cons of continuous discovery. Megan covered topics like the definition and purpose of continuous discovery, the relationships between democratization and continuous discovery, and enabling non-researchers to participate in continuous discovery.

If you missed it, or want to revisit the highlights, read our recap below. If you’d like to watch a recording of the full AMA, follow this link.

Key Takeaways

🔑 Continuous discovery can refer to regular customer interviews or continuous research and provides opportunities for non-researchers to build empathy, generate ideas, and improve problem-solving skills.

🔑 Implementing a continuous discovery approach requires careful preparation, including logistical support, infrastructure, and powerful tooling like Rally, but it's important to accept initial imperfections and learn from them.

🔑 Continuous discovery is not always necessary, especially in research-heavy environments, but can fill gaps between project-based research and provide regular customer touchpoints for product teams.

🔑 PMs and Designers engaging in continuous discovery should avoid using it for decision validation, instead focusing on empathy building and idea generation — the ultimate success metric for continuous discovery is enhancing customer understanding.

Who is Megan?

Megan is currently a Research team of one at Teachable. For the last year, she’s been supporting a Product org of 8 designers, 13 PMs, and over 60 engineers. “Part of my role is enabling those folks to do Research,” she said. “And another part of my role is to actually going out and conducting research.”

Like many in the User Research and Research Ops world, Megan didn’t begin her career in Research. She started out in fine arts, spending a number of years after college as a painter. “I eventually found my way from painting into the world of psychology,” she said. After getting a Master’s degree in counseling psychology, Megan spent time as a mental health researcher and an on-site therapist.

“When I was working in mental health research in sort of an academic setting, a lot of the the projects I was working on were online interventions delivered to patients,” she said. “And so a lot of what I ended up doing was was actually more UX research than one would think.” Megan was able to use her background to transition into UX, spending time in agencies and then moving into the startup world in 2018. She’s been in EdTech ever since, mostly working at smaller companies with less than 200 employees, “which is important context for how I think about continuous discovery,” she said.

What is continuous discovery?

"Continuous discovery often implies different things to different people," Megan began. “So when we talk about continuous discovery, it’s crucial to clarify the term, since there isn’t really an industry definition.” Megan’s first encounter with the concept was at her previous company, Outschool, where she was asked by incoming Product Managers (PMs) to set up weekly customer interviews, only to receive vague responses when she inquired about their goals. She realized, "When product managers use this term, they're often referring to conducting an interview a week with a customer.”

Megan pointed out the influence of Teresa Torres' book Continuous Discovery Habits on this interpretation, acknowledging its popularity among product professionals. However, she also presented an alternative perspective: "Continuous discovery can also mean conducting discovery or research on a continuous basis, akin to the dual-track agile model.”

What are the pros and cons of continuous discovery?

PROS

• Boosts learning: Enhances research skills and provides insightful feedback.
• Increases understanding of new users: Helps comprehend new user journeys.
• Inspires problem-solving: Direct customer interaction can fuel innovative solutions.
• Builds empathy: Fosters understanding and empathy for the research team.

CONS

• Demands time: Can be a significant time investment.
• Presents logistical challenges: Might become overwhelming without ReOps support or proper tools.
• Invites bias: Confirmation and recency biases can occur.
• Fails at validation: Due to small sample sizes, it isn't suitable for validation
• Requires UX maturity: A certain level of investment in Research is needed.

Why continuous discovery?

Megan's passion for continuous discovery stems from her belief that it's often misunderstood, with both its advantages and disadvantages often overlooked. She said that continuous discovery can sometimes be applied as a solution before a problem has been fully understood. “I love to dig into how this looks at different organizations to better understand how it can be both beneficial and problematic.”

What continuous discovery looks like at Teachable

{{quote-megan-1="/cms"}}

"Our head of product is a big advocate for research and very interested in continuous discovery," Megan shared, noting the positive influence this has had on their approach.

With the support of a dedicated product operations manager, they initiated a pilot program to evaluate the potential of a continuous discovery approach at Teachable. "This really did come from our head of product," Megan clarified. Megan said her role has been more focused on providing support rather than driving the initiative.

The pilot program is still in its early stages, with teams only recently starting their weekly interviews. Megan explained their approach: "The very first thing that we did was get some infrastructure in place, including adopting Rally to help us support the logistical side of things without a lot of headcount."

What are the initial impacts of continuous discovery at Teachable?

"I set a pretty clear boundary that I was not going to take on the logistics of continuous discovery," she said. Given her position as a one-person team, managing logistics like recruiting and scheduling weekly interviews wouldn't be feasible nor a good use of her skills.

Megan said she set a simple goal for the three teams participating in the pilot: Conduct one interview a week for a month. Afterward, they would regroup, conduct a retro, and evaluate what they learned, including the benefits and drawbacks.

The program has “gotten off to a bit of a rocky start,” she said. This is mainly due to the time constraints on those conducting the research. "People who do research are doing many other things with their time," Megan explained, pointing out the challenges of finding the right interviewees, handling logistics, and conducting the interviews.

Despite the varying degrees of progress among the teams — one team has conducted three interviews in three consecutive weeks, another has done one, and another hasn't yet started. “I think that’s okay,” she said. “We’re not going to be perfect in this pilot.”

What is continuous discovery good for?

1. Learning opportunities for non-researchers: Continuous discovery offers regular opportunities for non-researchers, such as PMs and designers, to interact with customers and increase their research skills and get valuable feedback.
2. Developing a better understanding of new users: For product teams focused on growth, understanding the new user journey is essential. Continuous discovery can help overcome the challenge of recruiting new users by turning it into a rolling research program, which Megan explained is more logistically feasible.
3. Increasing motivation & problem-solving: “It’s really motivating to connect with your customers,” said Megan. “There’s something especially magical about doing it 1-1.” Giving your PWDRs (people who do research) the opportunity to talk to users directly can be great motivation to solve problems and design effective products and services.
4. Building empathy for the craft of research: “Anytime you have someone in your org step into a research role and trying to own the end-to-end process of finding the right people, asking the right questions, they can better understand how much work it takes, which builds empathy and support for your research team.”

{{quote-megan-2="/cms"}}

Is continuous discovery necessary in a research-heavy environment?

For Megan, continuous discovery is not always necessary, especially when there's already a significant amount of research being conducted. "A healthy product development process is already going to be conducting research on a pretty regular schedule. If you're not doing that, you don't have a healthy product development process," she said.

However, she acknowledged that continuous discovery could fill gaps between concerted, project-based research efforts, helping product teams stay connected with customers. She added, "Your PMs, designers, and engineers should all already have access to customers through that research that's already taking place."

Despite these points, Megan recognized a specific situation where a regular cadence of customer interviews could be helpful. "Where I do think that a regular cadence of interviews with customers can be particularly helpful is with growth teams," she explained. Since growth teams often work in a different cycle, running quick experiments without much time for deep discovery, she said a weekly touchpoint with customers to generate ideas for experiments might be beneficial. However, she cautioned that this approach might not be the best fit for all product teams.

When should you choose to launch a study instead of continuous discovery?

Megan stressed that when goals or questions hint at validation, like confirming a hypothesis, continuous discovery may not be suitable. She cautioned against validation efforts with small sample sizes and suggested traditional research methods instead. Continuous discovery, she explained, is great for building empathy, understanding customers, and idea generation.

{{quote-megan-3="/cms"}}

How do you avoid confirmation bias in continuous discovery?

• Balance your sources of insight: "Don't over-rely on continuous discovery as your only source of insights," Megan advised, emphasizing the need for a well-rounded product development process.
• Acknowledge inevitable biases: When non-researchers regularly interact with users, some decisions may inevitably be biased. "They'll probably make some decisions influenced by recency bias or the specific users they're talking to," she cautioned.
• Document poor decisions. “If a bad decisions is made, as long as it’s not going to be harmful for your users in some important way, consider letting it go,” she said. “But make sure to document it.”
• Involve higher leadership. “If you see a trend of poor decisions, have a conversation with someone like the head of product. You may need to make a case for shifting research practices, potentially integrating other types of research alongside continuous discovery."

Should you offer incentives for continuous discovery sessions?

"I almost always offer incentives for any research we're doing," Megan said. She is a big proponent of always offering incentives, though she noted that not everyone does this consistently.

She stressed the need for understanding user dynamics, particularly in B2B settings. "Whenever we're talking with clients, I check in with my customer success manager partner to better understand whether a small gift card would be appreciated or seen as trifling. I always consult those who know our users best before making decisions about incentives."

Continuous discovery and UX maturity

"Your company has to be at a place where it's willing to invest in Research Operations to some extent in order to make continuous discovery even possible," said Megan.  She emphasized that without this investment, the experience of research participants could be dramatically sacrificed.

"If you're not in a place where you have some buy-in for bringing on tools, like Rally, or maybe a Research Ops headcount, I wouldn't encourage introducing continuous discovery in your organization."

She also addressed the possibility of people asking for continuous discovery despite the lack of organizational readiness. "That's where you just have to make a decision about how much you want to stand in the way," she said. "Use your persuasion skills to direct folks elsewhere, or let something sort of happen and run its course while you focus somewhere else."

Research democratization vs. continuous discovery

This goes back to how continuous discovery is defined. “A lot of us, myself included, are influenced by Teresa Torres’ approach and definition,” said Megan. "In Teresa’s model, a product manager, designer, and maybe an engineer are conducting weekly interviews, so by its very nature, according to her definition, continuous discovery is a democratized approach."

This doesn't mean product teams can't have a discovery track led by a researcher, but Megan emphasized that when most people mention continuous discovery, they're referencing a democratized approach. "Often when we hear the term 'continuous discovery,' I think folks are using it to reference a pretty democratized approach wherein the researcher is not necessarily the one conducting interviews."

Encouraging PMs to conduct their own research

Megan first stressed understanding the reasons behind encouraging PMs to do research. "Why are you encouraging PMs to do research? Are you trying to make better product decisions? Are you trying to build customer empathy?" she asked. She clarified that she doesn't subscribe to the view that PMs should always be doing research.

If a PM shows interest in research and it aligns with the team or organization's needs, but they lack confidence in their research skills, Megan offered a few suggestions for support:

• Remind PMs that they don’t need to be perfect.
• Partner with your PM to the extend that you have the time to do so.
• Shadow interviews your PMs conduct, let them lead but also provide feedback.

Guiding PMs and Designers in continuous discovery conversations

When it comes to guiding her PMs and designers in continuous discovery conversations with users, Megan stresses the importance of understanding its purpose. "Continuous discovery is not a great way to make decisions or validate decisions. It's a great way to build empathy for users and generate ideas," she shared.

She carefully monitors the goals set by her teams, ensuring they don't lean toward validation based on single data points, which she considers a poor practice. "You really don't want to be trying to validate your ideas or making decisions based on an n of one," Megan advised. If she sees their goals veering toward validation, she nudges them toward a different focus.

Each team establishes its own goals for the interviews, but the overarching success metric for the pilot is enhancing customer understanding, measured via internal surveys. Before the program's implementation, they conducted a benchmarking survey evaluating customer proximity and understanding. The ultimate hope is to see confidence numbers rise as a result of continuous discovery. "We're hoping to see those sorts of confidence numbers go up," Megan said.

{{quote-megan-4="/cms"}}

How to help PMs take action from their User Research

"At Teachable, we're still in the early stages," Megan began, when asked about frameworks and best practices for helping PMs analyze and act on their learnings from customer conversations. "We have very scrappy ways that we share the information gleaned from these interviews right now. For instance, we have a spreadsheet where, once you conduct an interview, you paste your recording and link to your note docs.”

For the second phase of their pilot, Megan said they will start figuring out how to ensure these learnings are shared widely and in a way that makes sense for the entire company. Megan emphasized the importance of setting realistic expectations: "It's okay not to immediately have a perfect way to do sense-making for all the interviews taking place. That can come as a second step, but it definitely should come."

What's critical, she continued, is the analysis that takes place after these interviews. "This analysis and interpretation of information, and figuring out how to take action, is often a part that non-researchers might overlook or be unaware of."

Even though Megan couldn't share exact experiences on building this capacity among non-researchers at Teachable, she expressed excitement for the future. "While we're still figuring out the best practices for helping PMs analyze and take action from their learnings, I'm looking forward to diving deeper into it," she said.

Megan’s final advice for implementing continuous discovery

1. Spend time evaluating all your options. Ask yourself why you want to implement a continuous discovery approach. Ensure that the results will align with your goals.
2. Get your infrastructure and logistical ducks in a row. At Teachable, we brought on Rally and have dedicated support from a product operations manager, which have both been amazing. Before we started our continuous discovery pilot, we made to sure to adopt and onboard people onto Rally.
3. Don’t be a perfectionist. As long as you are confident that safety and confidentiality will be protected, let your people who do research start talking to users and trust that they will learn as they go.
4. Protect your time and set boundaries. Specifically set boundaries around the support you will offer for a continuous discovery program.
5. Allow for imperfections. It does not have to be perfect. Allowing it to be messy and just jumping into it can be the best thing. By overcoming the hurdle of perfection, product managers, designers, and researchers can engage more directly with customers and start learning.

{{quote-megan-5="/cms"}}

Supercharge your Research Ops with Rally

Rally’s User Research CRM helps you scale up research processes, empower teams to engage with their users responsible, and grow the impact of User Research across your company. With Rally, you can build templates, track research activity, set up SSO and user permissions, create governance rules, and more. Book a demo now.

“I love this topic and would love to talk with anyone,” said Megan. “If you want to follow up, don’t be a stranger.” We’re grateful to Megan for joining us and sharing her thoughts and experience with continuous discovery. If you’d like to watch the full webinar, follow this link.

‍