What separates rigorous research from customer conversations?

Companies with even a rudimentary commitment to user experience understand that they must learn from their customers in order to better serve users. UX Research is one of the primary ways that they do that, but certainly not the only way. Data scientists analyze how users interact with a product in the aggregate. Sales people, account managers, and customer service representatives all have direct conversations with users all the time. And there is a long tradition of product managers talking directly to customers to understand their problems, their needs, and their hopes in relation to a product or problem space. 

These same product managers occasionally observe professional Researchers interviewing research participants. On the surface, these research sessions may not look all that different from the conversations that they themselves are used to having with customers. The surface-level similarities can lead to a fundamental misunderstanding of what Research is. 

I’ve known great high-level product leaders who were big advocates for users and for UX Research who didn’t seem to truly grok these differences. They would fund and support UX Research and encourage their teams to listen to our recommendations. But in deeper conversation with them, I sometimes found that they did not understand very much about what professional researchers do. They didn’t understand the difference between a research-driven insight and a painful story that they personally heard from a customer. Some saw these learnings as essentially equivalent. But there are a multitude of ways in which they are radically different. 

This article articulates many of the ways in which rigorous research differs from casual customer conversations. Both modes of learning are valuable and have their place in product development. My aim is to help product developers better understand how and when to use each mode of learning. 

This chart offers a bird’s-eye view of the differences. Read on below to get more detail and set your product teams up for better success with both modes of inquiry.

What is the difference between customer conversations and rigorous research?

As a democratization program lead, I wholeheartedly believe that folks who work in product development can and should be learning directly from customers. Professional UX researchers are the experts at how to do this, especially when it comes to the most consequential strategic research. But there is a place and a time for casual customer conversations, as well as for more rigorous research. (There is also a place and time for rigorous evaluative research conducted by cross-functional partners, under guidance from a professional UXR. More on that topic here and here.) 

We’ll get into more detail below, but rigorous research is typically conducted by professional researchers, using very intentional methods, sample sizes, screening criteria and analysis techniques. Meanwhile, customer conversations are typically conducted by one product manager or executive, in more casual conversation with a smaller number of customers.

Customer conversations and rigorous research are two different modes of learning. Both modes are valuable, but they serve different purposes. The methods are different. The outputs are different. And most crucially, learnings from these two different modes should be applied to the product development process very differently. 

In short: Customer conversations are valuable for building empathy and sparking new ideas, while rigorous research can help product developers make specific product decisions with confidence. When stakeholders and executives do not understand the difference in these two modes of inquiry, they may make consequential product decisions with misplaced confidence. Because someone spoke to a customer or two, they think of the decision as having been validated by “research,” when in fact, the data is anecdotal at best, and at worst, it could be completely false. 

{{quote-cnash-1="/cms"}}

Read on to learn more about:

• What each mode of learning is
• The costs and risks associated with each mode
• What each mode is particularly good for
• How Research teams might (or might not) choose to support customer conversations

What defines rigorous research?

When professional researchers plan and execute a study, they do so using a variety of established research methods and techniques, informed by their education and/or previous research experience. I won’t attempt to detail a full research education here, but here are some of the big concepts that go into producing research findings that a company can rely on with confidence:

• Defining appropriate research questions – distilling the multitude of things a team might want to know down to a focused and coherent set of goals for a particular study
• Methodology selection – identifying the best methods to use to answer the defined questions reliably
• Participant selection – identifying and recruiting the right people to effectively answer the relevant research questions
• Sample size – understanding how the number of people you talk to relates to the relative confidence you should have in your findings 
• Objective inquiry – phrasing questions in a way that minimizes bias, and moderating sessions with consistency so that you collect consistent data
• Disciplined analysis – analyzing data systematically and thoroughly to minimize confirmation bias and develop a set of learnings that are representative of a larger population
• Appropriate humility – discerning and effectively communicating how reliable specific findings and recommendations actually are, and identifying other sources of information that might increase confidence

Producing reliable research is a complicated endeavor, requiring extensive education and experience to do well. Making one consequential mistake in any one of the dimensions above can seriously compromise the validity and reliability of an entire study. And the list above is in no way comprehensive. There are many more crucial skills that are required to develop reliable research insights. 

What are the benefits of rigorous research?

If effective research is such a large undertaking – requiring significant skills, time, and resources – how do we justify the resources required in a business environment that often values speed over diligence? What is rigorous research good for? 

In a nutshell, rigorous research enables organizations to de-risk decision making. By engaging with users (or potential users) of a product in a disciplined way, research empowers teams to understand how product decisions are likely to impact consumers, and to make more informed and less risky decisions as a result. While research is by no means the only input to business decision making, it is the primary means by which we can truly understand consumers of a product and how a product change is likely to be perceived by and used by those consumers. 

{{quote-cnash-2="/cms"}}

Findings and recommendations from rigorous research are typically well documented, allowing anyone in the organization to understand what was done, who the participants were, and what was learned. These documents can be invaluable in socializing findings and getting new members of a team up to speed quickly. Well-documented findings live on beyond the immediate product decision and can inform additional work, sometimes for years to come. 

In a mature organization, rigorous research is a collaborative effort. Most studies are overseen and conducted by a single researcher, but it takes a village to produce a good research study. Product designers, product managers, and often engineers and product marketing managers are all involved in helping to define the goals of the study. Product designers create research artifacts. Everyone should be involved to some degree in observing sessions, taking notes, and synthesizing the data. This collaborative effort leads to richer and more well-rounded insights. A PM or engineer may have different perspectives on an observation than a researcher, based on their area of expertise. And this group effort means that multiple people benefit from direct exposure to real human users, which helps the entire team to internalize the learnings, and socialize them beyond the immediate team.

{{quote-cnash-3="/cms"}}

What are the risks and costs of rigorous research?

The risks of conducting rigorous research are minimal. It is important to follow best practices to protect proprietary information, to ensure a quality experience for participants, and to protect the privacy of participant data. These best practices are table stakes for professional researchers and research operations personnel. 

Increasingly however, cross-functional partners are conducting their own research. In a proper research democratization program, researchers and Research Ops professionals should be guiding partners to ensure those best practices are followed. If partners are attempting to conduct research without the guidance of UXR professionals, then there are also significant risks that research skill will be lacking and the effort will produce unreliable insights. Product decisions could be made with false confidence, based on sub-standard research practices. 

Though the risks of rigorous research are quite minimal, quality research does come at a considerable cost in terms of time and resources. Even if your research operation is a well-oiled machine, the simplest, most targeted unmoderated research project will likely take at least a week to complete. 
‍
It takes time to plan and code an unmoderated project, get alignment with stakeholders, watch session videos, analyze the data, write up findings and share them with your team. Plus there are the costs of an unmoderated research tool and of participant incentives. If you are talking about a moderated evaluative research project, you must add more time for recruiting participants and conducting live sessions. A one-off moderated research project will likely take a minimum of two weeks. And bigger generative research projects might take months to complete. In our fast-moving business environment, research costs valuable time. 

{{quote-cnash-4="/cms"}}

Rigorous research requires significant resource investment, but if done correctly, that investment helps stakeholders make better-informed product decisions. It can quite literally save millions of dollars in the long run, if you find and correct a fatal flaw in a product before building and launching it. 

What is a customer conversation?

When I talk about customer conversations, I’m really referring to any direct contact between product development teams and customers that does NOT employ the rigor described above. Most commonly, customer conversations are conducted by product managers and/or product designers. They might schedule interviews with customers to seek feedback about the current product, better understand the context in which customers use the product, or bounce new product ideas off of customers to get their reactions. 

What are the differences between customer conversations and rigorous research?

Let’s look at a few of the dimensions across which customer conversations differ from rigorous research. 

Sample size: A rigorous research study requires that you engage with at least 5 participants, and often many more, depending on the goals and methods of the study. Conventional wisdom is that a focused usability study can uncover 85% of usability problems in a digital experience by testing with just 5 appropriate participants. But if you are trying to create personas to categorize a diverse user base, you’d want to interview more like 40-50 customers. And in the case of a survey, you’d want hundreds of respondents to produce reliable quantitative data. There are important nuances and some controversy around sample sizes, but my point is that in rigorous research, sample sizes are intentional and match the selected methodologies. In contrast, sample sizes for customer conversations are often based on convenience, and usually involve far fewer customers, often just one or two. 

Competing motivations: The primary motivation of a research study is to learn about your customers’ needs in relation to a product or feature. The goal is to develop insights that are as accurate, objective, and widely applicable as possible. However, customer conversations often happen in the context of multiple competing motivations.

For example, imagine a customer is frustrated by the lack of a specific feature in a product. Their account manager knows that the product team is currently working on a feature that might meet the customer’s needs, so they suggest a call with the customer and the product manager. In this scenario, competing motivations are going to introduce significant biases into the conversation.

• The account manager wants to retain the customer by reassuring them that their concerns will be addressed, so they may paint an unrealistically rosy picture.
• The customer wants to impress upon the company how important their specific needs are, so they may exaggerate or selectively curate the way they describe their needs.
• The product manager wants to reassure the customer, and to inform development of the specific feature they are working on, so they may also paint a rosy picture and/or focus the conversation on their specific feature, at the expense of a more holistic view of the product as a whole.
  

During this conversation, the product manager may very well gain valuable context about how the customer is using or wants to use the product. But it would be a serious mistake to rely on this bias-laden conversation with a single customer to make consequential product decisions that will affect all customers. 

Multiple internal perspectives: In a mature research practice, a rigorous research study is usually a group effort. Multiple people are involved in designing the study, observing sessions, taking notes, and hopefully synthesizing the data. Researchers hold debriefs after sessions to hear about what other observers saw and found important. However, a customer conversation is typically planned and conducted by one or maybe two people. They make sense of everything they have heard alone, without the benefit of other’s observations, and they are much more likely to fall victim to confirmation bias.

Analysis: Researchers employ rigorous analysis techniques that transform a series of participant anecdotes into a set of insights that are more broadly applicable. In contrast, customer conversations do not involve enough participants to conduct that kind of analysis. Rigorous research analysis transforms anecdotes into more broadly applicable data. Customer conversations result in anecdotal data, which may or may not apply to a larger population. 

Documentation: A proper research report should include a description of the relevant characteristics of the customers who were interviewed, a description of what transpired during the sessions, findings and recommendations that are informed by a rigorous analysis, and a discussion of the relative level of confidence readers should have in individual findings. 

In contrast, customer conversations are rarely documented, and if they are, documentation is unlikely to call attention to the anecdotal nature of the data. A product manager or product designer has a discussion with a single customer. They will likely write up a few notes, and they might share those notes with their immediate team. Those notes are unlikely to include a bright red caveat at the top saying “this data is a single anecdote and should not unduly influence how you think about our customer base as a whole.” 

A reader of that document will be missing many dimensions of context that might be crucial to an accurate understanding of the customer’s feedback, and they may use these anecdotal learnings to inform crucial decisions without recognizing that the data isn’t broadly applicable or representative of their wider user base. 

Reliability: All of the above factors contribute to a distinct contrast in how reliable learnings from a professional research study are compared to learnings derived from less rigorous customer conversations. This is the most consequential difference between the two. 

If a research study was conducted with rigor and documented thoroughly and accurately, then it should be clear just how reliable the findings and recommendations are. The product team should be able to make decisions with relative confidence in the insights produced. However, when a non-UXR team member has a conversation with one or two customers, their “learnings” are anecdotal at best, and should not be relied upon to inform product decisions. And yet, because customers were consulted, their teams will be tempted to think of their decisions as “backed by customer research,” resulting in over-confident decision making. Customer conversations produce the illusion of insight, without the reliability. This illusion breeds misplaced confidence and can lead to disastrous product decisions. 

What is the value of customer conversations?

Should we just never let anyone other than researchers talk to users then? While some might make that argument, I absolutely do not.

Direct connections between product developers and customers can be very beneficial – as long as conversations are not mistaken for rigorous research. I firmly believe that we should encourage more direct connections between people who build products and the people who use them.

Watching someone use your product is a uniquely humbling experience. Customers almost never use our products in the ways that we imagine. They use some features but are unaware of others. They apply our products to problems that we never dreamt of. They use our products in complex ecosystems of other tools. They may not be allowed to use certain features due to IT policy or industry regulation.

No matter how much thought we put into product development, we literally have no idea how our products function in the real world, unless we take the time to learn from the humans that use them. Every opportunity a product developer has to learn about how their product fits into a customer’s real world environment encourages humility, empathy, and innovation.

{{quote-cnash-7="/cms"}}

Customer conversations are great for generating empathy and humanizing your users. The first time you watch a real human user try to use your product to accomplish something, all of the assumptions you made when designing that product suddenly come into stark relief. 

That CTA that you thought was unmissable? Turns out it’s totally missable. That phrase you worked so hard on crafting? Turns out it’s being met with blank stares. There is no substitute for seeing your product through someone else’s eyes. 

But just because you saw one person struggle with a phrase doesn’t mean that no one understands that phrase. That customer may have been an outlier, or maybe they were not at their best that day because they were distracted by something in their personal life. 

When you see a single customer struggle with something, that is typically not enough data to inform consequential changes. But seeing someone struggle with your product will change the way you think about developing your product in the future. You will think about individual human users a little more, and abstracted behaviors and metrics a little less. 

{{quote-cnash-8="/cms"}}

What are customer conversations and rigorous research uniquely good for?

The real danger comes in mistaking casual conversations for rigorous research. Both have their place. Both can provide real value. But how each one is valuable is totally different. 

Customer conversations are about understanding the perspective of a single customer. They are about possibilities, seeing your product through fresh eyes, and seeing firsthand what kinds of problems your users face. They should inspire you to think about new ways to create value for the humans that use your product. And a customer conversation might very well inform more rigorous research to confirm and/or expand on what you learned when you talked to a customer or two. 

Rigorous research is about deeply understanding customers on a more general level and with more conviction and more nuance. A thorough research study will allow you to discern differences in attitudes and behaviors between different types of users. You’ll get a more nuanced view of how different variables might influence how a customer perceives something. 

{{quote-cnash-9="/cms"}}

No research study is 100% foolproof, but a good rigorous research study should empower you to make product decisions with reasonable conviction about how that decision will affect your user base (or at least the segments of users that were included in your study). Rigorous research can significantly reduce the risk involved in making specific product decisions. 

What roles should customer conversations and rigorous research play in product development?

Both modes of learning can meaningfully contribute to product development. And we know that we don’t want to confuse one for the other. So practically, how should we use learnings from each mode in the context of product development? 

It is absolutely crucial to contextualize where information is coming from. When an idea is injected into a product conversation, it can quickly become accepted as “fact” in the context of that conversation, especially if it is based on an interaction with customers. So it is crucial that ideas be offered alongside the context of relative conviction. 

Imagine a conversation among a growth team about whether users should be encouraged to invite other people to use the product during their initial onboarding or at a later time. If an influential PM says “I’ve talked to customers and they can’t wait to get their teams onto our platform,” that statement alone can easily steer the direction of this feature. It becomes “known” that earlier is better, and the conversation turns to what that invitation flow should look like. Was that PM referring to a casual conversation or to rigorous research? We don’t know, but decisions will now be made as if this information is both true and generalizable. 

Now instead, imagine that the PM says “When I talked to an engineering manager at Enterprise Customer X, she was very motivated to get her whole team onto the platform as quickly as possible.” That makes it clear that the information is anecdotal. It may or may not be representative of folks in other disciplines or other roles or at other companies. As the conversation proceeds, team members might offer additional evidence to confirm or contradict this idea. They can discuss relative risks and where they might get additional data to inform a decision. The customer anecdote is understood as a single data point  and prompts further inquiry.

Or alternatively, if the PM says “When we looked at this in the onboarding research study, we saw executives excited to send invites right away, but the middle managers were reluctant to invite their teams until they had more context about the platform.” Being clear about where an idea comes from makes a big difference in how that idea is received, and the conviction the team should have in its veracity (plus, you’d never be able to discern that kind of nuance from a customer conversation). Now the team’s conversation turns to which audience group is more important to design for and whether it's possible to serve both groups well. The team can now make nuanced decisions that will better serve both users and the business. 

{{quote-cnash-10="/cms"}}

What role should a Research team play in customer conversations?

Research teams should focus their time on rigorous research projects. But what role, if any, should they play in customer conversations? There are a few options:

1. Prevent or impede them: I do not recommend this path. Customer conversations can play a valuable role. And regardless of value, this is a losing battle in most cases. Product teams are going to talk to customers whether you like it or not.
   
   
2. Ignore them: You may not be able to prevent product teams from talking to customers, but you also don’t have to be involved at all. Let product teams who want to talk to customers forge their own path. If you are a research team of 1, or your team is perpetually overburdened at whatever scale, this may be your only choice. But this path does come with increased likelihood that folks will mistake customer conversations for research, and will make ill-informed product decisions as a result. Even if you must choose this path, it will behoove you to spend some time educating your stakeholders on the ways in which customer conversations differ from rigorous research.
   
   
3. Gatekeep them: You could choose to ignore the content of these conversations, but get involved in the logistics enough that your cross-functional partners can at least benefit from your knowledge of participant recruiting and best practices around participant privacy. For example, if you have a Research Ops function, you might offer (or insist) that partners leverage ReOps’ existing infrastructure for consent and privacy protection at minimum, and even offer recruiting assistance if your ReOps team has the bandwidth. You may want to prioritize your ReOps bandwidth for supporting the research team. In that case, I would suggest an explicit policy that assisting rigorous research projects always takes precedence over assisting customer conversations. This path doesn’t do much to prevent ambiguity around conversations vs research, but it does at least provide some protection against compromising privacy and non-compliance. 

4. Enable and up-level them: If your Research team has the bandwidth, you might choose to get more closely involved in these conversations in a multitude of ways:

• Provide full ReOps recruiting support for partners who are talking to customers. 
• Give partners access to whatever tool you use for moderated studies to help facilitate a more professional experience for the folks they are talking to.
• Offer some basic education about how to talk with customers effectively – putting participants at ease, asking open unbiased questions, etc. This could come in the form of periodic workshops,  individual coaching sessions, or a video or document that offers pointers. You could even make ReOps support contingent on a partner participating in whatever education you provide. 
• Provide a discussion guide template or outline and include some basic education within that template.
• Offer consulting to review and give feedback on discussion outlines. A detailed review of their planned questions is a great educational opportunity. When you make suggestions, explain the why behind each suggestion. This is one of the most effective ways I’ve found to teach some basic interview techniques.  

Professional researchers understandably favor rigorous research. They sometimes bristle at the idea of cross-functional partners connecting directly with customers. They have legitimate concerns about how professional those conversations will be and how the resulting data might be used. But customer conversations also play an important role in product development. 

The key is to ensure that each mode of inquiry is used intentionally, responsibly, and transparently. Learnings from either mode should be shared with complete context about the methods employed and the confidence with which they are held. Use each mode for what it is good for, don’t confuse the two, and your product teams – and ultimately your users – will benefit from the increased understanding they both bring.