Judd Antin, Research, Product, & Design Leader,  joined us for a Rally AMA on February 28 on future-proofing your research practice. If you haven’t already heard Judd on Lenny’s Podcast, or come across his writing on Medium, you are missing out. But lucky for you, we had a jam-packed hour full of amazing questions and incredibly honest and insightful answers that we’re excited to share. 

If you missed our live event, or want to revisit the highlights, read our recap below. If you’d like to watch a recording of the full AMA, follow this link. Continue reading for Judd's thoughts on topics like the current job market, creating lasting impact, democratization, where research should sit in an organization, how to make research less boring, and much more.

Who is Judd?

Judd has spent the last 15 years building research, design, and product teams at Facebook and Airbnb. As a self proclaimed “battle-tested dumpster firefighter” Judd does not shy away from chaos – he thrives in helping teams navigate crises and solve their trickiest problems. 

Judd joined us to paint the picture of a bright future where UX Research is core to an organization’s innovation and success – his exciting vision calls for systemic change, empowering researchers with diverse methodologies, and closer cross-functional collaborations to drive meaningful impact.

Some thoughts on the famous “The UX Research Reckoning is Here”

I want to kick things off by expressing how much I appreciate the many conversations I've had with this wonderful and crazy community of humans throughout my career, but especially since last summer when I wrote a post on the evolution of the field. In the most niche possible way that one can go viral, I have because of it, though that was never my intention. 

I used the phrase “the research reckoning” and I have some mixed feelings about that because now I’m “the reckoning guy” and I’m not sure about that. Now, I think “reckoning” is actually the wrong word because it feels serious and scary. But I don’t want to create anxiety; actually I feel very positive about the future. 

{{quote-juddama-1="/cms"}}

In my article, I made the argument that it might be a good time to take stock of our research practice over the last 10-15 years and see what we could learn and identify maybe some opportunities to pivot and evolve over the next 15 years. I focused specifically on two points, one being the belief that in many situations we’re doing the wrong research. We’re doing too much blunt, mid-level questioning during product development (e.g. How do users think and feel about X? What concerns or challenges are users facing?). 

These are reasonable questions that we’ve been trained we’re supposed to ask, and which others frequently ask of us. But they aren’t pointed enough for us to have real business impact. I argue that we should focus on a different set of questions and work. 

1. Low-level, micro questions; usability; & technical work 

A lot of researchers think this sort of questioning is beneath them – that it's what interns and junior researchers do – and I think that’s bullshit. This work and questioning drives a huge amount of business value and if you want to talk about measurable ROI for research, you should definitely start here. 

2. High-level strategic questions & work

The business impact you have through the first kind of work can earn you social capital you can utilize to focus on high-level strategic work. And by strategic, I mean questions that help set the roadmap for the next quarter, year, five years, etc. 

People use the term “strategic” to mean a lot of different things (you can let me know what you mean when you say “strategic research” here) and I have a strong feeling even those who confidently use the term don’t have a clear definition that everyone can agree on. 

Ultimately, we can be change agents – we can be asked to do different things. For a lot of researchers, capitalism is the boogeyman. But the boogeyman is the worst form of capitalism – and we should avoid the most extractive forms of capitalism, but that doesn’t mean we can’t be people who are intimately tied to business success. That is how we get more power, more influence, and more jobs in industries that pay our salaries.

I think there’s a shift there and we should have a different conversation. This AMA’s theme is future-proofing your research practice. These are a few things I hope to cover in context of the questions asked: 

It’s so important to learn the language of the business and integrate it deeply into our research practice and do that with joy and not shame. We aren’t stooping low by knowing how to read P&L and shareholder reports. For example, deeply integrated knowledge of how conversion funnels work can be used to motivate pointed research questions. 

To future-proof our research practices, we need to become multi-method and we have to acknowledge that a primarily qualitative model of UX Research might not be as common or successful as it has been in the past. Every researcher benefits themselves by understanding survey design basics, statistics, how to interact with data, AI tools, etc. 

We need to be comfortable working with AI tools. I don’t think AI will take away anyone’s job anytime soon. AI tools are going to make us more efficient and powerful as a group – if we learn to use them effectively. 

We need to focus on different questions and turn our practice inward, using our research skills to develop relationships and influence. Research leadership is a research practice. Many of the things that great leaders do are things researchers are well-equipped to do. So let’s do them.

{{quote-juddama-2="/cms"}}

The ultimate moment of success is when researchers are so deeply embedded from beginning to end that you can’t have a meeting without them. Researchers don’t always have a say in how the product process works. If we as an industry put ourselves in a position of demonstrating why that model is so valuable and building skills so that when we are in those rooms, we aren’t just influencing but making decisions, we’ll be so obviously adding value that it becomes easier do things the right way and harder to them the wrong way. 

No one has any idea. I say this to myself on a regular basis to combat imposter syndrome but also to remind myself that the humble learner vine is the most important for me. Everybody around us, especially in tech, kind of pretends that they know what they’re doing and they have their shit together – like there’s some secret handshake that we don’t know. And they don’t. We don’t. No one does. I firmly believe that we should all just hang out and learn from each other. And when I left Airbnb, I created this website as a reminder of this point, I hope you look at it. 

How can research challenge a focus on short-term gains to create lasting impact in organizations prioritizing shareholder value?

My understanding of this economic moment is that many businesses are in short-term mode, just trying to weather the storm, if you will. 

Part of being business-first is being focused on the priorities that the business is focused on, not just trying to convince the business that they’re focused on the wrong priorities. So, how do we demonstrate value in that environment? My answer is mostly the same as in any other environment – start by understanding your organization’s top priorities and work on those. Even if you think those priorities are not the most important thing, working on the things that the rest of your organization thinks are important is a key way that we can demonstrate business value. 

{{quote-juddama-3="/cms"}}

However, if it’s a short-term thing, and you find yourself doing more, say, micro-level research, remember that usability tests can also glean useful mid- and long-term insights. Even when the company is focused on the short-term, I think synthesizing the things you’ve learned and creating a framework to understand the potential for future value is a great thing too, but use the language the business is already using to define their priorities. 

How can you learn the language of the business to effectively make an impact?

You have to learn by doing. Find the last shareholder letter or quarterly report – it doesn’t even have to be from the company you work at. Listen to the earnings report call and watch the presentation. Scour your internal drives. Ask questions like:

• How does the company make money? 
• Where does revenue come from? 
• What does it cost to generate that revenue?
• What are the biggest drivers of changing revenue and cost? 

Find those moments where you are uncomfortable or don’t understand. There’s a lot of lingo and language you probably haven’t encountered before. But it’s important because it’s how the company and executives are talking about users and opportunities. Over time, as you expose yourself to this information, sure you won’t be getting a job as CFO anytime soon, but it’s not all that complicated. 

We’re researchers, we understand different ways of knowing the world all the time and see through the eyes of others – just see through the eyes of your CFO, an investor, a stock market analyst, etc. 

Once you understand, you need to use that knowledge to specifically motivate the research questions and findings you have. Tie your research questions directly to places in the conversion funnel or to new opportunities because you, say, read the stakeholder letter or the competitive report. Use what you learn to motivate your work. 

Is the current job market a blip or a trend? How should researchers better position themselves in this market? 

The job market is tough – there are fewer jobs and many of the jobs out there have skewed more senior, which reflects a fundamental mistake that many executives make by assuming you only need highly experienced people to do research jobs rather than a full range of people from all careers levels and skill sets. 

The number one thing you can do in any environment, but especially in this one, is to figure out what your unique angle is on being a researcher. 

Think of the hiring manager that sifts through probably 100 CVs and portfolios in as little as an hour. Every one of those CVs is from an experienced multi-method, empathy-first researcher with a deep commitment to high-quality products and user perspectives. It’s not wrong, it’s just boring. 

The first thing you can do to stand out is to figure out your special sauce. Think of yourself as a product person who happens to do research and then pitch yourself that way because increasingly companies, especially small- and medium-sized companies, are looking for people who are less stuck in their lane. They want Swiss Army knives. They aren’t just researchers. You aren’t just a researcher. You’re someone who works on a product in a business. 

{{quote-juddama-4="/cms"}}

That’s why I think business knowledge can really help researchers. So when you’re pitching yourself, the point is not just to do research, it’s to build a great product and therefore a great business. 

Where should research sit in an organization?

The real answer is: It depends. 

I used to have a stronger opinion about this but then I became familiar with the vast organizational diversity out there. Research reports into different functions in different places depending on the team. I don’t think there’s one clear answer, but my main thought is that research should report into the part of the organization that has the most power. 

It really doesn’t matter exactly what research reports into, we can adapt as a practice for what the company needs at the moment. My question is: Who is going to be your executive advocate? Who is the person who holds the purse strings, who has the voice in the C-suite room, and will bang the drum for you and your work? 

That’s who you want to report to. And it doesn’t really matter where that person comes from – it mostly matters that they control the funding and have your back. We shouldn’t be precious about it. Let’s position ourselves in a place where we’re best positioned for success.

What are your thoughts on democratization?

I think the fear about democratization that most people don’t articulate have is the fear of losing control of the way we generate value. Many people worry if all research doesn’t run through them, then no one will need them and then their job is at risk.

Democratization of research is inevitable because there are now and forever will be tools that allow everyone to participate more fully in and conduct aspects of research. So, get on board. 

It’s important to note that we’re talking about the production of raw materials, not what you do with them. This is partly why I think the most important thing is deeply integrating researchers in the end-to-end product development process. The value you provide is not only or even primarily from your ability to produce primary research materials, it’s from your role as an active participant in the decision-making and design process. You can’t democratize that. 

When you democratize research, create a structure and a set of relationships that are trusting and influential and put yourself in a decision-making role because you can never outsource an insights professional’s seat at the table. 

Some people worry that research will become a commodity, but isn’t that a good thing? Commodities are just things that are common and interchangeable and exchanged widely. Not all aspects of research should be commoditized because there are things that don’t scale. But there are parts of research that are much easier to democratize and we shouldn’t be afraid of that.

We should embrace democratization and figure out how to be the source of insight and own decision-making and build impactful relationships, even in a world where there are research insights coming from everywhere. 

{{quote-juddama-5="/cms"}}

How can a single researcher make a big impact without the large team or resources to be involved in every discussion/decision?

I speak to so many researchers who are in this spot, maybe they’re on an island or maybe they are just literally the only one. The best advice I can give is to create pain by focusing on no more than three things. These should be the most important things, and you should not be deciding what they are. Find a way to rely on a set of company priorities that stem directly from your managers, director, or executives. 

Prioritize your work based on this list of three things and make it clear that your full load is maximum two to three projects and that you prioritize these projects based on leadership and business priorities. 

A lot of researchers have problems with this because they want to wrap their arms around everyone. You need to confidently state to whomever is asking you to do more or cover an area too big that this is a strategy for failure and that there is no way to do a good job. 

For growth, my strategy was always that I didn’t want to be the one advocating for headcount. I want the people who felt underserved to see the impact that was happening in research. I wanted them to go to their bosses and, for example, the engineering director to be willing to convert one of their engineer head-counts into a researcher. That happened to me maybe 20+ times. 

Why is it important for traditional qualitative researchers to transition into multi-method individuals? 

The fundamental reason why to become more multi-method is because if all you have is a hammer, then everything looks like a nail. If you have a fuller suite of methods, you can combine them in different ways to answer more questions with research. 

Here are six tools that a well-rounded research needs:

1. Formative qualitative research
2. Evaluative qualitative
3. Survey design
4. Basic applied statistics 
5. SQL query proficiency / data infrastructure navigation
6. AI tools

Most of us work in industries that have a global or wide geographical scale and a large amount of data about customers and potential customers. If you can’t interact with a global audience, you can’t interact with the data that companies are brimming with. To me, that’s the fundamental argument for being more T-shaped. 

How do you start to learn? YouTube. I’m not kidding. YouTube is full of wonderful educational material about all of these methodologies and it’s a great place to start. 

How can you make research less boring?

Swearing a lot. ;)

One thing I do all the time is use MidJourney. You have the power now for a low low monthly price to create the visual aid you need for your research communication or your report that is going to convey your message in just the right amount of emotive detail or spiciness. 

If you get good at prompt engineering, you can get exactly what you need. If you’re investing in better research communication and storytelling – which can make research less boring – then you’re isolating an insight per slide with one amazing visual that makes the point of why it matters. 

{{quote-juddama-6="/cms"}}

My tip? Use “sticker art” as the style because the images are often bounded in a way that makes it easy and quick to put them on a background. 

What is your view on centralized vs. embedded researchers? 

I’ve built teams that are both and it’s important to know the pitfalls of each. 

Embedded: When researchers are too embedded, everyone gets siloed. When researchers are deeply integrated with their product team, their designer, their engineer, they are filling up their roadmap, but not necessarily with stuff that’s important. Other potential side effects on your researchers include:

• They get lonely.
• They lose a community of practice around a specific product area.
• They aren’t necessarily working on the most important things.

Centralized: You lose the ability to build context on parts of the product. It’s harder to build relationships in that situation and to be trusted when you’re just parachuting in like some kind of medic onto a battlefield. But, you can also move around to the top priorities. 

Recently on Lenny’s Podcast, he had on Elizabeth Stone, the CTO of Netflix. She used to run consumer insights, which basically meant everything under the sun. People would do media research, content communication, consumer trends, landscape marketing, along with UX Research. She was discussing how beneficial a centralized team can be. But I would say centralized teams only work if you have an engaged and knowledgeable leader like Elizabeth who can properly steer that ship. 

Ultimately, there’s no perfect solution, you just have to know the pros and cons of each and choose what fits your organization best. 

How can research become central to decision-making in organizations where it's already valued & well-positioned?

Stop talking about our job as researchers as influencing decision-makers and decisions. That’s not what we want to do. We want to be the decision-makers. 

• Learn the language of the business. It can be hard for folks to think you should be in the room as a decision-maker when they just see you as an influencer from the outside advocating for users. That’s not our role. Our role should be as matchmakers between what the business needs and what the users need. To do that, you’ve got to understand both sides of it. 
• Spend less time focusing on primary research and more time on relationships. The most successful researchers and research leaders I’ve seen deeply understand the needs of those they are working with and build a relationship that makes it impossible for them not to be included in decision-making moments and meetings. When that happens, you’re no longer on the outside looking in, trying to be influential – you’re in the room being a decision-maker, directly participating with your colleagues. 

{{quote-juddama-7="/cms"}}

We should still stay user-focused, but not uniquely user-centered. 

Where do you think research leadership has failed the most to get the industry to this state? How can they fix it? 

The biggest failure I see in research leaders overall is the failure to shift from a focus on research practice to focus on product leadership. One thing that made me successful at Airbnb was I started as Head of Research, but I also had a variety of different jobs. I was an interim general manager for a product group and I was head of the design studio when I left. 

When you are a leader, it actually does a disservice to your team to be primarily talking about research and your research practice, because you are the only person in that leadership position. You are the only person who can form certain relationships, convey certain information, and translate it effectively. 

{{quote-juddama-8="/cms"}}

You are not just a functional leader, but a partner in building product and business. You can earn credibility for your team. Building out Research Ops, advocating for research communication practices, and creating skill frameworks and ladders are important, but it’s not necessarily the unique value you can add. 

Judd’s brief thoughts on AI

Think of AI as an amazing on-demand research assistant, not as a replacement for any core research activities. I think some of the anxiety around AI is fueled by hype and the hype does not match the reality of these products today. AI will not replace a researcher, AI today can barely be a partial, terrible researcher.

⚡️ Lightning Q&A

What is your perspective on future-proofing the ReOps discipline?

1. I think ReOps in the future can focus on things that can’t as easily be outsourced. For example, integrating disparate tools into one system, running efficient processes internally, and doing it with personality and culture that’s unique to a company and their users.

It is hard for product teams to differentiate between well-done research vs poorly done research, if the insights are communicated in visually stunning way. Do you think this is a problem in the long term for good quality research?

• Maybe, but right now I rarely see this problem. Mostly, researchers, good and bad, aren’t nearly good enough at communicating clearly, compellingly, and contextually appropriately. I’d love to have the problem where we get so good at communicating that we can pass off bad work.

If designers are also doing user research but are active in using those insights to influence product and design decisions, where can a User Research team provide value?

• Sounds like a confusing structure. Role specialization is usually fantastic for efficiency and quality, but in some cases leaders haven’t decided on the structure they want. The worst thing is doing a job without clear expectations for (among other things) scope and focus. It’s fair to ask your leadership to work through that with you.

Should we see democratizing research and continuous research as a "threat" or a great support and way to include our team in learning about their target group?

• IMO, seeing it as a threat comes from insecurity about our role, and a desire to maintain control of research. That’s silly, because powerful tools for democratizing research are here to say. No sense screaming at the wind. Get on board and add your unique value on top of those tools!

Considering the performance evaluation heavily weighs on strategic contributions, what advice would you give for navigating this scenario where the role's structure seems to inherently limit the potential for recognized strategic impact?

• This is a sadly common situation. It my be because research leadership has institutionalized a fantasy about what “strategic” research looks like, despite the inability to actually do it. Or it may be because of a poorly structured product/design process, or a misalignment between research and other functions on what constitutes value. No matter what, this is a terrible situation to be in, and one that leadership needs to resolve not on the basis of what “should be” but what actually is.

Can you share a concrete example of how you’ve seen qual-oriented teams and quant-oriented teams work really well together?

• Survey scientists + qual UX research — IDIs reveal rich insights and hypotheses which don’t scale (because that’s not possible given the method). Survey scientists work with those researchers to develop a larger-scale survey to check generalizability. I’ve seen this more times than I can count.
• Data scientists + qual UX researchers — Results of an A/B test defy the expected logic, are confusing or contradictory. Qual folks jump in to follow up with research that helps explain the hows and whys of what the experiment is showing. This is a beautiful reason to make your local data scientist your BFF.

When we do good research and “falsify” a team’s beliefs — the message isn’t always well received. It’s jarring and hard to deal with, even though we know we can’t sugar coat the truth. How should researchers deliver negative results? What are their responsibilities, and how should they communicate effectively without upsetting others?

• The trick to this is not leaving it to the end. Talk up front about what it would look like to falsify a team’s assumptions. Start to create a culture where that’s normal and even desirable - because it’s what good science is! And then never wait til the end to drop the bad news (or good news, or any news!). Ideally your team was with you all along the way, but if not make sure you’re sharing back with them and discussing on a regular basis, not just at the end.

What's your advice for being a first researcher in general, and for integrating research culture into a highly engingeering/technical culture? 

• Find your biggest advocate, most engaged partner. Form a strong bond, and a consistent relationship with them. Solve their problems, scratch their itches. Make their life easier. Don’t do work ONLY on the basis of what you think ought to be done, but also what will drive the most impact via that relationship. Build your strong voice with them. Demonstrate the impact in that narrower space, and it becomes the best possible argument for integrating research elsewhere.

Connect with Judd

If you enjoyed Judd’s AMA: 

• Check out his website and writing. 
• Give him a follow on LinkedIn and introduce yourself!

Thank you, Judd!

We’re grateful to Judd for joining us and sharing his insight, experience, and passion for research, product, and design. If you’d like to watch the full webinar, follow this link.