On Jan 26, we hosted 4 Research Ops leaders who helped build Airbnb’s ReOps team, which is one of the fastest growing in the tech industry. Research has always been at the core of decision-making at Airbnb.

The team covered topics like building impactful resources for researchers, scaling and elevating Research Ops, increasing awareness, and driving impact.

If you missed it, or want to revisit the gold nuggets the team shared, we’ve put together highlights from some of the topics they covered. If you’d like to watch the full event, head to our YouTube channel.

Key Takeaways

🔑 When building processes for researchers, focus on what reduces their anxiety and makes their lives — and yours — easier.

🔑 When thinking about scaling, make sure you know exactly what scaling means to you, your team, your stakeholders, and your org.

🔑 Creating a mission statement can help spread the definition of Research Ops and make its value clear.

🔑 When determining impact, it’s also important to pick what you want to do and share it with people

🔑 Be champions of your own work — research and Research Ops shouldn’t be invisible things happening the background.

Who were our panelists?

Tim Toy, Research Operations Manager 🟡 Tim got his start in Research Ops at Adobe in 2007 before people were calling it Research Ops! He went on to start, build, and manage Research Ops teams at Electronic Arts and Airbnb from 2010 - 2021. After Airbnb, Tim turned his career to an IC path in Research Ops at Splunk and now is currently back at Adobe!

Eva Frieden, UX Research Operations 🟡 At Airbnb, Eva focused on the Participant Experience, supporting process improvements throughout the Recruitment Pillar; as the first ResOps hire at HelloFresh, Eva has built the ResOps Program from ground up, enabling research at scale as the Global UX Research Team has tripled in size!

Wyatt Hayman, Program Manager, Research Team 🟡 Wyatt joined Airbnb in 2015, starting as a Product Specialist before moving onto the Research team where he built out Airbnb’s International Community Panel to enable multilingual, global, and qualitative research. This eventually grew into a part of Airbnb’s newly formed Research Operations team. Since Airbnb, he’s built research ops programs for Adobe and currently manages Research Ops at Faire.

Joey Encarnacion, Program Manager, Experience Research Operations 🟡 Joey joined the Research Ops team at Airbnb after a long stint in the Trust & Safety space. He’s passionate about innovation and has carried that through his times at Airbnb, Slack, and now Twitch.

Oren Friedman, Co-Founder and CEO of Rally & Panel Moderator 🟡

If you’d like to connect with any of our panelists or have more questions for them, you can access their contact information at the bottom of this recap.

The early days of Airbnb Research Ops

When Tim Toy joined the team, Airbnb’s 40 researchers were being supported by one Research Ops person: Liza Meckler. “A 1-to-40 ratio is not super great,” said Tim. Liza was doing 80-90 hours of work every week supporting things like recruitment, equipment management, and lab  management.

“A lot of the responsibilities were being dispersed throughout the rest of the research team,” said Tim. “So everything was a little bit muddled and confused.”

As a small Research Ops team, Joey Encarnacion said he, Liza, and Tim first focused on re-centralizing everything. “This created the expectation that you have a dedicated team to help you get studies off the ground in a much more efficient and effective way.

“We wanted to make sure everyone had a role and responsibility,” said Tim. “Liza focused on participant recruitment, Joey was the mad scientist building up all his technological wonders, and then Wyatt joined us later to handle panel management and brought on international panels for all of Airbnb.”

The team was able to accomplish all of that in about one year. They were then able to expand and bring others, like Eva Frieden on. As Airbnb grew, so did the research team, jumping from 40 to nearly 70 researchers in just one year.

How did Joey go about building impactful resources and being a “mad scientist”? 😉

“I focused on the things that annoyed me the most,” said Joey. “Mostly it was people grabbing gear without signing up for it, usually resulting in the gear going missing.”

That eventually led to Joey focusing on the most immediate problem that was bothering the team. “A big part of it was just having empathy for researchers and what they go through,” he said. “Not a lot of people really understand just how hard research can be.”

Back in the early days of Airbnb, there was still a lot of field work being done by researchers. This would involve going out somewhere you’ve maybe never been, carting tons of gear, talking to people you’ve never met, and having to spend a lot of time with them. After the session, you’d have to make sure you got the needed footage, took notes, and then later you had to communicate with stakeholders. “I really wanted to focus on things that made researchers’ lives easier,” said Joey. “And that was my Northstar.”

{{quote-joey-1="/cms"}}

And the goals of the Research Ops team weren’t just about making the researchers’ lives easier. “We also wanted to make sure our lives were easier and doing both in a sustainable way that was setting ourselves up for success,” said Tim.

Examples of how the ReOps team improved researchers’ lives

“I try to think in the mind of researchers,” said Eva. “If I were a researcher, what am I doing today that I would rather not do?”

Other things Eva shared that Research Ops can do to aid researchers include:

• helping build the research plan
• recruiting participants
• paying incentives
• making sure researchers have the right tools

“There's a lot that goes into each research project,” she said. “Research Ops is behind the scenes and optimizing what researchers are doing to help manage the amount of work and moving pieces that go into every project and study.”

{{quote-eva-1="/cms"}}

One thing Eva and the Airbnb team did was hold regular meetings where everyone could discuss the research they were doing. It was a way to increase transparency and also ensure that things weren’t falling through the cracks. This was especially helpful since Airbnb had researchers embedded in various product teams and communicating regularly with different stakeholders.

Wyatt joined Airbnb to help build an advisory panel to support Airbnb’s goal of listening to hosts. It was decided that Wyatt should turn what he was building into a real research tool. In doing so, he spent a lot of time investigating the problem space.

Wyatt realized that it was difficult to reach international participants, something that was important to Airbnb, which was focusing on global belonging and true local experiences. “We were almost exclusively doing research in English and the United States,” he said.

Essentially, what Wyatt built solved a problem while also enabling a new type of research. “We continued to iterate on that first panel we put together, and it became different over time as we learned more.”

Challenges and lessons from scaling Research & Research Ops

“One of the biggest learnings I took away was that it's important grow sustainably,” said Tim. “Since the research org was growing so quickly at Airbnb, it was difficult to maintain that pace with the researchers.” The ratio of 10 researchers to 1 ReOps only got more difficult as the team continued to grow. "We had to figure out how to step up quickly," said Tim.

When scaling, it’s important to understand what scaling means, said Joey. “When I was at Slack, I had a very difficult set of conversations where I had to ask what scaling research meant to certain people.”

For example scaling research could mean many different things like:

• more research, period
• research done faster and easier
• more research consumed by stakeholders
• more participants reached out to

“This is a trap that anyone can fall into,” said Joey. “You have to be clear on the definitions.”

Tim said it’s important to get research leadership to understand and have a philosophy around what Research Ops is and does, especially when you’re scaling from a team of one. “Figure out what is most crucial to your specific set of researchers and your team within your organizational context.”

{{quote-tim-2="/cms"}}

Utilize Emma Boulton’s Eight Pillars of Research Ops and make sure you know what is relevant for your organization, he said.

🌶 Spicy Take from Joey 🌶

“Another trap that folks can fall into is the idea of democratization of research,” he said. Though Joey is not against democratizing research, he said it’s important to be very clear on what democratization means.

It can seem like a great idea to give stakeholders the tools to reach out to whoever they want, but the “trap there is that stakeholders have their own jobs.” Designers should be spending time designing, product managers should be spending time product managing, he continued. “Giving them the tools and saying ‘you do it’ is not fair to them and undersells how difficult research can actually be.”

Self-service programs can be great in the interim when you’re building your team or getting more people on board, Joey said. “But I don’t think it’s a good permanent solution.”

How to elevate Research Ops as a strategic partner NOT an admin

“At Airbnb, people really liked our first attempt at international community panels,” said Wyatt. “We learned that teams wanted panels set up specifically for the research they were conducting that would support their needs and what they were investigating.”

So, Wyatt set up conversations with the leaders of those teams. “It was really exciting because it put research at the table.”

When scaling, make sure each person who joins the team knows their role and responsibility, Tim recommended. “We really tried to make sure everyone within our Research Ops team was aligned to a pillar. For example, Joey ran the technology pillar, Wyatt ran the panel pillar, and Eva ran the recruitment pillar,” said Tim. Every one of their direct reports, depending on their interest, was able to align themselves to a pillar. This helped Research Ops not feel like an admin job.

Another thing the team did was create a really defined service-level agreement for every program they created that would explain what Research Ops role was for the program, where the role began, and when it would end.

How can you measure the impact of Research Ops?

“It’s not easy,” said Eva. “But being able to show impact and value helps make the case for scaling and increasing headcount.” Within an organization, many teams will have their own dashboards that show tangible, quantitative impact. “How can we create a dashboard that captures how successful ReOps is?” she asked.

For a lot of Research Ops, impact is word of mouth, said Eva. Are researchers coming back for more help? Are people sharing with other teams how Research Ops helped them?

“A big portion of the process of determining impact is picking what you want to do and sharing it with people,” said Joey. “We put out a very clear vision and mission statement around our team that encapsulated some of the different pillars within Research Ops, defined what Research Ops is, and what the team does.”

{{quote-joey-2="/cms"}}

Then when you have a mission statement and clear vision, you can more easily say “this is what we do, if it’s not under that, then we don’t do it.”

Tim shared some good questions that can provide some Research Ops metrics:

• how many studies do we support?
• how many participants showed up to sessions?
• how many participants canceled?
• how much incentives did we send out?
• how much did we spend?
• how many tools got checked out?

“Answer these questions monthly and keep the reports in your back pocket for when you want to talk about scaling,” said Tim. “You can use these numbers to make the case for growing the Research Ops team, so that research can continue being supported.”

Many Research Ops teams are scrappy and expected to build quick solutions. But scrappy solutions, though important and often effective, likely won’t scale. So how do you transition scrappy solutions into a scalable function?

“I don’t have a golden answer, but I do think there’s a lot of value in getting something set up to kind of give people a taste of what Research Ops can do,” said Wyatt. “Use that as a bargaining chip to have a structured conversation and seat at the table where you can advocate for more resources instead of just keeping it growing organically.”

{{quote-wyatt-1="/cms"}}

How to get started on reporting impact

“When you’re starting out, you’re like an archaeologist digging for facts and clues around what happened in the past,” said Tim. “You’re trying to get a sense of the landscape of research within the team and organization.”

Tim says even if you are not really big on producing KPIs or tracking, you can still track very simple things like what he listed above. “They’re very simple but very powerful to track over time. It helps tell a story around how you’re doing, your efficiency, and whether Research Ops is making things better.”

“Research Ops has a responsibility to be almost an air traffic controller, keeping a good view and awareness of what’s going on,” said Tim. One metric that Tim and the Airbnb team would constantly share with leadership was the percentage of studies Research Ops was supporting.

{{quote-tim-3="/cms"}}

What did the Gang learn that they have brought into their new roles to drive impact?

🤝  Invest deeply in alignment and getting people to understand what you can potentially be doing as a team. - Wyatt

“One of the biggest challenges of Research Ops is resourcing,” he said. “That partly stems from lack of understanding of what Research Ops actually is, meaning the people making decisions around resourcing don’t necessarily have the ammunition to fight for the resourcing you need.”

📣  Be champions of your own work — research and Research Ops shouldn’t be invisible things happening the background. - Wyatt

"A lot of people in research roles are used to and comfortable with working in the background — that's often what led us into this work,” said Wyatt. “So you just have to either find that voice or find a partner within your company that can support you in being that voice. Otherwise, you can't really get your message across, and you're never going to get the resourcing or headcount that you need.”

🔢  Metrics give visibility into impact. - Wyatt

“People don’t drill down into the details of Research Ops because they don’t have time for it,” said Wyatt. “They just want you to figure it out.”

“If you make the case that Research Ops makes sure that research is conducted in a way that’s compliant and following best legal practices… this is a pressure you can put on people to understand the need for ReOps.”

⚖️ Utilize the need for compliant and ethical research to build awareness. - Wyatt

“If you make the case that Research Ops ensures that research is conducted in a way that’s compliant and following best legal practices — this is a pressure you can put on people to understand the need for ReOps.”

📐 Set expectations up-front. - Tim

After leaving Airbnb, Tim joined Splunk as a Research Ops team of one. “One of the first things I did was set up a co-creation exercise with the research leadership and researchers to make sure that the support Research Ops would be giving aligned with what the researchers needed.”

The researchers at Splunk had never worked with Research Ops before so Tim set up a document that set clear expectations: “I’m not your personal recruiter, I’m not going to book your labs, etc., but here’s what I am going to do. And if that’s a problem,” he said. “Let’s talk about it. Then when you’re three months down the line and being overwhelmed with requests, you can confidently identify what is your job and what isn’t your job.”

“It allowed all the researchers to feel like they had some buy-in and a better understanding of what I was doing,” said Tim. “It gives a sense of ownership both to you and the researchers when they clearly know their role and responsibilities.”

👏  Involve everyone when you talk about expectations. - Joey

“Co-creation is the most alignment-generating tool that I have used throughout my time at both Slack, Twitch, and Airbnb,” he said. “When you bring everyone together to talk about expectations, you provide an opportunity for everyone to help build the team.”

{{quote-joey-3="/cms"}}

👨‍⚖️  Don’t get sued. - Tim

“At Airbnb, one of our mission statements was ‘don’t get sued or fired,’” said Tim. “GDPR and the onslaught of new rules and regulations were incredibly confusing to us and our researchers. We decided we needed to do something.”

Together, he and Liza Meckler built a Compliance Alliance, which included a small group of researchers that were particularly interested in the legal side of things. They would meet with the lawyers every two weeks to ask questions like:

• what are the topics that are important to us?
• what do we need to stay aware of with GDPR?

“It was a nice opportunity for Research Ops and researchers to come together in a partnership and drive something forward for the greater team,” he said. “It was also a way for Research Ops to be proactive and strategic and give researchers what they need before they needed it, instead of constantly being reactive.”

👷 Build programs. - Joey

“At Slack, our mission statement is ‘building programs and products.’ Though there is some admin work, the majority of what we do as Research Ops is build products and programs that help researchers do their jobs,” said Joey. “If I look at Research Ops from 10,000 feet up, that’s really what it’s about. You’re building the infrastructure to enable research to happen, not necessarily doing all of the work on the ground.”

{{quote-joey-4="/cms"}}

🗣  Spend time talking to your researchers and stakeholders. - Eva

“Many of my first months at HelloFresh were spent talking to researchers, cross-functional partners, product managers, and folks in Customer Care,” she said. “I kind of felt like a saleswoman bringing a presentation to explain what Research Ops is.”

At the same time, Eva said she was getting to know them, hearing their pain points, and then helping inform what Research Ops should focus on. “Having these conversations can help you know where to prioritize your time,” she said. “It can also be a great opportunity to connect, chat, and get people excited about research roles that they may have never known about before.”

Relationship between Research Ops and Design Ops

“One of the great things that you can do with DesignOps is to align on the product development process,” said Joey. “Of course, it would be great to get Product Ops in the room as well.”

Doing this ensures that products get built and launched, he said. Design can also help educate on when design work should be involved. “If you can generate alignment between the design process, the product process, and the research process, you end up being able to hit the marks at the right time, which tends to be a very difficult problem in the research space.”

{{quote-joey-5="/cms"}}

Ideally, Tim said, Research Ops should be a bridge between Design Ops, designers, and researchers and help reduce friction. “We haven’t gotten there yet at Adobe, but it’s something that I’m aspiring to.”

Eva recommends meeting with members of teams like Design Ops once a month. This proved a great time to connect over similar problems, workshop issues you may be dealing with, and hear each other’s thought processes. “Being able to have that outlet to chat was really enjoyable.”

What is the number one point you’d use to articulate the value of Research Ops?

🥇 Researcher retention - Wyatt

🥇 Not getting sued, meaning making sure that research is done in an ethical and compliant way - Tim

🥇 High quality research is being produced and research team is growing, happy, and high-performing - Eva

🥇 Describe what it’s like to do research, and how Research Ops reduces that anxiety - Joey

Scale Research Ops with Rally

Choosing the right tool can help you to scale your Research Ops and build processes that are efficient and effective. At Rally, we’ve built a User Research CRM that helps teams of all sizes manage each stage of the Participant Management and Recruitment process. If you want to learn more, join our waitlist!

We LOVED having Wyatt, Eva, Joey, and Tim join us to share their experience and insights from Airbnb and the roles they’ve had since. If you’d like to watch the full webinar, head to our YouTube channel.