Lawton Pybus on how UXR teams hire today

AMAs

March 28, 2024

Lawton Pybus on how UXR teams hire today

Lawton Pybus, User Research leader and consultant, joined us for a Rally AMA on March 28 on how UX Research teams hire today. If you missed our live event, or want to revisit the highlights, read our recap below. If you’d like to watch a recording of the full AMA, follow this link. Continue reading for Lawton’s thoughts on topics like building your UXR portfolio with and without research experience, searching for new roles, and how to improve the hiring process.

Who is Lawton?

Lawton is a co-founding principal of Drill Bit Labs, a boutique consultancy dedicated to helping teams do better user research. With a PhD in human factors and experience at UserTesting, MeasuringU, Charles Schwab, and Cabela’s, Lawton brings practical knowledge, academic rigor, and strategic insight to the table. He is known for his contributions to the UX community through his newsletter, The ¼” Hole, and speaking at industry events like UXPA and Boulder Startup Week.

Lawton’s research on the UX Research hiring market

When it comes to the UX Research hiring market, my co-founder Thomas Stokes and I conducted three efforts:

Started tracking inventory. Since the summer of 2021, I’ve been going into Indeed once a month searching specific terms and noting down open UX Researcher roles. This has resulted in a pretty robust database.
Conducted a series of studies. Thomas and I did qualitative interviews as well as a larger-scale survey with UX Researchers, hiring managers, and job seekers to determine the state of the hiring process, specifically what it looks like and the common pain points. We presented this research at UXPA in Austin last year.
Collecting and analyzing job descriptions. For this ongoing project, we’ve been systemically tracking job descriptions. We currently have over 1,000 that we’re hoping to analyze to determine what these job descriptions say about the role itself along with career progression.

What activities are commonly included in User Research job interviews?

There are 6 activities that are really common. These activities don’t necessarily each constitute a point of interaction or “round” of interviewing. So, you may have only three rounds or touchpoints in the interview where you're actually talking with the company, but these activities tend to get combined.

Phone screening typically with a recruiter
1-1 interview with a hiring manager
Technical interview (ex: Tell me about a time that you did a usability test, how did you determine sample size?)
Behavioral interview (focuses more on interpersonal abilities like how you collaborate and work with a team)
Panel interview (typically with hiring manager, members of the research team, cross-functional partners, etc.)
Work sample test (popular formats include portfolio review or take-home test)

With our research, Thomas and I wanted to better understand the disconnect between hiring managers and job seekers. Both hiring managers and job seekers likely have their own incentives in this process and we wanted to discover those and learn why the two parties are not aligned. For example, hiring managers may be invested in having a really long interview cycle because it reduces risk for the company. Meanwhile, a job seeker is going to want to get a role as quickly as possible and would often prefer a shorter hiring process. In our original research, 60% of those surveyed – both hiring managers and job seekers – wanted three-round hiring processes, quite similar to the results of this live poll.

This poll was conducted live during the AMA and had 52 respondents.

Most people do want a streamlined process, but unfortunately there are many things that get in the way of a quicker process such as department mandates. What we discovered through our research is that ultimately hiring managers and job seekers have many shared interests. Hiring managers want to find the best person for the job and job seekers want to show that they’re qualified for the job. We ought to lean into that more as whether we’re seeking a new job or finding the right candidate for an open role.

What are the main points and areas for improvement in the UX Research hiring process?

The biggest pain point we found through our research was that for job seekers, the hiring process is very cumbersome. Job descriptions are unclear, application processes are often redundant or require unnecessary, time-consuming steps, and for many people, it’s difficult to determine if they are actually a good fit for the role.

Many people reported feeling like the hiring side of the process isn’t well put-together. And hiring managers reported often not having the best relationship with the recruiters they are working with, which can further complicate the process for both the hiring side and for job seekers. Recruiters may not fully understand the exact needs of the team and what makes a good candidate vs. a poor candidate. We also found that most hiring managers, while they probably exhibit great technical skills and a fair amount of soft skills, they don’t have formal training in proper interview procedures.

What advice did Lawton collect from his research for hiring managers and job seekers?

The biggest theme we saw was that every interview goes two ways – the hiring side is evaluating you as a candidate and you’re evaluating the company. When interviewing, you should also be determining if the role and company are a right fit for you and your career. This requires some prep work and determining the questions you want to ask to confirm the role is a good fit for you.

For those hiring, make sure to leave time for candidates to ask questions and properly evaluate the role and company for themselves. It’s smart to build this into your process and to continually remember that the hiring process is a two-way vetting process.

How can interviews be structured more effectively to assess a candidate’s problem-solving approach?

This poll was conducted live during the AMA and had 42 respondents.

Take-home tests, white-board exercises, and portfolio reviews are all examples of a work sample test. They are the most valid part of the hiring process because you're showing that you've done this before and it has empirically been shown to predict future job performance. Here’s a little more about each one:

Take-home tests: Assignments candidates complete on their own time, demonstrating their skills and problem-solving abilities relevant to the role.
Whiteboard exercises: Interactive tasks where candidates outline, develop, and explain their strategies for a specific problem or task in real-time during the interview.
Portfolio reviews: Presentations where candidates present and discuss their past projects/work.

While a take-home test can be a great predictor of success and a great representation of what the candidate will do in the role. The problem is that it doesn't transfer. Often, all of the time and energy that a candidate puts into a take-home project is sort of lost.

While a whiteboard assignment is similar where you may lose the time and effort, it is often less time-consuming and intensive as a take-home assignment. Whiteboard assignments do tend to reward people who think quickly on their feet and can improvise. While it’s great to have those skills, it can be difficult if those skills aren’t critical to the role.

The portfolio review or case study presentation was by and large the most preferred one by candidates we surveyed. The problem for hiring managers is that it may or may not cover the kinds of experiences that are interesting for the role depending on what the candidate chooses. I think this is another area of opportunity, especially for hiring teams, but also for candidates. When you know you’ll be doing a portfolio review, clarify with the hiring manager what they are looking for so you can prepare relevant case studies.

How many job applications do User Research professionals typically submit?

This poll was conducted live during the AMA and had 44 respondents.

We found that if someone is actively looking, that generally means that they are quite active. So we're having the results come in about 60% or more of our sample had applied to over 25 rolls to get their last roll.

We found that 60% or more of our sample had applied to over 25 roles before landing their most recent job. In terms of a conversion rate, regarding how many job applications you need to submit to get a job offer, what we saw from our research was 1:20, though there was a wide variance. Some people have success with fewer applications, whereas for some it will take a lot more. For my first role as a researcher, I had a spreadsheet that I kept almost like a CRM tracking how far along I was getting for each role. I think it took me well over 100 applications to get my first internship and I don’t think that’s uncommon.

As you move forward in their career, you’ll likely build a deeper network and be able to better leverage things like referrals. Your network will hopefully provide you an advantage and even provide a foot in the door for some roles.

What are the most effective ways for User Research professionals to find job opportunities?

LinkedIn and other job boards produce the most leads, but a referral will give you the highest quality leads. What people surveyed rarely said they did was go directly to a company page unless they were obsessed with that company. Generally though, LinkedIn and other job boards like Indeed are how most User Research professionals are finding roles. If you know someone from a past role, there’s no harm in reaching out and seeing if you can leverage that connection for even a little bit of an edge in the hiring process

What are the biggest pain points of job descriptions?

I think the biggest one is that it's hard to discriminate between, for example, a senior researcher at company X versus a senior researcher at company Y. Both jobs will require things like 5 years of experience and the ability to do qualitative and quantitative research. But it doesn’t necessarily describe the products you’ll be working on, the composition of the team, the organizational structure/hierarchy, or stakeholders/cross-functional partners you’ll work with. Ultimately, most job descriptions end up sounding the same.

On the hiring side, we're sort of inviting people to just apply scattershot volume instead of reading the job description closely and seeing if they are a really good fit and self-selecting. Instead, most candidates are just skimming through job descriptions and shooting off applications. I think there is a huge opportunity here to construct job descriptions that are more specific to the role so that candidates can genuinely assess themselves ahead of time.

What advice does Lawton have for job seekers?

As someone who's been on the hiring side, I've noticed I get a ton of direct messages from folks wanting to set up a call. While I get where they're coming from, I'm all about keeping the hiring process fair for everyone, so I don’t always respond to those messages. That being said, I'm totally open to answering questions about the job and the job description. I’m always happy to clear things up in an email or private message.

If you're on the job hunt, definitely use tools like LinkedIn. Sometimes you can catch hiring managers posting about the job there. And if not, maybe you know someone who knows someone – networking like that can really work in your favor. Doing a bit of digging and reaching out in the right way can really make a difference. It shows you're keen and you've done your homework, which is always a good look.

What are some best and current practices for building & evaluating portfolios?

This poll was conducted live during the AMA and had 54 respondents.

In our research, we discovered three conflicting truths about UX researcher portfolios:

Clarity is often missing in job descriptions: Only about one in five job descriptions explicitly mentions the need for a portfolio or case study review. This leaves some uncertainty about whether companies actually expect this from candidates.
Readiness of portfolios varies among researchers: When we asked researchers if they have a portfolio, the results varied. A decent number have an up-to-date portfolio, but the majority either have an outdated one or don't have one at all.
Portfolios hold significant value in the hiring process: Despite the ambiguity, the work sample test, which includes portfolio or case study reviews, is crucial in hiring, being one of the top six activities. While this is extremely popular, most UX Researchers seem unprepared for it.

A great way to build and maintain a quality portfolio is to update it as part of your regular project close-out process. Not every project makes it into the portfolio, but if a project had a significant scope, involved strategic work, employed a unique methodology, or presented a challenging problem that spurred growth, it's worth considering.

When a project concludes, while the details are still fresh, sanitize the materials for the portfolio. Given that most work is under NDAs, the focus should be on methodology, research questions, and thought processes rather than sensitive specifics.

Finally, you don't need to be fancy with the presentation. As researchers, our work often exists in reports, so whether it's an email, a presentation deck, or a Word document, just make sure to transfer and sanitize your work for the portfolio. Incorporating this step into your regular workflow is a strategy that the most successful professionals follow.

Should you tailor your portfolio to the roles you’re applying for?

Tailoring your portfolio for specific roles you’re applying to can be beneficial, and it relates to understanding job descriptions, which are often vague. The better the job description, the better you understand the role. If possible, ask the hiring manager for guidance and inquire which of your projects would most resonate with them or align best with the role’s demands.

A practical approach is to have a selection of about six projects ready to showcase. This way, you can selectively highlight the work most pertinent to the position you're targeting, ensuring your portfolio is tailored and relevant to the interviewer's interests and the job's specific needs.

How can you showcase your abilities when you don’t have clear UXR experience?

It's a little bit of a chicken and egg problem, right? You need to get a role to develop work experience, but to get that role, you need to have work experience. However, there are ways to showcase your capabilities even if you haven’t had research experience.

A great strategy is to repurpose your work. Maybe you worked in a tangential career, like as a market researcher, or maybe you did some school projects as an academic. You can repurpose those, focusing on the things that are relevant to your research.

Another thing is to teach a UX course. When I was in grad school, I taught an undergrad intro to UX course where I introduced folks to the concept of the user experience and how if something’s not user-friendly, there may be an issue with design. It seems really obvious to those who have been in the field, but for newbies, it's revelatory – like seeing the Matrix.

I encourage people to harness that energy. When you have a user experience interaction that could be improved, write it down. Use that as an idea list, and start some of your own pet projects to improve that experience. That can be a case study.

Engaging with a real stakeholder offers another robust avenue. Collaborating with a nonprofit that resonates with you can be incredibly rewarding. These organizations are always in need of strategies to boost donations or volunteer engagement. By applying your research skills to help them meet these goals, you create a tangible, impactful project.

Taking it a step further, consider joining apprenticeship groups like Hack for LA or Tech Fleet, which collaborate with nonprofits and early-stage startups. These groups provide a setting to work in cross-functional teams and tackle real-world projects. Such experiences are highly valued by hiring managers, as they demonstrate your ability to apply research skills in a team setting and make a significant impact. These strategies are designed to propel you forward in demonstrating your research prowess."

Do you recommend online courses for new UX researchers? How can they overcome barriers to gain genuine experience in the field?

Researchers are inherently lifelong learners, constantly seeking to enhance their skill set. If you're primarily a qualitative researcher and you're curious or even a bit daunted by quantitative research, there are myriad ways to bridge that knowledge gap, especially now. The array of learning resources available is vast — from YouTube to books, to platforms like Udemy.

While I can't pinpoint specific courses without knowing the exact topic, I did write an article on boot camps. The piece wasn't an outright endorsement but rather a suggestion to identify and fill the gaps in your resume. For example, if you're eyeing a senior role and you notice it demands quantitative expertise you lack, that's your cue to seek out targeted learning opportunities, whether through videos, courses, or other educational materials.

So, while my advice may seem generic, the essence is to get specific about what you need to learn or improve based on your career goals and the requirements of the roles you aspire to.

When searching for roles, how do you differentiate or narrow your search when there are so many different titles?

When seeking roles, if you're employed, check if your company has a career ladder to understand the progression from entry-level to senior positions. This helps identify where you might need to grow to advance. If you're not employed, resources like the ResearchOps Community's Research Skills Framework can guide you. This framework allows researchers to evaluate their skills and identify areas for improvement.

I'm also excited about the latest research Thomas and I are conducting to be wrapped up in June 2024. We're analyzing thousands of job descriptions to create a company-agnostic career ladder, defining what is typically expected at various levels.

To generalize, the Research Skills Framework suggests that entry-level researchers focus on developing potential skills. Mid-level researchers are in the phase of skill development, while senior-level researchers have reached a high proficiency, possibly even teaching others. These distinctions can help guide your search and development in the UX research field.

How do you translate education skills from courses into industry experience?

If you're currently employed, consider discussing with your manager and colleagues your desire to enhance your quantitative skills, especially if your background is primarily in qualitative studies. Share your self-education efforts and express a willingness to apply these new skills practically, possibly through shadowing or contributing to a project. Clearly stating your intentions can create opportunities for collaboration and practical experience. For those not currently employed, engaging with volunteer groups or exploring freelancing can provide a pathway to apply and demonstrate your skills in a real-world context, aiding in the transition from theoretical knowledge to industry experience.

What are future trends of UXR roles in the hiring process?

There's a pertinent shift in staffing models within UX, particularly in how hiring teams consider resourcing. When I first dipped my toes into UX about eight years ago, there was a lot of talk about "unicorns" – UX professionals who could handle design, research, and maybe even coding, tasked with solving all user experience needs. As teams grew and matured, we saw a move toward more specialization, with emerging ratios like one designer to every ten developers and one researcher to every five designers. In the short term, despite budget cuts and hiring freezes leading some companies to seek versatile, T-shaped professionals who cover research and design, the long-term trend leans toward specialization. The value of having specialized roles is becoming increasingly clear because there is a real benefit to having specialists on your team who can bring a lot of depth and expertise.

How do you expect AI to impact the hiring process?

AI is beginning to make its way into the hiring process, though it's not yet a huge force. We're at a point where the integration of AI is accelerating and the landscape could look quite different next year. For the time being, there's a hesitancy to fully embrace AI in hiring due to concerns about potential biases and unfairness, despite the real advantages it could bring. T

Regarding AI's utility for job seekers, particularly in areas like portfolio building or resume tailoring, there's noticeable potential. For instance, LinkedIn's new AI feature to enhance user profiles has received positive feedback. AI could streamline the resume customization process, making it less burdensome for candidates applying to multiple positions.

Connect with Lawton

If you enjoyed Lawton’s AMA:

Check out his writing and website.
Give him a follow on LinkedIn and say hello!

Thank you, Lawton!

We’re grateful to Lawton for joining us and sharing his insight, experience, and research. If you’d like to watch the full webinar, follow this link.