Building a user research participant pool from scratch

Transforming a 300+ person department to invest in UX research

Our next release was in less than two weeks.

“Was this usability tested?” I asked, somewhat perplexed by the solution I had just seen.

“Oh yes, I forgot to mention we did run this through a few task-based tests and didn’t find any confusion from users.”

“Great!” I exclaimed. Who did you end up testing with?

“Well… er…” there was an awkward pause.

The project’s appointed usability champion jumped in to save the silence, “I helped him draft up a few tasks that we used for hallway testing.”

I felt underwhelmed by the response. Then nervous. Our next release was looming and any changes would be difficult to make in time.

“So we only got feedback from coworkers?” I asked, underwhelmed.

“Well, we talked to 5 people from our department. They weren’t software engineers though. And after all, we’re all patients and use the product too, right…”

“We’re all users and use the product too…” — If you ever hear this phrase, tread carefully.

At that time I was the Lead UX Designer and Design Manager on the company’s flagship product suite, MyChart. I led a small but mighty team of three other designers. As a team, we had a lot of ground to cover. There were over six distinct product areas and our tiny team was easily outnumbered by the 150+ software engineers working within our department.

For context, our product suite, MyChart, was used by over 150 million healthcare patients to get and stay healthy. It’s used by many of the top health systems in the world to give their patients access to care and health information and ranks as the #1 patient portal on the market.

Despite our product’s successes, back then we were lucky if a project went through more than one round of user testing, and doubly lucky if that testing had feedback from real, actual users who didn’t work within our company.

I realized this was becoming more and more of a problem. Design review conversations were based on individual opinions instead of evidence. Roadmap conversations were solely based on input from executives and sales calls. I noticed phrases like “Well I would think…” or “<insert app name> does it like this, so…”

Despite us feeling like the user was there, in truth, the user was absent in the conversation. It was leading us down a slippery slope of solving the wrong problems in the wrong ways. We weren’t doing real testing or getting real feedback. I didn’t see things changing much on their own, so I committed myself to figuring out how to change the culture around research and usability testing. Needless to say, we had a long road ahead to get to where we needed to be.

Why was it so difficult to conduct user tests?

Before taking any drastic measures to introduce process or give some big speech, I needed to first understand where we were at. Why wasn’t user testing being done? Why weren’t we talking to actual users?

I created a short Google Forms survey that I sent out to my entire department of over 200 people to try and identify some of the biggest barriers to conducting sound research.

Within a week, the survey results were in. To no surprise, people were feeling overwhelmed. It was a “too much to do in too little time” mentality and user testing was first on the chopping block.

• “I need to talk to people who have experience managing chronic disease. Where do I even begin looking?”
• “I’m so busy, I simply don’t have the time to coordinate all of the logistics for a user test right now.”
• “I’ve tried to recruit a few people, but I never get answers or people don’t show up so I just stopped trying.”

I analyzed the survey results and started to dream up the perfect research process. This was the fun part. If I had a magic wand, what could research look like within my department?

What if a project team could get in front of any user type within 1–2 weeks and all they had to worry about was prepping their own study materials ahead of time? I made this my goal. My hypothesis was that if project teams didn’t need to worry about sourcing participants, coordinating logistics, or handling reimbursements, more teams would engage in user research and usability testing. This would be a big mountain to climb!

Overcoming the first obstacle

I began by first researching how other well-known companies conduct research. I listened to podcast episodes and read countless articles on the topic. I even went undercover and signed up to be a research participant at other companies like Google. I analyzed how and when they communicate with their research participants, what survey questions they asked on their intake questionnaires, and more. I was inspired!

I started crafting my first participant intake form using Google Forms, however, shortly after taking the first steps toward fixing our research culture I encountered the first big roadblock on the journey — legal.

Our company’s legal team got word of what I planning and immediately got involved. I was told we should stay away from collecting patient information as it was a potential risk to the business. There were also concerns and doubts around participant management and correspondence as I wanted the ability to do in person interviews and user tests.

I held numerous meetings where we would debate intake survey questions and pick apart the proposed process with a fine-toothed comb. I would go back countless times to make tweaks to phasing used throughout the survey, email templates, or parts of the process in order to comply with my company’s legal standards. After about a month of consistent tweaks the process was ultimately approved!

Coordinating recruitment efforts

At this point I had a legal-approved participant intake form, but no participants. Where to begin?

I researched numerous third-party tools like UserTesting.com, however, these tools were either too expensive for our budget or didn’t have a diverse participant pool that would meet the demands of our teams. It became evident that if I was going to create the ultimate research experience, it had to be done grass roots style.

Across all of our teams we serviced more than 7 unique (and very specific) persona types. If this process was going to be successful I needed to have a sizable and reliable source of qualified users that represented target audience of the various teams within our department. How could I find enough research participants? And once I do find them, how could I convince them to be a part of our research pool? I had some major challenges on my hands.

I was stuck. I spent weeks trying to recruit participants with very limited luck. The only participants I was able to recruit were a few employee relatives. Somehow I needed to make participating in research feel more exciting and rewarding.

Creating an incentive plan

It became obvious to me that I wasn’t ready to recruit just yet. I needed to take a step back and develop an incentive plan. This quickly led me to my next obstacle — budget approval.

Unfortunately no matter how hard I tried I was unable to get approval from legal and the company to issue monetary incentives – it was simply off the table. I needed to think of a way to incentivize participants to show up and complete a study without simply giving them cash.

My answer was to turn this into a rewarding and fun experience. Our campus was known to be one of the most quirky and wild architectural experiences in the area, so much so, we had campus tours every week or so. Additionally, we have a top-notch culinary team of over 90 chefs that produce fantastic plated meals from sushi and steak to tasty homemade peanut butter and French silk pies. I worked with our campus tour group and culinary teams to offer up a self-guided tour experience of the campus as well as a free lunch and company calendar for anyone who participated in a usability study.

I also felt it was important to share the impact the volunteers had on the product directly back to the community of volunteers. To do this, I created a quarterly newsletter that would get sent out to all participants on the recruitment list with brief write-ups on the impact their involvement had on the features we’re creating. Armed with these incentives, I felt it was finally time to tackle participant recruitment.

A multi-faceted recruitment sourcing effort

To prevent analysis paralysis, I took a step back and asked “What would this look like if it were easy?” and I started writing out a multi-faceted recruitment plan and I brought on a few additional teammates to assist me in the efforts.

“What would this look like if it were easy?”

Local retirement communities

I cold called retirement communities in the local area and volunteered to put on free classes for their residents to learn about MyChart and how to sign up for a free account. At the end of the session, I’d pass around a sign up sheet to volunteer to participate in research and usability studies. In total we probably recruited around 90 participants through this effort.

Internal postings and email blasts

I worked with our internal teams to get a posting drafted on our company’s intranet as well as an email that went out to notify employees of the recruitment efforts. We had employees tell their family and relatives about the opportunity and passed around our Google intake form for them to sign up. All in all, we probably sourced around 40 participants via this method!

Company building tour groups

Our company’s campus was famous for its elaborate decor and themed buildings, and I knew that we did tours from time to time, particularly with older adults and groups such as the red hat ladies. This was perfect for our meeting some of our target demographics. I pitched the idea to our front desk staff and got approval to advertise for our research pool. This method superseded my expectations and I learned we do multiple group tours a week with people who visit from around the country! This gave us a passive recruitment tally of around 2–4 participants a week!

It became a fun team ritual to check the recruitment metrics via the survey at the start of each weekly meeting to see how the numbers increased from the prior week. With steady recruitment numbers coming in thanks to holding product educational sessions and campus tour recruitment we had surpassed over 100 participants and counting! I didn’t feel it was quite ready for prime time, however, so I reached out to one of the project teams within our department and asked if they would like to trial our new research pool process. Onward!

Fine-tuning participant communication

Shortly after our first pilot kicked off I got feedback from one of our study schedulers that she didn’t have all the proper information from the team to find the right participant and coordinate a study. In order to scale this to dozens of project teams at once, there needed to be a more streamlined systematic process. This brings me to my next roadblock as we approach the peak of the mountain!

I needed a way to to ensure communication flowed smoothly and consistently between the teams requesting a study and potential participants. To accomplish this, I designed a simple Google form that project teams could fill out to submit a request for participants. In the form teams would specify the type of study, and specific user demographic requirements, how much time was needed for the study, etc. From there, our schedulers could filter our research pool by criteria passed over from the form.

We also designed a few email templates based on common scenarios we encountered during our pilots. This helped us standardize and streamline communication between our company and usability participants. After a participant completed a study, it would get marked in the participant pool and the participant would receive their lunch coupon, campus tour, extra company swag, and a thank you note. Things were starting to flow pretty smoothly, but there was one last hurdle to overcome…

Employee awareness & evangelism

At this point I was quite proud of how far this effort had come. We had a legal-approved intake form and process, a series of recruitment vehicles in place, a small team of schedulers, and a formalized project intake form and email templates. The finish line was in sight, however, I quickly learned I needed to pour more effort into educating our internal teams about the process and to convince them to use it if I was going to experience any change in culture.

I began by creating a presentation deck that myself along with a few other members from our small team gave at numerous team meetings to explain how the process works and how to get started. We also made allies with our pilot project team and had them speak about their experience using the process and the benefits it had on the outcome of their project. Lastly, I partnered with the R&D lead of our department and had him encourage teams to use our process and he implemented a best practice that each medium to large size project must bring their work through user testing at least once before launch.

These efforts were massive to getting teams to change their habits. Over the course of about 4 months we were seeing an average of 3 user tests per project that was of a medium size or larger, and most of these tests were being coordinated via the user research process I had set up! It was a common occurrence in design review where reviewers would cite usability study findings to back up their approach which ultimately led to less churn and more productive design debates.

Closing thoughts

Changing culture can feel like an uphill battle littered with obstacles along the way. With the help of a few allies I was able carefully lead our division of over 300 people to reinvent how we get our work in front of users so we can product software with the user at the center. To me, that’s a fight worth fighting.