What we learned testing AI in our UX Research process.

Artificial intelligence (AI) is buzzworthy. 

Over the past year, it’s all anyone can talk about in the digital product world. But all the buzz we saw focused on what it can do and tips for using it, but not much (at all) about the human implications, motivations, and decision-making happening between people’s ears when it comes to AI. 

We wanted to lend our knowledge to this space by learning about consumer behaviors and opinions. We’re experts in generative research and directing our clients on what they should do based on what we learned. So, we conducted a generative consumer study with a follow-up quantitative survey to learn what people think about AI, helping product teams better introduce their own AI features. You can read that report here.

‍

The kicker? We used AI to conduct some of the work. Not the actual research—but we employed AI to try its hand at distillation and analysis, survey writing, summarizing, and drafting. There are so many speculative articles out there about how AI is going to impact UX research and our careers—we wanted to put it into practice to see what happened.

Rather than prioritizing questions such as, “Do we need AI?” or “Is it going to take our jobs?” we wanted to understand: “Can we work alongside it?” and, if so, “What does that relationship look like?” And ultimately, “Does it help us work more efficiently?”

Is it worth it to implement AI into your research process and, if so, where? So where better to use AI than in an experiment about AI?

‍

A few caveats before we dig in: 

• This was an experiment run by our excellent researchers who know what quality research outputs look like, therefore, we could tell if what the AI was producing was quality or not. We would warn against using AI to augment too much of your research work until you are fully comfortable with doing the process yourself.
• We recruited real users for this study. Synthetic Users, and their validity, are a huge topic of conversation right now, too, but we won’t cover that in this article.
• AI is not new as a technology and we’ve used several of them for years (machine learning, spell check, transcription, etc). But Large Language Models (LLM), like ChatGPT, are mostly new technologies for us and thus the focus of this experiment.
• Research ethics is also at the top of the AI conversation, and for good reason. Though we won’t go into this topic in this article, it’s important to note that all participant data was anonymized before being put into AI tools and our qualitative interviews contained no mention of PII.
• We understand that AI is a sensitive topic and thus would never implement it in our client work without first consulting any guidelines or rules regarding its use at the company level.

Let's dig in.

How did we use AI in this project and how did it go?

1. Drafting interview and survey questions.

We used Pollfish AI and ChatGPT to draft interview and survey questions for both our qualitative and quantitative efforts.

Traditional Process

‍Typically, we’d write our surveys (and other research materials) using just our brains, what we’ve learned about our client’s objectives, and research best practices.

How it Went

‍These tools provided a plethora of question ideas and got us 60% of the way there. While some of the questions were great, they were pretty starter-level questions that most researchers would include. There were a lot of redundant questions and, specifically for the survey, some that just made no sense at all.

Learnings

‍If you’re having a hard time getting started or would rather start with a big list and narrow it down, AI can help. But it’s not going to write the whole thing yet. If you’re a seasoned researcher and can churn out a great discussion guide or survey quickly, this is unlikely to save you much time.

‍

2. Transcribing qualitative interviews.

We used Dovetail to transcribe our qualitative interviews, which is not new to our process.

Traditional Process

‍We’ve used Dovetail for several years for transcription and it has made a huge difference in our speed of delivery and quality of work. Before Dovetail, we were hand-writing/typing notes and had to have two researchers in each interview session—one interviewer and one notetaker. With Dovetail, we just upload a video of the interview and it transcribes everything for us, meaning everything that was said was captured in exact detail and we save time in the actual interview session.

How it Went

‍Sometimes the transcript is incorrect, but it’s nothing more than a quick fix here and there.

Learnings

‍Having a tool for transcribing interviews is somewhat of an industry standard and something we will continue to use.

‍

3. Summarizing interview transcripts and tagging.

We used ChatGPT to summarize qualitative interview transcripts and then tagged the summaries.

Traditional Process

‍This is something we would normally do manually, taking the time to read all of the raw data and tag it according to important themes and research objectives. Then, we use affinity diagramming within each theme to define it.

How it Went

‍Typically we read the transcript and tag it linearly, but with AI we had to copy and paste specific pieces of the transcript into ChatGPT and then tell the AI to summarize the answer to a specific question— if you put the whole transcript in at once, the AI wasn’t able to parse out the answers to all of your questions. There is also a character limit in ChatGPT that is a big barrier to this process; it’s just not able to accept the whole transcript for a 45-minute interview.

Learnings

• We thought AI would save us time in the tagging process but it probably made it take longer. And it made a relatively simple, albeit laborious process, more convoluted. 
• However, we’re curious if AI can help ideate on the tag taxonomy. If we do this again in the future, we'll only use the AI summary for internal knowledge-sharing or to provide quick, interactive updates, but not as the final analysis content.
• Using the summaries within the analysis also made it difficult to pull participant quotes—a huge drawback because quotes are what really bring insights to life for stakeholders.
• Lastly, using AI in this way made it more difficult to figure out what our insights were and, thus, to write the final report. While the normal process of manually tagging is laborious, it is important to immerse yourself in that knowledge to better empathize and advocate for the users.

‍

4. Summarizing screener and survey questions.

We also tried ChatGPT to summarize our open-ended screener and survey questions.

Traditional Process

‍Normally, we would save this process for last, reading through each answer and manually tagging it in Google Sheets, then analyzing the counts of each answer.

How it Went

This was way faster than reading 400+ open-ended survey responses manually, though pulling out specific, interesting quotes was more of a challenge down the line.

Learnings

‍This was worth it given the effort vs. value provided by the responses. The open-ended questions didn’t end up providing any new value to the research, so we saved time by not doing this manually.

‍

5. Analyzing survey data and persona clusters.

We used BigML to analyze our survey data to form persona clusters.

Traditional Process

‍This is something we’ve been doing for years and works great as it can look at thousands of pieces of data and see the groups within it, whereas that would be almost impossible for a human to do with any sort of accuracy. 

How it Went

‍There can be some trial-and-error with the clustering and figuring out how many to create or what data to use.

Learnings

‍We will continue to use tools like BigML to process large amounts of data, specifically for persona creation.

‍

Was using AI throughout our research process worth it?

While the industry is embracing AI in a big way, it’s important to recognize where it’s not the same as a human doing the work. 

From this experiment, we learned that AI tools will (if they haven’t already) take over the manual labor of research—editing for bias and grammatical error, interview transcription, etc.—alleviating the workload of UX researchers. However, despite this, we don’t ultimately see AI replacing us but, instead, augmenting our skills and making space for what we’re truly good at: strategy, creativity, and empathy.

In some ways, AI made this project faster and showed us where we could be more efficient by using it. It’s imperative to our success and growth as a company—and UX researchers as a profession—to continue to experiment with AI and figure out how to adopt it in a meaningful way. As the quote from Curtis Langlotz, MD, PhD goes:

Artificial intelligence will not replace radiologists … but radiologists who use AI will replace radiologists who don’t.

The tools in this space are always evolving, so we will continue to experiment as they roll out. For us, it’s not a “never” it’s a “not yet.”

So, what do you think? How have you used AI in your research work and what did you learn from that experiment? Share your thoughts with us at hello@zocodesign.com or on our social media!

What's next?

Have a project you’re working on and need some support? Reach out to us.
‍
Do you just want to chat about product, UX, research, process, and methodologies? We’re down for that too. Let's chat.‍

Do you want to avoid talking to another human being right now? We get it. Sign up for our Curious Communications newsletter to stay up to date on all things UX and other curiosities. We’ll hit your inbox every few weeks.