
Nov. 17, 2026
Contact: Eric Stann, StannE@missouri.edu
University of Missouri researchers have developed a way to determine whether artificial intelligence is uncovering meaningful insights from people’s words or simply sorting them into categories.
From interviews and focus groups to open-ended survey responses, researchers look for patterns in what people say to better understand their experiences and perspectives. But identifying meaningful themes takes time, and interpreting people’s words requires more than simply grouping similar responses together.
To determine whether AI can identify meaningful patterns without losing important nuances, a Mizzou team spanning journalism, data science and nursing developed a method to evaluate computer-generated topics. The approach examines how closely those topics match themes identified by human researchers and whether they provide useful insights into the information.
“AI can help researchers see patterns across large amounts of information, but the real question is what those patterns tell us about people’s experiences,” said Chau Tong, an assistant professor in the Missouri School of Journalism with a joint appointment in the MU Institute for Data Science and Informatics. “By bringing together different areas of expertise, we can better evaluate how these tools support research and where human judgment remains essential.”
Putting AI’s analysis to the test
The team tested the evaluation framework, called the Alignment and Usefulness Rating Audit (AURA), by analyzing 32 interviews with nursing faculty, practitioners and students about their experiences with generative AI.
Researchers first identified key themes in the interviews, then compared their findings with topics generated by a computer model. AI identified many of the same broad themes as human researchers, but it often missed important distinctions in meaning and context.
Eleven of the 12 themes identified by human researchers closely matched at least one topic generated by the computer model. However, a match did not necessarily mean the model captured the full meaning of a theme or organized the information in the most useful way.
The model sometimes combined several distinct themes into one broad topic or divided a single theme into several smaller ones.
“When people talk in interviews, they don’t usually stay on one topic from start to finish,” said John Robert Bautista, an assistant professor in Mizzou’s Sinclair School of Nursing with a joint appointment in the MU Institute for Data Science and Informatics. “AI can help organize those conversations so researchers can more easily see what people are saying.”
Where AI helps and where it falls short
While AI helped researchers organize large amounts of information and identify patterns, the team found less evidence that it consistently generated meaningful new insights beyond what human analysts identified.
“Our findings suggest the technology can help researchers manage large collections of interviews, focus groups and survey responses,” said Temitope Eyinla, a doctoral researcher in the MU Institute for Data Science and Informatics. “But human expertise remains central to interpreting the results.”
The evaluation could support research into issues such as patient experiences, educational challenges and workplace concerns, where understanding people’s perspectives is essential. However, the study examined a relatively small set of interviews, so additional testing is needed to determine how well AURA works across different subject areas and types of research.
The study, “Comparing BERTopic outputs with human-coded themes: The Alignment and Usefulness Rating Audit (AURA) workflow for qualitative analysis,” was published in the International Journal of Qualitative Methods.