AI-Assisted Thematic Analysis of Transcripts with Multi-Stakeholder Triangulation

Qualitative text analysis is fundamental to research in education and the social sciences, but it is time-intensive, difficult to scale, and sensitive to researcher subjectivity. Large language models (LLMs) have shown promise for supporting qualitative coding; however, existing validation studies often compare LLM outputs against human judgments within a single stakeholder or expert group. Whether LLM-assisted pipelines can support multi-stakeholder data-source triangulation, in which thematic structures must remain coherent across diverse points of view while also revealing convergence and divergence, remains an open question. This paper presents a local LLM-assisted qualitative coding pipeline and, to our knowledge, provides one of the first systematic comparisons of LLM-assisted coding against human coders in the context of multi-stakeholder triangulation.

The pipeline uses locally hosted models for open coding, semantic embeddings, or LLMs for code consolidation, and clustering for thematic grouping. Analyses are executed independently for each stakeholder group before cross-group comparison, allowing the method to preserve distinct perspectives while identifying shared themes. We validate the pipeline through a case study of Innovation Competitions and Programs based on interviews with 97 participants across three stakeholder groups. These interviews had previously been analyzed by three independent human coders. We compare LLM-generated codes against the human reference analysis using optimal matching, embedding-based similarity, and unified category construction. Results indicate that the pipeline recovers most human-identified themes while surfacing additional valid codes, with consistent performance across stakeholder groups. These findings suggest that locally hosted LLM workflows can support scalable qualitative analysis and multi-stakeholder triangulation.