Ensuring Quality Assurance in Qualitative Research: Organising and Analysing Expert Reviewer Commentary for a Qualitative Interview Guide

Abstract

This guide provides a 10-step process for organizing, classifying, and analyzing expert reviewer commentary collected during the quality assurance of a semi-structured interview guide. This process is for the GUIDE Project (Generative Understanding, Inclusive Design, and Ethical Assessment) at the University of Calgary's Postplagiarism Research Lab, but may have wider applications. Scholarly literature provides an evidence base for expert review of qualitative data collection instruments, but few sources provide procedural instructions for how research teams should handle the resulting commentary. Research assistants are guided through step-by-step process for compiling raw reviewer feedback into a structured commentary log, mapping each comment to its corresponding interview item and thematic construct, classifying comments by appraisal dimension (relevance, clarity, specificity, and correspondence) and action code (retain, revise, rewrite, add, or remove), assessing convergence and divergence between reviewers, organizing findings by construct, building a revision frequency table, documenting the revision trail, identifying cross-construct patterns, and preparing materials for a second review round. Each step includes decision rules, worked examples, and instructions for populating a companion Excel workbook (the Expert Review Analysis Template). The process operates within an interpretive qualitative paradigm. We do not employ numerical validity indices or psychometric scoring. Instead, we treat reviewer commentary as qualitative data and prioritize systematic organization, transparent classification, and an auditable decision trail. A dedicated section outlines permitted and restricted uses of generative AI tools within the process, consistent with the University of Calgary Faculty of Graduate Studies' Guidelines for Generative AI Use in Graduate Studies. The intended audience is researchers and research assistants conducting quality assurance for qualitative interview guides, though the process may apply to other forms of expert review in qualitative research. Keywords: qualitative research, interview guide, quality assurance, expert review, instrument development, research methodology, construct correspondence, question clarity, audit trail, generative artificial intelligence