Detecting Knowledge Gaps from Conversational AI Interactions Using Curriculum Prerequisite Graphs

The study presents a pipeline that utilizes a few-shot text classifier to map student questions directed at a conversational AI teaching assistant to specific curriculum topics, leveraging a prerequisite knowledge graph extracted from GPT-4. The classifier achieved 80.0% accuracy across 43 labels when evaluated on 1,340 question events from 164 graduate students, with significant correlation (rho = 0.491, p = 0.008) between topic-level question volume and self-reported difficulty. This approach highlights the potential of conversational AI interaction logs as diagnostic tools for identifying knowledge gaps, offering instructors insights into which topics require further focus.