Do LLMs Reliably Identify Correct Information Units in Aphasic Discourse?

This study investigates the ability of instruction-tuned large language models (LLMs) to classify Correct Information Units (CIUs) in aphasic discourse, utilizing sixteen annotated transcripts. Four models—Llama-3.1-8B, Qwen2.5-7B, and Mistral-7B—were evaluated under zero-shot and few-shot prompting conditions, with few-shot prompting yielding mean F1 scores between 0.776 and 0.817, while highlighting issues with precision and performance variability based on severity of aphasia. The findings suggest that while LLMs can aid in CIU identification, they require human oversight for reliable application in clinical settings.