Improving Alignment Between Human and Machine Codes: An Empirical Assessment of Prompt Engineering for Construct Identification in Psychology

This study presents an empirical framework for optimizing large language models (LLMs) in the context of construct identification in psychology through prompt engineering. It evaluates five prompting strategies, revealing that the most effective approach combines codebook-guided empirical prompt selection with automatic prompt engineering for few-shot classification tasks. The findings emphasize the importance of construct definitions and task framing in prompt design, offering a systematic method for enhancing LLM alignment with expert judgments, which is crucial for practitioners in fields requiring precise classification.