PRIME: Evaluating Prompt Resolution Under Incompatible Instructions in LLMs

The article introduces PRIME (Prompt Resolution under Incompatible Meta-Instructions Evaluation), a framework designed to assess how large language models (LLMs) handle conflicting instructions. By evaluating five instruction-tuned open-weight LLMs across balanced and naturally distributed settings, the study reveals that the type of conflict significantly impacts model behavior more than model scale, highlighting the necessity for conflict awareness in instruction-following assessments. This research provides insights for practitioners on the limitations of current benchmarks and the importance of understanding response dynamics under conflicting prompts.