Expert Preference-based Evaluation of Automated Related Work Generation

The article presents GREP, a multi-turn evaluation framework designed to assess the quality of automatically generated related work sections in scientific writing. It integrates classical evaluation criteria with expert preferences, breaking down the assessment into fine-grained dimensions and utilizing contrastive examples for contextual guidance. This approach demonstrates a more robust evaluation capability compared to standard LLM judges, highlighting the challenges state-of-the-art LLMs face in meeting domain-specific standards, which is critical for enhancing human-AI collaboration in scientific writing.