RECAP: Regression Evaluation for Continual Adaptation of Prompts

The article introduces RECAP, a benchmark designed to evaluate continual-learning phenomena in prompt optimization under a proactive adapt-then-test protocol, addressing the limitations of current benchmarks that assume static constraints. The evaluation of six methods across four LLMs revealed no significant performance improvements, highlighting the inadequacy of existing techniques for proactive adaptation in dynamic deployment scenarios. This underscores the necessity for developing robust prompt adaptation methods that can effectively handle evolving constraints in production environments.