CARE: Controlling LLM-Generated Policies through Auditable Review of Evidence in Scientific Experimentation

The article introduces CARE (Controlling LLM-Generated Policies through Auditable Review of Evidence in Scientific Experimentation), a system designed to optimize high-throughput experimentation (HTE) while maintaining safety and accountability. CARE employs a non-LLM incumbent optimizer as the default, allowing LLMs to propose challenger ranking policies that are only executed if supported by pre-selection evidence, with decisions logged for auditability. It demonstrates significant improvements on the Minerva/Olympus and ChemLex benchmarks, with scores rising from 80.0 to 88.5 and 83.9 to 92.1, respectively, highlighting its potential for safer and more effective LLM integration in scientific experimentation.