Towards Spec Learning: Inference-Time Alignment from Preference Pairs

The paper introduces "spec learning," a framework designed to align large language models (LLMs) with user preferences at inference time without requiring parameter updates. It utilizes brief user instructions and a small set of preference judgments to create natural-language prompts that condition LLM behavior, demonstrating improved performance over direct preference optimization (DPO) on specialized datasets. This approach enhances interpretability and transparency in model responses, making it a valuable tool for practitioners seeking efficient and effective model steering methods.