Test-Time Training with Next-Token Prediction

The paper introduces Test-Time Training with Next-Token Prediction (TTT-NTP), a method that enables fast-weight adaptation in pretrained long-context language models without requiring modifications to the model architecture. TTT-NTP uses the model's next contextual hidden state to supervise updates, allowing it to leverage the self-supervised next-token prediction signal effectively. Benchmark results show that TTT-NTP improves performance on RULER Full-13 across models like Llama-3.1-8B and Mistral-7B-v0.3, as well as on the LongBench-v2 QA benchmark, making it a valuable technique for practitioners seeking to enhance the capabilities of existing LLMs.