Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in LLM Agents

The paper presents findings on the decision-making failures of LLM agents in tool selection, demonstrating that the issue lies not in the visibility of tools but in the readout process. Through experiments, it was shown that models attend to the correct tool 80% of the time, yet still misselect it, with readout-side interventions recovering 59-91% of failures. The study introduces a training-free, gold-free selector based on per-segment attention, which significantly improves selection accuracy across various model sizes (3-32B), indicating a critical area for enhancement in LLM decision-making processes.