AgentMeter: Evaluating Model-CLI Matching for CLI-Based Local Task-Solving Agents

AgentMeter introduces a benchmark for evaluating the effectiveness of model-CLI configurations in CLI-based local task-solving agents, highlighting that the same model can yield varying performance metrics depending on the CLI used. The benchmark employs two datasets, Benchmark90 and Core30, to assess 24 model-CLI combinations, revealing that optimal configurations differ based on criteria such as success rates and cost efficiency. This work emphasizes the importance of considering both model and CLI in tandem for practitioners, as it impacts task success and resource consumption in real-world deployments.