Is it agentic enough? Benchmarking open models on your own tooling

Hugging Face has introduced a new benchmarking harness that measures not just whether a coding agent arrives at the correct answer, but how much work it took to get there, using the company's transformers library as a case study [1]. The tool evaluates agent performance across several axes, including match percentage, median time, median tokens consumed, and the percentage of runs that end in an error [1]. The goal is to provide library maintainers with concrete data on how changes to a repository affect agent interactions, and to help users assess how different models handle real-world tasks [1]. The benchmark runs entirely on open models driven by the pi coding agent, with every combination of model, library revision, and task executed on identical hardware through Hugging Face Jobs [1]. Hugging Face, a New York-based company, is known for its transformers library and its platform for sharing machine learning models and datasets [7]. The harness was designed to test a specific hypothesis: that adding a dedicated command-line interface (CLI) and a packaged "Skill" to the transformers library would reduce the effort required by an agent. The company had previously redesigned its hf CLI to be agent-optimized, resulting in agents using 1.3–1.8× fewer tokens, and up to 6× fewer tokens in some cases [1]. For large, capable open models, the benchmark confirmed a tradeoff. The introduction of the CLI and Skill reduced the median time agents spent on tasks, but increased token consumption on the "clone" variant, where the agent has access to the full source code. The median input tokens rose from roughly 4,000 to 6,400 because agents began reading the new CLI implementation and example scripts to learn the interface before using it [1]. This token cost is a worst-case scenario, as the benchmark runs each task with a fresh agent that must rediscover the CLI from scratch [1]. The broader context for this work is the rapid evolution of large language models (LLMs), which are trained on vast amounts of text for tasks like language generation [2]. A specific class of these, known as reasoning models, are trained to solve complex, multi-step problems in logic, mathematics, and programming, and can revise their own reasoning steps during inference [3]. The coding agents benchmarked by Hugging Face operate in this domain, where a model might write a 40-line Python script to classify sentiment, while another simply executes a single CLI command to achieve the same 0.9999 result [1]. For smaller, local models, the benchmark revealed a different dynamic. The "skill" tier, which provides curated documentation and examples, improved the match percentage for larger models but caused a drop for the smallest ones [1]. The company's findings are part of a competitive landscape where firms like Google and DeepSeek are also advancing agentic capabilities. Google's Gemini models, for instance, have been updated throughout 2025 to enhance "agentic capabilities for autonomous research and software development" [4]. DeepSeek, a Chinese AI company, made headlines in January 2025 with its DeepSeek-R1 model, which provided responses comparable to GPT-4 and was trained at a reported cost of US$6 million, significantly less than competitors [5].