CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks

A new evaluation framework called CollabSkill measures how effectively humans and AI agents work together on real-world occupational tasks, moving beyond autonomous benchmarks to capture the dynamics of genuine collaboration, according to research posted to arXiv on April 20, 2026 [1]. The framework pairs human workers with AI agents on tasks aligned to their professional backgrounds, gathering data that reflects the complexity of economically valuable work and the actual usage patterns of real people [1]. To account for differences between individual workers, CollabSkill uses a Bayesian skill rating system that separates and quantifies the skill contributions of both the human and the agent [1]. The study draws on more than 1,500 prompts collected across 386 working sessions involving 93 human workers [1]. On the agent side, the rankings produced by CollabSkill diverge meaningfully from those seen on fully autonomous benchmarks. Where Codex leads in standalone evaluations, Claude Code ranks first when measured on collaborative performance [1]. This finding underscores a gap between how models perform in isolation and how they function as partners in a shared task. On the human side, the analysis points to practical experience as the primary driver of collaboration skill. Hands-on collaboration meaningfully shifted workers' AI literacy, the researchers report [1]. The concept of collaboration itself — derived from the Latin com- meaning "with" and laborare meaning "to labor, to work" — involves parties strategically choosing to cooperate in order to accomplish a shared outcome [3]. CollabSkill applies this principle to the human-agent relationship, treating the interaction as a measurable skill rather than a binary success-or-failure metric. Large language models, the underlying technology for agents like Claude Code and Codex, are machine learning models with many parameters trained on vast amounts of text through self-supervised learning [4]. The CollabSkill framework represents an effort to evaluate these models not by their solo capabilities but by their ability to augment human workers in occupational settings [1]. The researchers express hope that the framework will spur development of AI agents that genuinely augment human workers rather than simply automating tasks in isolation [1].