TxBench-PP: Analyzing AI Agent Performance on Small-Molecule Preclinical Pharmacology

A new benchmark designed to test AI agents on preclinical drug-discovery decisions found that no system could reliably recover accurate conclusions, with the top configuration passing only 59.3% of endpoint attempts, according to a paper posted to arXiv on June 17 [1]. The benchmark, called TxBench-PP, is the first focused slice of a broader TherapeuticsBench effort that aims to span drug-discovery stages and therapeutic modalities [2]. It contains 100 evaluations indexed by program stage, assay type, and task structure, covering areas such as mechanism-of-action reasoning, compound-target engagement, causal target validation, developability and safety, and translational efficacy [3]. The evaluations supply agents with realistic workflow snapshots and ask them to inspect files in a coding environment before returning structured answers that are graded deterministically [4]. Researchers tested 16 model-harness configurations, comprising 11 models and 4,800 trajectories [1]. The strongest configuration, Claude Opus 4.8 paired with the Pi harness, passed 178 of 300 endpoint attempts, yielding a 59.3% pass rate with a 95% confidence interval of 51.1 to 67.6 [5]. The second-strongest configuration, GPT-5.5 with Pi, passed 166 of 300 attempts, or 55.3%, with a confidence interval of 47.0 to 63.6 [3]. Claude Opus 4.8 with Claude Code followed at 54.7%, and Gemini 3.5 Flash with Pi reached 51.3% [4]. The overlapping confidence intervals suggest a leading group rather than a single clear winner [5]. Even the top configuration failed 122 of 300 attempts and passed only 41 of 100 tasks in all three replicates [3]. The benchmark was intentionally designed to penalize memorized solutions, testing whether agents can recover decisions from supplied data rather than leaning on well-understood mechanisms and literature knowledge [2]. The broader landscape of AI in drug-related reasoning includes efforts such as TxAgent, a tool-augmented model that achieved 92.1% accuracy on the DrugPC benchmark for drug reasoning tasks, outperforming GPT-4o by 25.8 percentage points [6]. That system dynamically retrieves and synthesizes knowledge from 211 biomedical tools to support patient-specific therapeutic decisions [6]. TxBench-PP differs by focusing on local preclinical pharmacology decisions rather than clinical treatment recommendations. The authors note that current agents remain unreliable on these tasks despite substantial differences in model family and harness implementation [5]. The benchmark is positioned as a verifiable evaluation layer for a stage of drug discovery that has historically lacked standardized, realistic testing frameworks [1].