CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression

A new evaluation protocol called Cavewoman finds that compressing the output of large language models can cut per-item costs by up to 3x, but compressing the input backfires, raising net cost even as accuracy drops, according to a preprint posted to arXiv on June 23 [1][2]. The protocol, described by researchers in a paper submitted to arXiv, scores model generations on task accuracy, realized per-item cost, and agreement with the model's own unconstrained reference text [1][2]. Eight models were tested across five datasets at five reduction levels, with both input and output channels measured on the same items [2]. Output compression reduced realized cost on most API models by 1.4 to 2.4 times per model, and by up to 3x in the best case [1][2]. All four open-weight models also saw cost reductions under public-tier pricing [2]. Input compression produced the opposite result. Across the five benchmarks, net cost rose by roughly 1.15x on average, climbing to 1.8x on the worst dataset and 2.7x under stronger compression [1][2]. The increase occurred because models generated longer responses to compensate, even as their accuracy collapsed [1][2]. The paper describes input compression as "a strict lose-lose" [2]. Surface-level text also diverged from the unconstrained baseline under input compression. On non-reasoning models, roughly half of all generations were factually correct, yet their surface wording no longer entailed the model's own unconstrained reference generation [1][2]. The divergence held after length-controlled re-scoring, multiple-comparisons correction, and replication under complementary semantic measures [2]. The findings arrive as the broader research community continues to refine evaluation frameworks for generative models. A separate scoping review of quantum circuit generation systems, for instance, found that none of the thirteen systems examined reported end-to-end evaluation on quantum hardware, leaving a gap between generated artifacts and practical deployment [3]. Platforms such as Hugging Face have also built infrastructure to link papers with models, datasets, and interactive demos, including an integration that embeds Spaces directly on arXiv abstract pages [4][5]. Code and data for the Cavewoman protocol are available on GitHub [1][2].