Summarization is Not Dead Yet

Large language models have raised the baseline for text summarization, but human-written reference summaries still hold clear advantages in informativeness and factual reliability, according to a multi-track evaluation published on arXiv [1]. The study, titled "Summarization is Not Dead Yet," assessed five state-of-the-art large language models across five diverse datasets using controlled human evaluation, bias-mitigated LLM-as-Judge protocols, factuality verification against external knowledge, and corpus-level linguistic analysis [1][2]. Researchers found that while LLM outputs are often preferred for surface-level coherence and fluency, human references remain more reliable for claims that require reasoning or synthesis [1][3]. "Surface fluency does not entail information fidelity," the authors note, adding that the dimensions on which human summaries excel are precisely those that single-track evaluations and surface-overlap metrics tend to miss [3]. Linguistic analysis further revealed a pattern of stylistic homogeneity across different model families, with LLM outputs remaining systematically less diverse at the lexical, syntactic, and discourse levels [2][3]. The paper concludes that current LLMs have raised the floor of summarization quality, but the ceiling of their performance remains below human capabilities [1][3]. The findings push back against an earlier, more provocative assessment. A 2023 paper had declared that "Summarization is (almost) Dead" after its own manual evaluation found that LLM-generated summaries surpassed reference summaries in many datasets, exhibiting superior fluency, factuality, and flexibility [4][5]. That study argued that roughly 70 percent of recent summarization papers primarily proposed new approaches validated on standard benchmarks, rendering much of the field's conventional work unnecessary [4][5]. The new multi-track evaluation does not dismiss those observations outright but reframes them. It acknowledges that LLMs have made substantial gains on form-oriented dimensions such as coherence and conciseness, yet it demonstrates that human references retain measurable advantages in the content-oriented dimensions of informativeness and faithfulness [2][3]. Factuality verification indicated that the human advantage becomes more pronounced when verification draws on external knowledge rather than surface-level checks [2][3]. "We encourage the community to treat summarization not as a solved capability but as a continuing testbed for advances in faithful reasoning, information compression, and linguistically diverse generation," the authors write [3]. The paper surfaces persistent gaps that it argues provide a grounded basis for identifying open problems and prioritizing future directions in summarization research [3].