Characterizing Narrative Content in Web-scale LLM Pretraining Data

A new study provides the first fine-grained analysis of narrative structure within a 3-trillion-token open pretraining corpus, finding that storytelling qualities are measurable at scale and unevenly distributed across web data sources [1][2]. The research, published on arXiv, examines the Dolma corpus, a 3-trillion-token open dataset used for training large language models [1][2]. Drawing on narrative theory, the authors designed a framework organized around three core narrative elements: agency, setting, and events. These were operationalized into 11 interpretable dimensions [1][2]. The team sampled and annotated a diverse set of 400 passages before fine-tuning NarraBERT, a model built on the RoBERTa architecture, to perform fine-grained narrative prediction [1][2]. Applying NarraBERT to 3 million passages produced a new dataset called NarraDolma, which the researchers have publicly released alongside the model [1][2]. The analysis revealed that narrative structure can be measured at scale across extremely heterogeneous data, uncovering a continuous, multidimensional narrative structure underlying web text [1][2]. The study also found that narrative qualities are unequally distributed across pretraining sources and topics in ways that current data curation practices neither measure nor account for [1][2]. This finding carries implications for how pretraining data composition may affect narrative reasoning tasks in downstream models [1][2]. The work arrives amid broader efforts to understand what large language models internalize from their training data. Artificial intelligence research has long drawn on fields such as psychology, linguistics, and philosophy to interpret model behavior [4]. The transformer architecture, introduced in 2017, accelerated advances in natural language processing and underpins the current generation of generative AI systems [4]. Understanding the narrative content of pretraining corpora adds a new dimension to these interpretability efforts, as narrative is a fundamental mode of human communication [1][2]. The researchers argue that their framework, dataset, and analyses provide a foundation for studying how data composition affects narrative reasoning [1][2]. By making NarraDolma and NarraBERT publicly available, the team aims to enable further investigation into the relationship between pretraining data characteristics and model capabilities [1][2].