Style or Content? Evaluating Style Classifiers with Controlled Content Overlap

A new study proposes a controlled method for testing whether style classifiers rely on superficial content cues rather than genuine stylistic patterns, using parallel Bible translations to systematically vary content overlap between style classes [1]. The research, submitted in 2026, introduces a parameter called alpha to measure how much content is shared across style labels, ranging from no shared content at alpha equals 0 to fully shared content at alpha equals 1 [1]. The parameter is defined as the normalized residual of mutual information between content identity and style label, making it independent of the number of style classes [3]. English Bible translations provided an aligned testbed because the same content is expressed across multiple stylistic variants [3]. RoBERTa-based classifiers trained under different overlap conditions showed distinct behaviors. Models trained with low overlap degraded when content cues were removed, while high-overlap models transferred more robustly to new settings [1]. A cross-style content retrieval probe further demonstrated that content becomes less recoverable as alpha increases, with retrieval accuracy above 0.7 at alpha equals 0 dropping below 0.1 at alpha equals 1 [3]. Style classification accuracy remained high even at alpha equals 1, indicating the model shifted from encoding content to encoding style [3]. The findings align with broader discussions in computational linguistics about the entanglement of style and content. A 2020 study at COLING found that style and content cannot be totally separated, noting that encoders mistakenly strip out content when attempting to remove source stylistic features [5]. That work concluded that style should be considered an integral component of a text rather than a set of explicit linguistic elements that can be isolated [5]. Content analysis, the systematic study of documents and communication artifacts, has long relied on labeling texts to identify meaningful patterns [7]. Machine learning classifiers can greatly increase the number of texts that can be labeled, though the scientific utility of doing so remains debated [7]. The controlled overlap framework offers a diagnostic tool for evaluating whether these classifiers are learning style or simply exploiting content shortcuts [1].