LLM Features Can Hurt GNNs: Concatenation Interference on Homophilous Graph Benchmarks

Concatenating large language model features into graph neural networks can systematically reduce accuracy on homophilous benchmarks, according to a preprint posted to arXiv on June 16. The study finds that under specific conditions, adding SBERT-encoded GPT-4o-mini TAPE features to an MLP backbone cuts PubMed test accuracy by 17.0 percentage points [1]. The paper, titled "LLM Features Can Hurt GNNs: Concatenation Interference on Homophilous Graph Benchmarks," documents a phenomenon the authors call concatenation interference [1]. While end-to-end LLM pipelines often succeed on these tasks, the simple approach of appending LLM-generated node representations to existing features can backfire. On the Cora dataset, the same method reduced accuracy by 4.3 percentage points, while CiteSeer saw a drop of 0.6 percentage points, a figure the authors note falls within seed noise [1]. The negative effect is not universal. The performance penalty attenuates when the experimental setup is relaxed—for instance, by swapping the MLP backbone for a GCN, GCNII, or GAT architecture, or by using random data splits instead of the standard Planetoid public split [1]. On datasets with medium homophily, the trend reverses: concatenation boosted accuracy on WikiCS by 4.4 percentage points and on ogbn-arxiv by 11.7 percentage points [1]. To anticipate when concatenation will help or hurt, the researchers propose a metric called Delta_sig, which measures the discriminability of the LLM features alone. Across nine datasets, Delta_sig showed a stronger correlation with concatenation cost than homophily did, with an r-squared of 0.38 compared to 0.06 [1]. The authors identified a bootstrap-best change-point of 13.8 percentage points and a rule—"Delta_sig <= tau predicts non-positive concat cost"—that correctly classified seven of the nine datasets [1]. Because 60% of bootstrap samples placed the threshold between 5 and 30 percentage points, the team presents Delta_sig as an interpretive lens rather than a strict filter [1]. The paper also rules out dimensionality and weight-decay as explanations for the accuracy drop. A controlled ablation on PubMed placed the LLM-feature deficit between a same-source PCA baseline, which caused a 2.3 percentage-point drop, and same-dimension Gaussian noise, which caused a 37.3 percentage-point drop [1]. Furthermore, nine PubMed configurations fit a power law where the absolute concatenation cost is proportional to the square root of the ratio of feature dimension to training examples, raised to the 1.31 power, with an r-squared of 0.97 [1]. The low-Delta_sig, small-training-set corner of this relationship is precisely where the headline 17 percentage-point PubMed deficit appears [1]. Large language models are a type of machine learning model designed for natural language processing tasks and trained with self-supervised learning on vast amounts of text [8]. The study's findings add a cautionary note to the widespread practice of augmenting graph models with LLM-derived features, a technique that has been widely reported to improve benchmark performance [1].