LLMs on Tabular Data with Limited Semantics: Evidence from Industrial Car Retrofit Prediction

Classical tree ensembles still outperform large language models on privacy-constrained industrial tabular data, according to a new study that tested three LLM-based strategies against strong machine-learning baselines for car retrofit prediction [1]. The research, posted to arXiv on 13 June 2026, examined a dataset linking a prototype-registration system covering 284,271 vehicles with a retrofit-management system containing 48,716 cleaned visits [1]. The authors — led by Ioannis Tzachristas — framed the problem as a multi-stage industrial planning task: estimating whether a newly registered prototype will need a retrofit, which of 15 retrofit packages it will require, and how long the work will take [1][3]. Three LLM-based approaches were compared against traditional tree ensembles. Embedding features were generated using Amazon Titan, direct prompted classification was run through Claude Sonnet 4, and a hybrid ML-plus-LLM stacking pipeline combined both paradigms [1][2]. The results showed a consistent pattern. Embeddings remained useful on tables, achieving a binary AUC of 0.982, but direct prompting collapsed once semantic signal was stripped by hashing, recording a binary AUC of 0.500 and a multiclass weighted F1 of 0.018 [1][3]. Hybrid stacking delivered the best manually built multiclass model, with a weighted F1 of 0.626, slightly ahead of Random Forest with SMOTE at 0.621 and XGBoost at 0.614. An automated AutoGluon pipeline remained the overall leader at 0.654 weighted F1 [2][3]. On the aggregated monthly benchmark, lag-based machine learning outperformed time-series foundation models, though Chronos-small stayed competitive in zero-shot forecasting [1][2]. The findings align with broader concerns about LLM robustness on tabular data. A separate 2025 study found that general-purpose LLMs exhibit a significant deficit in tabular reasoning when evaluated with an LLM-as-a-judge procedure, and that their performance degrades under realistic variations such as missing values and duplicate entities [5]. Earlier work catalogued inference-only LLM methods for tabular prediction and noted that non-fine-tuned models struggle with continuous attributes, though summary-boosting techniques can help on smaller datasets [4]. The retrofit study’s authors concluded that on privacy-constrained industrial tables, LLMs are more effective as complementary components than as replacements for strong tabular baselines [1][3]. The submission, classified under cs.LG, weighed 774 KB [1].