LLM Sparsity Prior for Robust Feature Selection

A new statistical framework called the LLM Sparsity Prior (LSP) uses large language models to guide variable selection in high-dimensional datasets while remaining robust when the model’s initial suggestions are inaccurate, according to a paper posted to arXiv on May 21 [1]. The method addresses a known weakness in earlier approaches such as LLM-Lasso, which can degrade substantially when the weights generated by a large language model are of poor quality [1][2]. To counter this, the authors first built a framework for quantifying the quality of LLM-generated weights, allowing systematic evaluation across different weight regimes [2]. The LSP integrates those weights into the prior inclusion probabilities of two established Bayesian variable-selection models: Spike-and-Slab and Spike-and-Slab Lasso [1][2]. Two interpretable hyperparameters govern global sparsity and weight concentration, and hierarchical hyperpriors placed on these parameters let the model dynamically discount uninformative or misleading weights [2]. The design means the model can still benefit from accurate LLM input without suffering when the input is noisy. Variable selection — choosing which predictors matter in a dataset with many candidate variables — is a long-standing challenge in statistics. In classical machine learning, reinforcement learning agents face a related exploration–exploitation dilemma, balancing the use of known information against the search for better actions in an uncertain environment [5]. The LSP’s hierarchical structure can be seen as a Bayesian way of managing a similar tension: it exploits the LLM’s domain knowledge when it is reliable and explores alternative feature sets when the prior information is weak. The paper also details principled prompt-engineering strategies and validates the method on a private medical dataset studying Acute Kidney Injury [2]. On that dataset, LSP improved prediction accuracy and identified clinically relevant features that baseline methods missed, with particular effectiveness in low-data regimes [2]. The method showed robustness to prompt variation, a practical concern given that the quality of LLM outputs can shift with small changes in how a question is phrased [2]. While the work focuses on feature selection, it sits alongside broader efforts to incorporate human or machine-generated feedback into model training. In reinforcement learning from human feedback, for instance, a reward model is trained on human preference data and then used to guide an agent’s policy, though collecting high-quality preference data remains expensive and can introduce bias if the sample is not representative [4]. The LSP framework similarly depends on the quality of its external input — the LLM-generated weights — but adds a statistical mechanism to mitigate the cost of low-quality signals.