SciText2Eq: Assessing LLMs for Explainable Equation Generation for Scientific Creativity

A new study finds that large language models can generate mathematical equations from scientific text with moderate lexical and syntactic accuracy but struggle with semantic correctness, according to research submitted on 14 June 2026 [1]. The work, titled SciText2Eq, constructs a dataset of AI research papers that pairs contextual passages with ground-truth equations and variable descriptions [1]. Researchers developed an explainable equation generation workflow and tested it across multiple open- and closed-source LLM backbones [1]. The evaluation protocol combined automatic metrics, LLM-based rubrics, and human judgments to assess accuracy, explainability, and alignment between human and machine assessments [1]. Results showed that while LLMs performed moderately on lexical- and syntactic-based similarity measures, they struggled with semantic accuracy [1]. Comparisons between LLM-based evaluations and human judgments revealed limited alignment, highlighting challenges in using LLMs to assess equation quality [1]. The authors provide code and data for reproducibility [1]. The findings contribute to a broader effort to improve scientific language models. Prior work in related domains has explored transfer learning between datasets to boost model performance. For instance, research on the Open Catalyst datasets demonstrated that models trained on the OC20 dataset could be fine-tuned on the smaller OC22 dataset, improving results through transfer learning [4]. Such approaches have also seen success in small-molecule and drug-discovery communities, where models transfer between varying levels of electronic structure calculations or between related tasks [4]. The SciText2Eq study extends these lines of inquiry into the domain of equation generation from unstructured scientific prose. The challenges identified in semantic accuracy echo broader difficulties in aligning machine evaluation with human judgment, a problem that persists across scientific machine-learning applications [1]. The authors note that prior work faced obstacles in unstructured grounding, multi-equation dependency, and human-aligned evaluation, which their dataset and protocol aim to address [1].