From Data to Insights: Exploring Program-of-Thoughts Prompting for Chart Summarization

A new approach to automated chart summarization uses zero-shot learning and Python programs to improve numerical accuracy without heavy computational demands, according to research submitted on 25 May 2026 [1][2]. The paper, posted to arXiv, addresses a persistent weakness in visual language models: the inability to reliably verify statistical facts when describing charts [1]. While VLMs have advanced in semantic understanding, the authors note that existing methods remain computationally expensive and lack robust fact-checking mechanisms [2]. To bridge this gap, the researchers propose routing lightweight VLMs through Python programs as intermediaries. The programs derive summary statistics before the model generates natural-language descriptions, a sequence the authors call a Program-of-Thought strategy [2]. A novel chart-to-dictionary auxiliary task replaces traditional chart-to-table conversion, offering a more flexible data structure that integrates with the reasoning pipeline [1][2]. Experimental results show the technique performs on par with current chart summarization methods across both semantic and factual metrics [1][2]. The code has been made available through an anonymous repository [2]. The work arrives as multimodal AI systems face scrutiny over factual reliability. Google's Gemini family of models, which process text, code, images, and video natively, suspended image generation of people in early 2024 after users reported historical inaccuracies and bias [3]. Subsequent updates through the Gemini 1.5 and 3 series, released through 2025, focused on reducing hallucinations and improving output dependability [3]. Chart understanding sits at the intersection of computer vision and structured reasoning, domains where even large-scale models have demonstrated brittleness. The arXiv paper's reliance on zero-shot learning means the system requires no task-specific fine-tuning, a design choice that reduces computational overhead compared to fully supervised alternatives [2]. The submission was facilitated through arXivLabs, a framework that allows community collaborators to develop and share new features on the arXiv platform [1]. The paper also lists integrations with Hugging Face for model and dataset distribution [1].