GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Models

Researchers have introduced the GeoNatureAgent Benchmark, a new evaluation tool for AI agents in environmental geospatial analysis, comprising 93 tasks across 18 categories.

The GeoNatureAgent Benchmark is designed to assess AI agents operating through structured tool calling against real APIs, a capability crucial for environmental scientists who spend significant time on data wrangling. Seven large language models (LLMs) were evaluated under three temperature-1.0 seeds, with Claude Sonnet 4 leading at 60.8% +/- 0.8%, followed by DeepSeek V3.2 at 56.3% +/- 3.1%^[1]. The benchmark revealed that the cost-accuracy Pareto frontier is dominated by open-weight models, with DeepSeek V3.2 offering 93% of Claude's capability at 11 times lower cost ($0.011 per case). However, comparison tasks remained challenging, with all models scoring 0% on close-value comparisons, highlighting systematic reasoning limits. Structured tool calling against a real API proved more discriminative than general-purpose GIS benchmarks, with accuracies 25-35 points lower. In a related development, a Multi-Modal Agent framework was proposed for power distribution defect detection, evaluating the performance of multimodal foundation models in perception, reasoning, and tool usage^[2].