← 研踪 PaperTrailarXiv:2604.24589 · v2
A systematic evaluation of vision-language models for observational astronomical reasoning tasks
首次提交 2026-04-27 23:11
版本更新 2026-10-26 15:52
雷达观测 2026-10-29 00:36
摘要
Vision-language models (VLMs) are increasingly proposed as general-purpose tools for scientific data interpretation, yet their reliability on real astronomical observations across diverse modalities remains untested. We present AstroVLBench, a benchmark of over 4,100 expert-verified instances across five tasks spanning optical imaging, radio interferometry, multi-wavelength photometry, time-domain light curves, and optical spectroscopy. We systematically evaluate six state-of-the-art frontier models and find that performance is strongly modality-dependent and uncorrelated with general benchmark rankings. Mechanistic cross-model ablations reveal three concrete bottlenecks. First, prompt grounding exhibits marked model and task dependence: physical guidance yields more balanced classifications on optical imaging, whereas phenomenological prompts better navigate undersampled time-domain data. Second, presenting one-dimensional measurements as numerical tables rather than rendered plots consistently improves light-curve classification across all models by up to 23 percentage points, demonstrating that plot rendering can obscure physically relevant signal. Third, an expert audit of optical spectroscopy reveals a notable "right-answer-wrong-reason" phenomenon: across evaluated models, an average of 26% of correct predictions are accompanied by physically inconsistent or invalid reasoning chains, showing that accuracy alone is insufficient for trustworthy scientific deployment. AstroVLBench provides the first systematic, multi-modal baselines for VLMs in observational astronomy and identifies the representation, grounding, and reasoning bottlenecks that must be addressed before VLMs can be trusted as autonomous scientific interpreters.
作者备注：42 pages, 15 figures, 11 tables. Accepted at NeurIPS 2026, Track on Evaluations and Datasets
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-10-26 15:52 · 最新观测版本
A systematic evaluation of vision-language models for observational astronomical reasoning tasks
Vision-language models (VLMs) are increasingly proposed as general-purpose tools for scientific data interpretation, yet their reliability on real astronomical observations across diverse modalities remains untested. We present AstroVLBench, a benchmark of over 4,100 expert-verified instances across five tasks spanning optical imaging, radio interferometry, multi-wavelength photometry, time-domain light curves, and optical spectroscopy. We systematically evaluate six state-of-the-art frontier models and find that performance is strongly modality-dependent and uncorrelated with general benchmark rankings. Mechanistic cross-model ablations reveal three concrete bottlenecks. First, prompt grounding exhibits marked model and task dependence: physical guidance yields more balanced classifications on optical imaging, whereas phenomenological prompts better navigate undersampled time-domain data. Second, presenting one-dimensional measurements as numerical tables rather than rendered plots consistently improves light-curve classification across all models by up to 23 percentage points, demonstrating that plot rendering can obscure physically relevant signal. Third, an expert audit of optical spectroscopy reveals a notable "right-answer-wrong-reason" phenomenon: across evaluated models, an average of 26% of correct predictions are accompanied by physically inconsistent or invalid reasoning chains, showing that accuracy alone is insufficient for trustworthy scientific deployment. AstroVLBench provides the first systematic, multi-modal baselines for VLMs in observational astronomy and identifies the representation, grounding, and reasoning bottlenecks that must be addressed before VLMs can be trusted as autonomous scientific interpreters.
查看此版本解读 →查看此版本原文 ↗