← 研踪 PaperTrailarXiv:2605.12154 · v2
MM-OptBench: A Solver-Grounded Benchmark for Multimodal Optimization Modeling
首次提交 2026-05-12 22:07
版本更新 2026-11-04 00:07
雷达观测 2026-11-04 23:39
摘要
Optimization modeling translates real decision-making problems into mathematical optimization models and solver-executable implementations. Although language models are increasingly used to generate optimization formulations and solver code, existing benchmarks for this capability are almost entirely text-only. This omits many optimization-modeling tasks that arise in operational practice, where requirements are described in text but essential instance information is often conveyed through visual artifacts such as tables, graphs, maps, schedules, and planning dashboards. We introduce multimodal optimization modeling, a benchmark setting where models must construct both a mathematical formulation and executable solver code from a text-and-visual problem specification. To evaluate this setting, we develop a solver-grounded framework that generates structured optimization instances, verifies each instance with an exact solver, and builds both model-facing inputs and hidden reference files from the same verified source. We instantiate it as MM-OptBench, a benchmark of 780 solver-verified instances spanning 6 optimization families, 26 subcategories, and 3 structural difficulty levels. We evaluate 9 multimodal large language models (MLLMs), including 6 frontier general-purpose and 3 math-specialized models, with aggregate, family-level, difficulty-level, and failure-mode analyses. The task remains far from solved: the best two models reach 52.1% and 51.3% pass@1, while on average across the six general-purpose MLLMs, pass@1 is 43.4% on easy and 15.9% on hard instances. All three math-specialized MLLMs solve 0/780 instances. Post-hoc extraction diagnostics and oracle-reading evaluation reveal difficulties in both multimodal grounding and downstream solver-artifact generation. MM-OptBench provides a concrete testbed for solver-grounded, decision-oriented multimodal intelligence.
作者备注：Paper accepted at NeurIPS 2026. The dataset and code are available at https://github.com/ZhongLIFR/MM-OptBench
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-04 00:07 · 最新观测版本
MM-OptBench: A Solver-Grounded Benchmark for Multimodal Optimization Modeling
Optimization modeling translates real decision-making problems into mathematical optimization models and solver-executable implementations. Although language models are increasingly used to generate optimization formulations and solver code, existing benchmarks for this capability are almost entirely text-only. This omits many optimization-modeling tasks that arise in operational practice, where requirements are described in text but essential instance information is often conveyed through visual artifacts such as tables, graphs, maps, schedules, and planning dashboards. We introduce multimodal optimization modeling, a benchmark setting where models must construct both a mathematical formulation and executable solver code from a text-and-visual problem specification. To evaluate this setting, we develop a solver-grounded framework that generates structured optimization instances, verifies each instance with an exact solver, and builds both model-facing inputs and hidden reference files from the same verified source. We instantiate it as MM-OptBench, a benchmark of 780 solver-verified instances spanning 6 optimization families, 26 subcategories, and 3 structural difficulty levels. We evaluate 9 multimodal large language models (MLLMs), including 6 frontier general-purpose and 3 math-specialized models, with aggregate, family-level, difficulty-level, and failure-mode analyses. The task remains far from solved: the best two models reach 52.1% and 51.3% pass@1, while on average across the six general-purpose MLLMs, pass@1 is 43.4% on easy and 15.9% on hard instances. All three math-specialized MLLMs solve 0/780 instances. Post-hoc extraction diagnostics and oracle-reading evaluation reveal difficulties in both multimodal grounding and downstream solver-artifact generation. MM-OptBench provides a concrete testbed for solver-grounded, decision-oriented multimodal intelligence.
查看此版本解读 →查看此版本原文 ↗