arXiv:2605.13322 · v3

KamonBench: A Grammar-Based Dataset for Evaluating Compositional Factor Recovery in Vision-Language Models

首次提交 2026-05-13 18:35

版本更新 2026-10-28 15:12

雷达观测 2026-10-30 10:22

摘要

Kamon (family crests) are an important part of Japanese culture and a natural test case for compositional visual recognition: each crest combines a small number of symbolic choices, but the space of possible descriptions is sparse. We introduce KamonBench, a grammar-based image-to-structure benchmark with 20,000 synthetic composite crests and auxiliary component examples. Each composite crest is paired with a formal kamon description language (kamon yogo) description, a segmented Japanese analysis, an English translation, and a non-linguistic program code. Because each synthetic crest is generated from known factors, namely container, modifier, and motif, KamonBench supports evaluation beyond caption-level accuracy: direct program-code factor metrics, controlled factor recombination splits, counterfactual motif-sensitivity groups under fixed container-modifier contexts, and linear probes of factor accessibility. We include baseline results for a ViT encoder with a Transformer decoder, two VGG n-gram decoders with and without learned positional masks, and fine-tuned Qwen3-VL-4B and Qwen3-VL-8B models. KamonBench therefore provides a controlled testbed for sparse compositional visual recognition and factor recovery in vision-language models.

作者备注：Preprint