← 研踪 PaperTrailarXiv:2611.19780 · v1
A Threshold-Free Visual-Semantic Matching Metric for Evaluating Consistency in Subject-Driven Image Generation
首次提交 2026-11-15 12:16
版本更新 2026-11-15 12:16
雷达观测 2026-11-17 12:06
摘要
In subject-driven text-to-image generation, quantitatively evaluating localized visual consistency across varying poses and environments remains a major challenge. Existing metrics either rely on global image embeddings (CLIP, DINO) and Vision-Language Models (VLMs) that pool spatial representations and miss fine-grained textural artifacts, or depend on scalar threshold-gated dense metrics such as Visual-Semantic Matching (VSM) that suffer from point-alignment dropouts, severe threshold sensitivity, and score instability. To overcome these limitations, we propose TF-VSM, a fully continuous, threshold-free visual-consistency metric paired with an extended feature-disentanglement retraining schedule for dual-branch ResNet feature-aggregation networks. Rather than enforcing rigid binary cutoffs , TF-VSM weights every object point by a non-linear semantic match confidence and continuously integrates visual similarity over all consistency levels , yielding a closed-form confidence-weighted score without any hand-tuned hyperparameters. On a controlled 100-pair benchmark with ground-truth inpainting oracles, TF-VSM attains Pearson and Spearman oracle correlation, improving over the pretrained VSM baseline () by after retraining, and drastically outperforming CLIP (), DINO (), and VLMs (). Six diagnostic ablations confirm that smooth confidence weighting is the right mechanism, that hard semantic gating is brittle (dropping to Pearson at ), and that the threshold-free metric is essentially invariant to the training-time loss balance that the thresholded metric still depends on.
作者备注:16 pages, 5 figures
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-15 12:16 · 最新观测版本
A Threshold-Free Visual-Semantic Matching Metric for Evaluating Consistency in Subject-Driven Image Generation
In subject-driven text-to-image generation, quantitatively evaluating localized visual consistency across varying poses and environments remains a major challenge. Existing metrics either rely on global image embeddings (CLIP, DINO) and Vision-Language Models (VLMs) that pool spatial representations and miss fine-grained textural artifacts, or depend on scalar threshold-gated dense metrics such as Visual-Semantic Matching (VSM) that suffer from point-alignment dropouts, severe threshold sensitivity, and score instability. To overcome these limitations, we propose TF-VSM, a fully continuous, threshold-free visual-consistency metric paired with an extended feature-disentanglement retraining schedule for dual-branch ResNet feature-aggregation networks. Rather than enforcing rigid binary cutoffs , TF-VSM weights every object point by a non-linear semantic match confidence and continuously integrates visual similarity over all consistency levels , yielding a closed-form confidence-weighted score without any hand-tuned hyperparameters. On a controlled 100-pair benchmark with ground-truth inpainting oracles, TF-VSM attains Pearson and Spearman oracle correlation, improving over the pretrained VSM baseline () by after retraining, and drastically outperforming CLIP (), DINO (), and VLMs (). Six diagnostic ablations confirm that smooth confidence weighting is the right mechanism, that hard semantic gating is brittle (dropping to Pearson at ), and that the threshold-free metric is essentially invariant to the training-time loss balance that the thresholded metric still depends on.
查看此版本解读 →查看此版本原文 ↗