跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.19780 · v1

A Threshold-Free Visual-Semantic Matching Metric for Evaluating Consistency in Subject-Driven Image Generation

首次提交 2026-11-15 12:16
版本更新 2026-11-15 12:16
雷达观测 2026-11-17 12:06
arXiv ↗PDF ↗

摘要

In subject-driven text-to-image generation, quantitatively evaluating localized visual consistency across varying poses and environments remains a major challenge. Existing metrics either rely on global image embeddings (CLIP, DINO) and Vision-Language Models (VLMs) that pool spatial representations and miss fine-grained textural artifacts, or depend on scalar threshold-gated dense metrics such as Visual-Semantic Matching (VSM) that suffer from point-alignment dropouts, severe threshold sensitivity, and score instability. To overcome these limitations, we propose TF-VSM, a fully continuous, threshold-free visual-consistency metric paired with an extended feature-disentanglement retraining schedule for dual-branch ResNet feature-aggregation networks. Rather than enforcing rigid binary cutoffs (Ts,Tv)(T_s,T_v), TF-VSM weights every object point by a non-linear semantic match confidence wj=ReLU(s^j)2w_j=ReLU(\hat{s}_j)^2 and continuously integrates visual similarity over all consistency levels r∈[0,1]r\in[0,1], yielding a closed-form confidence-weighted score without any hand-tuned hyperparameters. On a controlled 100-pair benchmark with ground-truth inpainting oracles, TF-VSM attains Pearson 0.6380.638 and Spearman 0.6740.674 oracle correlation, improving over the pretrained VSM baseline (0.454/0.5890.454/0.589) by +0.171/+0.116+0.171/+0.116 after retraining, and drastically outperforming CLIP (−0.052/−0.004-0.052/-0.004), DINO (0.087/0.1210.087/0.121), and VLMs (0.096/0.0030.096/0.003). Six diagnostic ablations confirm that smooth confidence weighting is the right mechanism, that hard semantic gating is brittle (dropping to 0.5330.533 Pearson at Ts=0.8T_s{=}0.8), and that the threshold-free metric is essentially invariant to the training-time loss balance αα that the thresholded metric still depends on.

作者备注:16 pages, 5 figures

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-15 12:16 · 最新观测版本

A Threshold-Free Visual-Semantic Matching Metric for Evaluating Consistency in Subject-Driven Image Generation

In subject-driven text-to-image generation, quantitatively evaluating localized visual consistency across varying poses and environments remains a major challenge. Existing metrics either rely on global image embeddings (CLIP, DINO) and Vision-Language Models (VLMs) that pool spatial representations and miss fine-grained textural artifacts, or depend on scalar threshold-gated dense metrics such as Visual-Semantic Matching (VSM) that suffer from point-alignment dropouts, severe threshold sensitivity, and score instability. To overcome these limitations, we propose TF-VSM, a fully continuous, threshold-free visual-consistency metric paired with an extended feature-disentanglement retraining schedule for dual-branch ResNet feature-aggregation networks. Rather than enforcing rigid binary cutoffs (Ts,Tv)(T_s,T_v), TF-VSM weights every object point by a non-linear semantic match confidence wj=ReLU(s^j)2w_j=ReLU(\hat{s}_j)^2 and continuously integrates visual similarity over all consistency levels r∈[0,1]r\in[0,1], yielding a closed-form confidence-weighted score without any hand-tuned hyperparameters. On a controlled 100-pair benchmark with ground-truth inpainting oracles, TF-VSM attains Pearson 0.6380.638 and Spearman 0.6740.674 oracle correlation, improving over the pretrained VSM baseline (0.454/0.5890.454/0.589) by +0.171/+0.116+0.171/+0.116 after retraining, and drastically outperforming CLIP (−0.052/−0.004-0.052/-0.004), DINO (0.087/0.1210.087/0.121), and VLMs (0.096/0.0030.096/0.003). Six diagnostic ablations confirm that smooth confidence weighting is the right mechanism, that hard semantic gating is brittle (dropping to 0.5330.533 Pearson at Ts=0.8T_s{=}0.8), and that the threshold-free metric is essentially invariant to the training-time loss balance αα that the thresholded metric still depends on.

查看此版本解读 →查看此版本原文 ↗