Visual Token Coding for Video Multimodal Large Language Models
Chenxin Fang, Tao Chen, JunChao You, Jun Peng, Yiyi Zhou, Rongrong Ji
原文摘要
In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding ($VTC$). Inspired by classical video coding principles, e.g., HEVC, VTC performs structured compression by predicting the I/P frames of a video and measuring their frame-wise residuals to estimate token redundancy. Based on this base paradigm, we further extend VTC with a set of dynamic budgeting designs towards more cost-effective token coding, termed $VTC_{dy}$. To validate VTC and $VTC_{dy}$, we apply them to three MLLMs and conduct experiments on a set of video understanding benchmarks. The experimental results show that as a base paradigm, VTC can achieve an average performance retention of 98.4% on Qwen3-VL under a 50% token compression ratio, and the enhanced $VTC_{dy}$ can further improve the retention to 100.3%, outperforming a set of compared methods. Moreover, as a plug-and-play design, VTC requires no additional tuning of MLLMs for token coding. Our code is given in the supplementary materials.
cs.CV