跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.21459 · v1

Balanced Twin Weight Factorization for Efficient MLLMs

首次提交 2026-11-17 01:43
版本更新 2026-11-17 01:43
雷达观测 2026-11-17 16:10
arXiv ↗PDF ↗

摘要

Weight Factorization (WF) methods aim to reduce a model's latency, memory, or both. These typically decompose a pretrained model into two or more factors that approximate the original model weights with fewer parameters and floating point operations (FLOPs). However, translating fewer FLOPs into reduced latency is challenging for WF methods in practice due to overhead introduced by replacing a single large weight matrix with multiple smaller ones. To address this, we propose Balanced Twin Weight Factorization (BTWF), which uses the unique structure of factorized models to simultaneously lower latency and increase learning capacity. Specifically, BTWF operates in the shared basis WF setting, where two or more layers are decomposed into a shared basis matrix and a layer-specific mixing matrix. If these shared layers have the same input (i.e., are twins), then we need only perform this shared forward pass once, reducing inference time. However, twin layers are often of varying sizes, and methods from prior work tend to favor the larger layer. In contrast, our approach boosts performance by combining Factor Component Balancing (FCB), which ensures every shared member is well represented in the selected factors, with Block Contribution Balancing (BCB), which considers downstream model effects. Further, we also show that an overparameterized BTWF, where some layer's factors result in higher FLOPs than the original during training, can boost task performance without additional resources at test time. Experiments on six models over nine benchmarks show BTWF obtains a 2-6% gain over the state-of-the-art without expensive finetuning (1-3.5% gain with it) at comparable computational costs.

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-17 01:43 · 最新观测版本

Balanced Twin Weight Factorization for Efficient MLLMs

Weight Factorization (WF) methods aim to reduce a model's latency, memory, or both. These typically decompose a pretrained model into two or more factors that approximate the original model weights with fewer parameters and floating point operations (FLOPs). However, translating fewer FLOPs into reduced latency is challenging for WF methods in practice due to overhead introduced by replacing a single large weight matrix with multiple smaller ones. To address this, we propose Balanced Twin Weight Factorization (BTWF), which uses the unique structure of factorized models to simultaneously lower latency and increase learning capacity. Specifically, BTWF operates in the shared basis WF setting, where two or more layers are decomposed into a shared basis matrix and a layer-specific mixing matrix. If these shared layers have the same input (i.e., are twins), then we need only perform this shared forward pass once, reducing inference time. However, twin layers are often of varying sizes, and methods from prior work tend to favor the larger layer. In contrast, our approach boosts performance by combining Factor Component Balancing (FCB), which ensures every shared member is well represented in the selected factors, with Block Contribution Balancing (BCB), which considers downstream model effects. Further, we also show that an overparameterized BTWF, where some layer's factors result in higher FLOPs than the original during training, can boost task performance without additional resources at test time. Experiments on six models over nine benchmarks show BTWF obtains a 2-6% gain over the state-of-the-art without expensive finetuning (1-3.5% gain with it) at comparable computational costs.

查看此版本解读 →查看此版本原文 ↗