← 研踪 PaperTrailCVF · CVPR 2026:cvf:1ddbb68db62746a94c22a9c6017c44d0 · 记录 #1
mVLM: A Vision Language Model for mNPUs
来源日期(仅到年份) 2026
来源记录日期 2026
雷达观测 2026-11-01 23:58
摘要
The proliferation of low-power intelligent processors with integrated Neural Processing Units (NPUs), called mNPUs, has created new opportunities for on-device generative AI, benefitting end devices like smart wearables and small robots. However, deploying Vision-Language Models (VLMs) on mNPUs is severely hindered by stringent memory constraints and limited operator support. To bridge this critical gap, we propose mVLM, the first lightweight-oriented VLM architecture designed for mNPUs. It is comprised of our proposed OverMod encoder and AttSSM decoder. OverMod is a lightweight dynamic convolutional network inspired by biomimetic vision, incorporating our novel Global Spatial Modulation mechanism to enable adaptive, high-fidelity feature extraction using only NPU-friendly operators. AttSSM leverages a highly efficient State Space Model (SSM) core, augmented with multi-scale feature fusion and Global Context Dynamic Modulation mechanism, to perform robust sequential modeling. Furthermore, we introduce a coordinated full-parameter quantization strategy that preserves precision across the encoder-decoder boundary, alongside hand-optimized operators for unsupported modules like SSMs. mVLM achieves a competitive CIDEr score of 117.8 on the COCO Karpathy test split and, for the first time, demonstrates the feasibility of millisecond-level VLM inference on a mNPU platform.
论文发现出处
这些页面提供论文线索,不代表同行评审或质量认证。
观测到的版本
平台记录号代表本站观测到的元数据变化,不是作者修订号。来源日期仅到月份或年份时,按该月或该年首日存档。
记录 #1 · 2026 · 最新观测版本
mVLM: A Vision Language Model for mNPUs
The proliferation of low-power intelligent processors with integrated Neural Processing Units (NPUs), called mNPUs, has created new opportunities for on-device generative AI, benefitting end devices like smart wearables and small robots. However, deploying Vision-Language Models (VLMs) on mNPUs is severely hindered by stringent memory constraints and limited operator support. To bridge this critical gap, we propose mVLM, the first lightweight-oriented VLM architecture designed for mNPUs. It is comprised of our proposed OverMod encoder and AttSSM decoder. OverMod is a lightweight dynamic convolutional network inspired by biomimetic vision, incorporating our novel Global Spatial Modulation mechanism to enable adaptive, high-fidelity feature extraction using only NPU-friendly operators. AttSSM leverages a highly efficient State Space Model (SSM) core, augmented with multi-scale feature fusion and Global Context Dynamic Modulation mechanism, to perform robust sequential modeling. Furthermore, we introduce a coordinated full-parameter quantization strategy that preserves precision across the encoder-decoder boundary, alongside hand-optimized operators for unsupported modules like SSMs. mVLM achieves a competitive CIDEr score of 117.8 on the COCO Karpathy test split and, for the first time, demonstrates the feasibility of millisecond-level VLM inference on a mNPU platform.
查看此版本解读 →查看此版本原文 ↗