HiLo-Token: Input-Adaptive High-Low Frequency Token Compression for Efficient Image Editing Model Serving
Haoran You, Yotam Nitzan, Lingzhi Zhang, Yifan Gong, Mang-Tik Chiu, Connelly Barnes, Yan Kang, Yuqian Zhou 等 11 位作者
原文摘要
Creative image editing tools, such as Adobe Photoshop's Remove and Generative Fill buttons, are central to everyday customer use and account for a major share of traffic. However, current generative AI models face significant latency challenges, which become even more pronounced as the field transitions from convolution-based U-Nets to Diffusion Transformers (DiTs). In our evaluation on hundreds of representative image editing samples spanning a wide range of mask ratios, the DiT module alone accounts for an average of 73% of the total model latency, even after being distilled from 50 timesteps down to 8 timesteps. To tackle this challenge, we propose $\textbf{HiLo-Token}$, an input-adaptive token compression framework that allocates a larger token budget to high-frequency, rich-context regions while assigning fewer tokens to low-frequency areas. Specifically, for the editing region specified by the user mask, we retain all tokens within a dilated mask to preserve strong locality and contextual relevance. Outside the editing region, we introduce a simple yet effective high-frequency token selection strategy based on spatial frequency to capture important local details, while using tokens from a heavily downsampled image to represent low-frequency components and preserve the coarse global structure. Extensive experiments on production-level evaluation data validate the effectiveness of the proposed method, which achieves 3.13$\times$, 2.59$\times$, and 1.67$\times$ DiT speedups on A100-80GB for image editing tasks across small, medium, and large mask ratio categories with average ratios of 6.38%, 15.92%, and 35.36%, respectively, without any regression in generation quality. In production, where the editing pipeline is served as disaggregated, independently autoscaled GPU workers, these per-request savings translate into a 33% reduction in the GPUs required to serve the models.
cs.CV · cs.AI