arXiv:2507.10029 · v3
Mix-Opt: Mixed Optimization for Memory-Efficient Personalization of Text-to-Image Diffusion Models
Mix-Opt:面向文本到图像扩散模型的内存高效个性化混合优化
摘要
Memory-efficient personalization is essential for adapting text-to-image diffusion models while preserving user privacy and operating within the limited computational resources of edge devices. To this end, we propose Mix-Opt, a novel mixed optimization framework that adaptively chooses between backpropagation on low-resolution images (BP-low) and zeroth-order optimization on high-resolution images (ZO-high), guided by the characteristics of the diffusion process. As observed in our experiments, BP-low efficiently adapts the model to target-specific features but suffers from structural distortions due to resolution mismatch. Conversely, ZO-high refines high-resolution details with minimal memory overhead but faces slow convergence when applied without prior adaptation. By complementing both methods, our framework leverages BP-low for effective personalization while using ZO-high to maintain structural consistency, achieving memory-efficient and high-quality fine-tuning. To maximize the efficacy of both BP-low and ZO-high, we introduce a timestep-aware probabilistic function that dynamically selects the appropriate optimization strategy during training. This function mitigates overfitting from BP-low at high timesteps, where structural information is critical, while ensuring that ZO-high is applied more effectively as training progresses. Experimental results across both U-Net- and DiT-based diffusion models demonstrate that Mix-Opt achieves competitive personalization performance while significantly reducing memory consumption. Mix-Opt enables scalable, high-quality personalization without modifying the inference process or increasing inference latency.
作者备注:Extended version accepted to NeurIPS 2026. The earlier 4-page version received the Best Paper Award at the ICCV 2025 LIMIT Workshop