Ping Luo, Jiahuan Wang, Ziqing Wen, Kele Xu, Dongsheng Li, Tao Sun
In natural science applications, federated learning (FL) enables collaborative training of models that approximate solutions to partial differential equations (PDEs) for repeated use in distributed settings. However, the input conditions of FL-PDE training samples involve multiple coupled parameters, making data heterogeneity across clients difficult to analyse. We propose solution-space PDE-Dirichlet, which groups existing solution labels into task-specific bins and distributes samples using the classical Dirichlet allocation rule in FL, providing a common partitioning approach across PDE tasks. Theoretical analysis shows an association between differences in solution-bin composition and gradient disagreement, while experiments on three controlled operators and four public simulation datasets quantify the relationships between this heterogeneity, gradient disagreement and prediction error. Based on this connection, we develop Solution-Bin Controlled Federated Optimization, which uses pooled bin-gradient prototypes to correct the component of client gradients associated with bin composition while preserving the sample-weighted training objective. Experiments on low-viscosity Burgers and Darcy flow show that this correction reduces prediction error compared with federated averaging and narrows the gap from bin-balanced IID training.
Ziqing Wen, Ping Luo, Jiahuan Wang, Xiaoge Deng, Tao Sun, Dongsheng Li
Score-based Generative Models (SGMs) have achieved remarkable success in generation tasks by establishing a diffusion process that gradually perturbs real data into Gaussian noise and then learning a reverse process to reconstruct the data from noise. However, conventional denoising methods, such as Langevin dynamics and numerical stochastic differential equation solvers, rely on randomness but suffer from slow generation speeds due to the high number of score function evaluations. To address this, we propose a plug-and-play Markov adaptive momentum sampling method, inspired by the strong connection between the SGMs' sampling process and stochastic gradient descent optimization. Our approach accelerates the transformation process without introducing additional hyperparameters. Theoretically, we establish a rigorous foundation by analyzing the relationship between the distribution of samples generated by our method and the original data distribution. This provides a novel perspective on the analysis of sampling distributions. Empirically, our method demonstrates superior efficiency, generating high-quality images/graphs in significantly fewer sampling steps, achieving a speedup of 2x to 5x while maintaining or surpassing baseline performance on various image and graph generation benchmarks. These results underscore the effectiveness of adaptive momentum in enhancing the efficiency of SGMs.
Xuyuan Xiong, Siyu Chen, Zihan Shao, Yue Su, Weiyang Jin, Ping Luo, Zhuoran Yang, Xihui Liu
Looped transformers reuse a shared block of layers to increase computational depth without increasing the number of trainable parameters. They are trained with backpropagation through time (BPTT). To save training compute, BPTT is often truncated to the last few loops, but truncation yields biased gradients that may weaken the benefit of looping. In this work, we study the value of truncation in looped transformer training in a systematic way: we jointly vary model size, training tokens, and forward and backward loop counts across nearly 600 training runs and fit a unified scaling law that captures the effects of looping and truncation through effective parameters. Despite the training-compute savings of truncation, our fitted law favors full BPTT when forward and backward loop counts are jointly optimized. In matched-compute runs, full BPTT through fewer loops consistently outperforms truncated settings of the same compute. The law further shows that at fixed model size, the compute-optimal loop count and training-token budget both grow with compute. When model size is also free, the law identifies two fully trained loops as the best looped setting. Experimentally, this setting achieves lower validation loss than the compute-optimal non-looped model with about 27% fewer non-embedding parameters and only 8% more training compute.