Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning
Yuan Zhuang, Yuexin Bian, Sihong He, Jie Feng, Qing Su, Songyang Han, Jonathan Petit, Shihao Ji 等 10 位作者
AI 速览：作者提出将 LoRA 作为从零训练评论家的结构正则化手段，冻结随机初始化的基矩阵、只训练低秩适配器，并在连续控制任务上评估。
Scaling critic capacity is a promising direction for improving off-policy reinforcement learning (RL). However, recent work shows that larger critics are prone to overfitting and instability in replay-based bootstrapped training. In this paper, we propose using Low-Rank Adaptation (LoRA) as a structural regularizer for critic learning from scratch. Our approach freezes randomly initialized base matrices and optimizes only the corresponding low-rank adapters, thereby constraining the learned weight changes to low-rank residuals. We evaluate our method on state-based continuous-control tasks, using SAC with SimbaV2 and BRC on DMC-Hard, and FastTD3 with its SimbaV2 variant on IsaacLab. Empirically, LoRA reduces critic loss during training and improves overall policy performance, achieving the best or competitive results on most tasks. Extensive experiments demonstrate that our low-rank updates provide a simple and effective form of structural regularization for critic learning in off-policy RL.
Ruiqing Sun, Dawei Feng, Sen Yang, Ronghang Wang, Huaiyuan Song, Bo Ding, Yijie Wang, Huaimin Wang
De novo 3D molecular design with multiple properties and fragment-retention constraints is formulated as a constrained multi-objective optimization problem. Existing 3D diffusion models provide strong capabilities for generating novel molecules, but adapting them to new conditions and multi-objective optimization tasks remains difficult. We present Diffusion-based Evolutionary Molecular Optimization (DEMO), a framework that draws solely on the unconditional molecular prior of a frozen pretrained diffusion model to generate chemically valid 3D candidates, while evolutionary search handles conditional generation and constrained multi-objective optimization, without retraining or fine-tuning. DEMO performs crossover and mutation on intermediate diffusion states through an Evolutionary-Guided Diffusion (EGD) operator, combines 2D topology and 3D geometry in Structure-Aware Environmental Selection (SAES) to preserve structural diversity, and uses three cooperating populations to coordinate structural exploration, fragment-constraint satisfaction, and property optimization. Experimental results demonstrate that EGD outperforms task-specific conditional generators in property targeting; SAES preserves structural diversity while retaining competitive hypervolume in multi-objective optimization; and DEMO improves objective convergence and generates diverse feasible molecules under fragment constraints. These results show how evolutionary search can reuse existing diffusion models for diverse molecular design requirements.
Renqi Chen, Zeyin Tao Alibaba Group, Jianming Guo, Jingzhe Zhu, Yiheng Peng, Qingqing Sun, Tianyi Zhang Alipay, Shuai Chen Ant Group
E-commerce risk management requires aggregating diverse, deeply embedded web data through multi-step, stateful interactions, which traditional scraping methods and most existing Graphical User Interface (GUI) agents cannot handle. These agents are typically limited to single-step tasks and lack the ability to manage dynamic, interactive content critical for effective risk assessment. To address this challenge, we introduce RISK, a novel framework designed to build and deploy GUI agents for this domain. RISK integrates three components: (1) RISK-Data, a dataset of 8,492 single-step and 2,386 multi-step interaction trajectories, collected through a high-fidelity browser framework and a meticulous data curation process; (2) RISK-Bench, a benchmark with 802 single-step and 320 multi-step trajectories across three difficulty levels for standardized evaluation; and (3) RISK-R1, a R1-style reinforcement fine-tuning framework considering four aspects: (i) Output Format Constraint, (ii) Single-step and (iii) Multi-step Level Reward, and (iv) Task Level Reweight. Experiments show that RISK-R1 achieves a 6.8% improvement in offline single-step and an 8.8% improvement in offline multi-step, using only 7.2% of the parameters of the SOTA baseline. Moreover, it attains a top task success rate of 70.5% in online evaluation. RISK provides a scalable, domain-specific solution for automating complex web interactions in e-commerce risk management. The code is available at https://github.com/RenqiChen/RISK-GUI.
尚无 AI 解读，展示原文摘要
cs.CL
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2026 · 正式发表