← 研踪 PaperTrailarXiv:2611.19329 · v1
Gradient-Based Co-Design of Systems and Reinforcement Learning Policies with Differentiable Simulators
首次提交 2026-11-14 18:51
版本更新 2026-11-14 18:51
雷达观测 2026-11-17 12:06
摘要
Co-design is the problem of jointly optimizing the design of a system and the policy that operates it. Neither can be chosen independently: a system design is only as good as the policy trained for it, and the optimal policy changes with the design. When that policy is learned by reinforcement learning (RL), evaluating a candidate design requires a full policy training run, and the design is usually optimized by black-box search such as Bayesian optimization (BO) or evolution strategies (ES). These methods scale poorly with the design dimension and ignore the differentiable structure of modern simulators and, in particular, the sensitivity of the optimal policy with respect to the design. We instead derive a gradient-based co-design method, where the design gradient decomposes into a direct effect of changing the design at a fixed policy and an indirect effect through the design-dependent optimal policy. The indirect effect requires the sensitivity of the converged policy to the design, which we obtain by applying the implicit function theorem to the stationarity condition of the policy gradient. Based on this sensitivity, we develop a proof-of-concept co-design algorithm with stochastic value gradient (SVG)-style policy optimization, prove that it converges to an approximately stationary design, and demonstrate on a differentiable networked-system benchmark with high-dimensional design spaces that it consistently outperforms BO and ES.
作者备注:31 pages, 5 figures, 3 tables
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-14 18:51 · 最新观测版本
Gradient-Based Co-Design of Systems and Reinforcement Learning Policies with Differentiable Simulators
Co-design is the problem of jointly optimizing the design of a system and the policy that operates it. Neither can be chosen independently: a system design is only as good as the policy trained for it, and the optimal policy changes with the design. When that policy is learned by reinforcement learning (RL), evaluating a candidate design requires a full policy training run, and the design is usually optimized by black-box search such as Bayesian optimization (BO) or evolution strategies (ES). These methods scale poorly with the design dimension and ignore the differentiable structure of modern simulators and, in particular, the sensitivity of the optimal policy with respect to the design. We instead derive a gradient-based co-design method, where the design gradient decomposes into a direct effect of changing the design at a fixed policy and an indirect effect through the design-dependent optimal policy. The indirect effect requires the sensitivity of the converged policy to the design, which we obtain by applying the implicit function theorem to the stationarity condition of the policy gradient. Based on this sensitivity, we develop a proof-of-concept co-design algorithm with stochastic value gradient (SVG)-style policy optimization, prove that it converges to an approximately stationary design, and demonstrate on a differentiable networked-system benchmark with high-dimensional design spaces that it consistently outperforms BO and ES.
查看此版本解读 →查看此版本原文 ↗