← 研踪 PaperTrailarXiv:2608.01745 · v2
Lagrangian or Lyapunov? Multi-Agent Reinforcement Learning under Coupled Finite-Horizon Constraints
首次提交 2026-08-03 14:11
版本更新 2026-11-07 16:21
雷达观测 2026-11-10 11:14
摘要
Radio resource management in dense networks is subject to budgets that must hold on every realized trajectory, carried by different agent groups: an energy budget at each base station (BS) and a handover budget at each user, coupled through the association decisions that consume both. Constrained multi-agent reinforcement learning (MARL) handles such budgets using Lagrangian multipliers updated across episodes or Lyapunov virtual queues updated per slot, but their relative behavior under coupled two-sided constraints remains unclear. We formulate this problem and introduce a windowed proportional-fairness objective that captures the temporal distribution of service beyond the classical horizon-average formulation. We characterize the distinct budget-regulation timescales of the two mechanisms through complementary theoretical bounds and identify queue-price buildup as a key mechanism governing finite-horizon budget pacing. Using a common heterogeneous MARL architecture, we compare three utility surrogates and both constraint-handling mechanisms. The results show that utility design primarily determines throughput and fairness, whereas constraint-price dynamics govern training efficiency and budget pacing. Under the default setting, the queue-based PF policy reaches near-final performance after a single training episode, but its unscaled penalty can lead to delayed pacing under tight budgets. Guided by the analysis, we show that queue-price calibration reduces the empirical drift constant by an order of magnitude and substantially narrows the performance gap to Lagrangian methods, which adapt constraint prices across episodes at higher training cost. These findings provide practical design guidelines for constrained MARL.
作者备注：Revised version with updated title, theoretical analysis, and experiments. 14 pages
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-07 16:21 · 最新观测版本
Lagrangian or Lyapunov? Multi-Agent Reinforcement Learning under Coupled Finite-Horizon Constraints
Radio resource management in dense networks is subject to budgets that must hold on every realized trajectory, carried by different agent groups: an energy budget at each base station (BS) and a handover budget at each user, coupled through the association decisions that consume both. Constrained multi-agent reinforcement learning (MARL) handles such budgets using Lagrangian multipliers updated across episodes or Lyapunov virtual queues updated per slot, but their relative behavior under coupled two-sided constraints remains unclear. We formulate this problem and introduce a windowed proportional-fairness objective that captures the temporal distribution of service beyond the classical horizon-average formulation. We characterize the distinct budget-regulation timescales of the two mechanisms through complementary theoretical bounds and identify queue-price buildup as a key mechanism governing finite-horizon budget pacing. Using a common heterogeneous MARL architecture, we compare three utility surrogates and both constraint-handling mechanisms. The results show that utility design primarily determines throughput and fairness, whereas constraint-price dynamics govern training efficiency and budget pacing. Under the default setting, the queue-based PF policy reaches near-final performance after a single training episode, but its unscaled penalty can lead to delayed pacing under tight budgets. Guided by the analysis, we show that queue-price calibration reduces the empirical drift constant by an order of magnitude and substantially narrows the performance gap to Lagrangian methods, which adapt constraint prices across episodes at higher training cost. These findings provide practical design guidelines for constrained MARL.
查看此版本解读 →查看此版本原文 ↗