Radio resource management in dense networks is subject to budgets that must hold on every realized trajectory, carried by different agent groups: an energy budget at each base station (BS) and a handover budget at each user, coupled through the association decisions that consume both. Constrained multi-agent reinforcement learning (MARL) handles such budgets using Lagrangian multipliers updated across episodes or Lyapunov virtual queues updated per slot, but their relative behavior under coupled two-sided constraints remains unclear. We formulate this problem and introduce a windowed proportional-fairness objective that captures the temporal distribution of service beyond the classical horizon-average formulation. We characterize the distinct budget-regulation timescales of the two mechanisms through complementary theoretical bounds and identify queue-price buildup as a key mechanism governing finite-horizon budget pacing. Using a common heterogeneous MARL architecture, we compare three utility surrogates and both constraint-handling mechanisms. The results show that utility design primarily determines throughput and fairness, whereas constraint-price dynamics govern training efficiency and budget pacing. Under the default setting, the queue-based PF policy reaches near-final performance after a single training episode, but its unscaled penalty can lead to delayed pacing under tight budgets. Guided by the analysis, we show that queue-price calibration reduces the empirical drift constant by an order of magnitude and substantially narrows the performance gap to Lagrangian methods, which adapt constraint prices across episodes at higher training cost. These findings provide practical design guidelines for constrained MARL.
Fixed control policies in Open RAN can require retraining when service requirements, traffic loads, or interacting applications change. We propose ATTUNE, a state-augmented learning method for an energy saving application that coexists with a learned mobility load balancing application under a service constraint. ATTUNE conditions the energy saving policy on a Lagrange multiplier and updates this scalar during operation using measured service degradation, while keeping both applications' network parameters fixed. To train a single policy across service requirements, ATTUNE samples target service levels uniformly and maps them to multipliers through an adaptive table. This procedure accounts for policy-dependent multiplier responses and improves energy savings compared with training over a widened fixed multiplier range. We characterize the induced distribution of service levels and the convergence and fluctuations of the online update. An exact finite-window identity establishes convergence of average measured degradation to the target when the projection corrections vanish on average. Simulations show that a single ATTUNE policy achieves energy savings comparable to policies trained separately for each service requirement. It adapts to changes in the target, load, and partner application without retraining, increases energy savings at light load, and maintains target tracking for separately trained application pairs. No statistically significant increase in cell switching is detected in the evaluated comparisons.