跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.12878 · v1

Adaptive Quantization Preserves Fast Rates for TD Learning over Rate-Limited Channels

自适应量化在速率受限信道上保持TD学习的快速收敛率

首次提交 2026-11-10 02:58
版本更新 2026-11-10 02:58
雷达观测 2026-11-10 15:22
arXiv ↗PDF ↗

摘要

Motivated by reinforcement learning subject to communication constraints, we study a policy evaluation problem with linear function approximation over a rate-limited channel. In our setup, an agent interacting with an unknown environment (modeled as a Markov Decision Process) transmits quantized estimates of a temporal-difference (TD) learning update direction to a server over a finite-capacity channel. Our goal is to investigate to what extent one can preserve the statistical convergence rates achievable when the channel has infinite capacity. To that end, we propose a new algorithm called Batched Adaptively Quantized TD (BAQ-TD), which uses batching to reduce the variance of the transmitted information, and adaptive quantization to encode innovations. We prove a high-probability finite-time error bound for BAQ-TD, and show that, above a certain finite channel capacity, BAQ-TD preserves the convergence rate of unquantized TD up to logarithmic factors. Specifically, for a dd-dimensional value function parameter, we establish that transmitting a total of O(dlog⁡(T))\mathcal{O}(d \log(T)) bits over a horizon of length TT suffices to achieve the order-optimal rate of O~(1/T)\tilde{\mathcal{O}}(1/T), marking an exponential improvement in communication-complexity over prior work.

AI 论文解读

基于摘要 · 非全文审阅

作者提出BAQ-TD算法,在有限容量信道上通过批处理与自适应量化传输TD更新方向,并证明在超过某有限信道容量后可保持未量化TD的收敛速率(至多对数因子)。

正在读取账号与研究记录…

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-10 02:58 · 最新观测版本

Adaptive Quantization Preserves Fast Rates for TD Learning over Rate-Limited Channels

Motivated by reinforcement learning subject to communication constraints, we study a policy evaluation problem with linear function approximation over a rate-limited channel. In our setup, an agent interacting with an unknown environment (modeled as a Markov Decision Process) transmits quantized estimates of a temporal-difference (TD) learning update direction to a server over a finite-capacity channel. Our goal is to investigate to what extent one can preserve the statistical convergence rates achievable when the channel has infinite capacity. To that end, we propose a new algorithm called Batched Adaptively Quantized TD (BAQ-TD), which uses batching to reduce the variance of the transmitted information, and adaptive quantization to encode innovations. We prove a high-probability finite-time error bound for BAQ-TD, and show that, above a certain finite channel capacity, BAQ-TD preserves the convergence rate of unquantized TD up to logarithmic factors. Specifically, for a dd-dimensional value function parameter, we establish that transmitting a total of O(dlog⁡(T))\mathcal{O}(d \log(T)) bits over a horizon of length TT suffices to achieve the order-optimal rate of O~(1/T)\tilde{\mathcal{O}}(1/T), marking an exponential improvement in communication-complexity over prior work.

查看此版本解读 →查看此版本原文 ↗