← 研踪 PaperTrailarXiv:2605.27980 · v2
Periodic and Harmonic Periodic RoPE for Length Extrapolation in LLMs
首次提交 2026-05-27 13:18
版本更新 2026-11-19 19:59
雷达观测 2026-11-20 16:53
摘要
The ability to process long contexts is crucial for large language models (LLMs) to perform long-horizon tasks. Rotary position embedding (RoPE) gives every token a new rotation angle, so a model eventually meets angles it never saw in training, i.e., position exhaustion. We propose Periodic RoPE (P-RoPE), a positional encoding mechanism that operates in conjunction with sliding window attention. It repeats the first sliding-window-size angles of the RoPE table, so the model needs only distinct angles. P-RoPE works but introduces a wrap error at the window boundary. We further propose Harmonic Periodic RoPE (HP-RoPE), which removes the error by choosing frequencies that complete whole turns in tokens. We compare models that share the same architecture, data, and schedule and differ only in the window-layer position encoding: standard RoPE, P-RoPE, and HP-RoPE. The results show that P-RoPE and HP-RoPE keep perplexity nearly flat from the training length to k, while RoPE, NTK, and YaRN rise. In a format-free Passkey test at k, they remain at --, against -- for those RoPE interpolations. The code is available at \href{https://github.com/Cominder/miniwin}{https://github.com/Cominder/miniwin}.
作者备注:5 pages
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-19 19:59 · 最新观测版本
Periodic and Harmonic Periodic RoPE for Length Extrapolation in LLMs
The ability to process long contexts is crucial for large language models (LLMs) to perform long-horizon tasks. Rotary position embedding (RoPE) gives every token a new rotation angle, so a model eventually meets angles it never saw in training, i.e., position exhaustion. We propose Periodic RoPE (P-RoPE), a positional encoding mechanism that operates in conjunction with sliding window attention. It repeats the first sliding-window-size angles of the RoPE table, so the model needs only distinct angles. P-RoPE works but introduces a wrap error at the window boundary. We further propose Harmonic Periodic RoPE (HP-RoPE), which removes the error by choosing frequencies that complete whole turns in tokens. We compare models that share the same architecture, data, and schedule and differ only in the window-layer position encoding: standard RoPE, P-RoPE, and HP-RoPE. The results show that P-RoPE and HP-RoPE keep perplexity nearly flat from the training length to k, while RoPE, NTK, and YaRN rise. In a format-free Passkey test at k, they remain at --, against -- for those RoPE interpolations. The code is available at \href{https://github.com/Cominder/miniwin}{https://github.com/Cominder/miniwin}.
查看此版本解读 →查看此版本原文 ↗