Periodic and Harmonic Periodic RoPE for Length Extrapolation in LLMs
Simin Huo, Ning Li
原文摘要
The ability to process long contexts is crucial for large language models (LLMs) to perform long-horizon tasks. Rotary position embedding (RoPE) gives every token a new rotation angle, so a model eventually meets angles it never saw in training, i.e., position exhaustion. We propose Periodic RoPE (P-RoPE), a positional encoding mechanism that operates in conjunction with sliding window attention. It repeats the first sliding-window-size \(W\) angles of the RoPE table, so the model needs only \(W\) distinct angles. P-RoPE works but introduces a wrap error at the window boundary. We further propose Harmonic Periodic RoPE (HP-RoPE), which removes the error by choosing frequencies that complete whole turns in \(W\) tokens. We compare models that share the same architecture, data, and schedule and differ only in the window-layer position encoding: standard RoPE, P-RoPE, and HP-RoPE. The results show that P-RoPE and HP-RoPE keep perplexity nearly flat from the training length \(768\) to \(16\)k, while RoPE, NTK, and YaRN rise. In a format-free Passkey test at \(16\)k, they remain at \(99\)--\(100\%\), against \(11\)--\(37\%\) for those RoPE interpolations. The code is available at \href{https://github.com/Cominder/miniwin}{https://github.com/Cominder/miniwin}.
cs.CL · cs.AI