arXiv:2607.09800 · v5
Predicting Lost Updates Is Not Enough to Choose a Rounding Policy
摘要
In low-precision training, a weight can round back to its previous value after a small update. We test whether predicting how often this happens helps choose a rounding rule. In Pythia-410M continuations from a common checkpoint, a higher-precision pilot accurately predicted the fraction of matrix weights left unchanged at each of the first 64 updates. Across three schedules with optimizer state retained or reset, nearest rounding gave lower validation loss at 256 updates than stochastic rounding or retaining 32-bit weights. The same ordering held at 512 updates with retained state. Accurately forecasting the unchanged fraction did not identify a benefit from switching policies in these runs.
作者备注:13 pages, 3 figures, 8 tables. This version adds matched Pythia-410M continuations under three weight-update policies, a paired optimizer-state comparison, and a 512-update extension. Ancillary files provide numerical data and replay scripts for all 36 runs