Meta-Scaling Laws: On the Invariance and Generality of Neural Scaling
Xing Han, Liu Ziyin, Suchi Saria, Paul Pu Liang
原文摘要
Neural scaling laws establish a predictable relationship between model performance and data or compute, offering crucial guidance for resource allocation in new domains and tasks. Yet such laws are most needed precisely where they are hardest to obtain: fitting one for a new model and task pair demands expensive sweeps that typically exhaust the very compute budget the law is meant to economize. We study meta-scaling laws: relations that characterize when neural scaling behavior is preserved and predict how it changes under data transformations, enabling forecasts across domains when the shift can be approximated by a compatible transformation. We address this by identifying the right invariants: scaling laws are preserved under bijective (information-preserving) transformations of the data and modified in predictable, information-theoretically grounded ways under non-bijective transformations that lower its information resolution $ρ$: a single axis along which a law fit in one domain can be transported to another. We validate this across language, vision, and speech, and demonstrate two cross-domain applications: predicting scaling for language models trained on electronic health records from laws fit on general text, and predicting time-series classification scaling under varying levels of noise injection, recovering the data-scaling exponents to within $3\%$ error.
cs.LG