Ellen Su, Andres Potapczynski, Shikai Qiu, Edward Hughes, Andrew Gordon Wilson
原文摘要
Modern systems are increasingly expected to transfer across tasks not specified during training. What data facilitates generalization to unanticipated settings? One hypothesis is that data with more structural information could contain shared circuits and subprograms that may be recycled in a wider array of downstream settings. Epiplexity, a recently proposed measure of the structural information a compute-bounded learner can extract from data, offers a mechanism to reason about this relationship. As existing estimators for epiplexity only operate on data already seen during training, we show how to operationalize epiplexity as an online training signal by greedily maximizing its stepwise gain in two settings. For data selection, we fit scaling laws to the training loss curves of natural data domains to predict the expected epiplexity gain as a function of training tokens, and use this signal to adaptively determine the sampling weights over domains during training. For synthetic data generation, where no domain structure exists, we reward a generator with the change in epiplexity of a learner's synthetic training data and use REINFORCE policy gradients to guide the generator toward an epiplexity-maximizing distribution. In both cases, we find higher epiplexity predicts improved downstream performance on zero-shot and fine-tuning tasks, supporting the hypothesis that data rich in structural information yield representations that better transfer across domains.
Ethan Baron, Shmuel Berman, Anirudh Satheesh, Akshay Vegesna, Samip Dahal, Andrew Gordon Wilson
原文摘要
As growing compute budgets outpace the growth of high-quality training data, large language models must be trained with both compute and data constraints in mind. Unfortunately, existing prescriptions for data-constrained scaling suffer from two important limitations. First, past works hold key hyperparameters such as weight decay fixed with scale, despite the fact that optimal hyperparameters vary wildly with model size and number of epochs. Second, past works neglect ensembling, a promising axis for scaling under data constraints. As a surprising consequence, prior prescriptions can lead to loss that increases with compute budget. We address both limitations by fitting power laws that prescribe how to transfer hyperparameters with scale and scaling laws that prescribe how to allocate compute with ensembling. Training on 100M tokens from FineWeb, our recipe reduces validation loss by over 0.15 nats compared to prior recipes, a 14% decrease in perplexity, and achieves monotonic improvements with compute through ensembling. These gains are achieved using prescriptions fit to smaller runs and validated at up to 12$\times$ larger compute budgets.