← 研踪 PaperTrailarXiv:2603.05432 · v2
Ensembling Language Models with Sequential Monte Carlo
首次提交 2026-03-06 01:54
版本更新 2026-11-07 16:05
雷达观测 2026-11-10 11:14
摘要
Combining the predictions of multiple models is one of the most effective strategies in machine learning. Thus, it is only natural to investigate the ensembling of language models (LMs). However, because LMs define distributions over strings, a proper ensemble requires aggregating predictions at the string level, which requires a computation that is generally intractable. In this work, we cast LM ensembling as an inference problem and propose the deployment of sequential Monte Carlo (SMC), an appropriate approximate inference scheme. Concretely, we introduce a unified framework for composing multiple LMs into ensemble distributions parameterized by a broad family of aggregation functions. To sample from these distributions, we introduce an SMC algorithm that operates in a shared byte space, enabling ensembles of models with mismatching vocabularies and consistent sampling in the limit. We evaluate a range of ensembles across prompt and model combinations for various text generation tasks, finding that consensus-seeking strategies such as the product consistently outperform probability averaging, and that better posterior approximations can yield better ensemble performance.
作者备注：NeurIPS 2026 camera-ready. 34 pages, 13 figures, 6 tables
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-07 16:05 · 最新观测版本
Ensembling Language Models with Sequential Monte Carlo
Combining the predictions of multiple models is one of the most effective strategies in machine learning. Thus, it is only natural to investigate the ensembling of language models (LMs). However, because LMs define distributions over strings, a proper ensemble requires aggregating predictions at the string level, which requires a computation that is generally intractable. In this work, we cast LM ensembling as an inference problem and propose the deployment of sequential Monte Carlo (SMC), an appropriate approximate inference scheme. Concretely, we introduce a unified framework for composing multiple LMs into ensemble distributions parameterized by a broad family of aggregation functions. To sample from these distributions, we introduce an SMC algorithm that operates in a shared byte space, enabling ensembles of models with mismatching vocabularies and consistent sampling in the limit. We evaluate a range of ensembles across prompt and model combinations for various text generation tasks, finding that consensus-seeking strategies such as the product consistently outperform probability averaging, and that better posterior approximations can yield better ensemble performance.
查看此版本解读 →查看此版本原文 ↗