HorizonBench: Long-Horizon Personalization with Evolving Preferences
Shuyue Stella Li, Bhargavi Paranjape, Kerem Oktar, Zhongyao Ma, Gelin Zhou, Lin Guan, Na Zhang, Sem Park 等 12 位作者
原文摘要
Personal assistants now talk with the same user for months, and users' preferences change as their lives change; an assistant that keeps serving a preference the user has outgrown fails that user. Real conversations do not record when or why a preference changed, and existing benchmarks either hold preferences fixed or have the user state the change, so they cannot tell whether a model forgot what the user said or failed to update its belief about the user. We introduce HorizonBench and its data generator, which samples how a user's preferences change and generates six months of conversation from that sequence; each change follows a life event that the user mentions without stating the new preference. A human longitudinal preference dataset that we collect and release, 83,024 ratings from 160 people over 13 weeks, sets how often and how much life events change preferences. HorizonBench contains 4,245 questions from 360 users, and when a preference has changed, its earlier value is one of the answer options. All 25 frontier models we evaluate choose the outdated preference more often than evenly spread errors would predict, and the best reaches 52.8% accuracy. Two of the newest models score no higher when given only the relevant conversations than with the full history, yet one added sentence stating the change raises their accuracy by 11 to 22 points, and retrieval and memory methods do not recover the lost accuracy. Remembering what a user said and knowing what the user prefers now are therefore separate capabilities, and HorizonBench, its generator, and the human dataset support research on both.
cs.CL · cs.AI