跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2610.19608 · v2

Forecasting Brand Mentions in LLM Answers: Frozen Predictions Evaluated from Hours to Weeks

首次提交 2026-10-14 13:30
版本更新 2026-11-16 04:44
雷达观测 2026-11-17 17:12
arXiv ↗PDF ↗

摘要

Companies and agencies now monitor which brands large language models (LLMs) name when buyers ask for recommendations. Answers vary between runs, so a single check can mislead and repeated checks cost money. We extend an earlier 20-day replication to ask how well past brand mentions predict later ones over seven weeks. We froze probabilistic forecasts before each of four prospective collection waves of Gemini 3.1 Pro answers to the same 54 buyer questions: nine question families in English, Estonian and Finnish, each for two markets, with four of 12 tracked brands per question. The waves were collected between 1.5 hours and 52 days after the history used by the earliest forecast, and each scored 648 brand and answer labels from 162 answers. A smoothed per-question history achieved a Brier score of 0.0599 at 1.5 hours, against 0.1131 for a per-brand rate and 0.25 for a constant 0.5. A pooled rate clipped to 0.05--0.95, named as the challenger before its first test, scored 0.0259 after 8 hours, 0.0269 after 30 days and 0.0328 after 52 days. Of the brand and question cells named in all or none of the earlier answers, 94.5% to 97.5% kept that pattern in each later wave, and none switched from always to never or back. With a search instruction, 100 of 162 answers returned search activity, compared with 1 of 540 earlier Pro answers; mentions stayed predictable (Brier 0.0563), and Pro history predicted a smaller model's answers less well (0.0787). Human coding agreed with 298/300 sampled automatic labels; two coders agreed on 99/100 shared items (kappa = 0.98). The results cover one provider family, a fixed candidate set and lexical mentions. They support per-question history as the baseline that any forecasting or monitoring method for LLM brand mentions should beat.

作者备注:13 pages, 5 tables, 1 figure. Expanded version retaining the original 20-day replication and adding follow-ups through 52 days, search and model-transfer comparisons, and independent human validation. Clarifies shared observations, registration scope and baseline histories. Supporting data: https://doi.org/10.5281/zenodo.24249785; original D1 data: https://doi.org/10.5281/zenodo.23351313

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-16 04:44 · 最新观测版本

Forecasting Brand Mentions in LLM Answers: Frozen Predictions Evaluated from Hours to Weeks

Companies and agencies now monitor which brands large language models (LLMs) name when buyers ask for recommendations. Answers vary between runs, so a single check can mislead and repeated checks cost money. We extend an earlier 20-day replication to ask how well past brand mentions predict later ones over seven weeks. We froze probabilistic forecasts before each of four prospective collection waves of Gemini 3.1 Pro answers to the same 54 buyer questions: nine question families in English, Estonian and Finnish, each for two markets, with four of 12 tracked brands per question. The waves were collected between 1.5 hours and 52 days after the history used by the earliest forecast, and each scored 648 brand and answer labels from 162 answers. A smoothed per-question history achieved a Brier score of 0.0599 at 1.5 hours, against 0.1131 for a per-brand rate and 0.25 for a constant 0.5. A pooled rate clipped to 0.05--0.95, named as the challenger before its first test, scored 0.0259 after 8 hours, 0.0269 after 30 days and 0.0328 after 52 days. Of the brand and question cells named in all or none of the earlier answers, 94.5% to 97.5% kept that pattern in each later wave, and none switched from always to never or back. With a search instruction, 100 of 162 answers returned search activity, compared with 1 of 540 earlier Pro answers; mentions stayed predictable (Brier 0.0563), and Pro history predicted a smaller model's answers less well (0.0787). Human coding agreed with 298/300 sampled automatic labels; two coders agreed on 99/100 shared items (kappa = 0.98). The results cover one provider family, a fixed candidate set and lexical mentions. They support per-question history as the baseline that any forecasting or monitoring method for LLM brand mentions should beat.

查看此版本解读 →查看此版本原文 ↗