← 研踪 PaperTrailarXiv:2610.32955 · v2
Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox
首次提交 2026-10-24 01:49
版本更新 2026-11-02 15:45
雷达观测 2026-11-04 23:42
摘要
Personal email agents decide, for every incoming message, its type and urgency and whether to draft a reply, create a task or ignore it. In production these decisions are delegated to a generative LLM or to rules, and their error rate is rarely measured. We describe Foyer, a self-hosted personal inbox agent, and a measurement protocol (stratified sampling, a frozen snapshot of the model input, thread-grouped splits, secret redaction, and human labels collected in the agent's own review loop) that produces ground truth before any training. On 231 real emails in Italian and English we compare the production rules, a TF-IDF + logistic-regression baseline, five typed decision models, local and hosted (Tev1, Nimble, Jev), and three hosted generative LLMs. Four results. First, the production rules for message type score 28% against a 58% majority class. Second, with about two hundred labels a logistic regression has the highest accuracy on every field (type 0.79, urgency 0.74, needs-reply 0.83, is-noise 0.90) at a latency three orders of magnitude lower; the best zero-shot system reaches 0.74 on type. Third, the sender's relationship to the owner is recovered from other emails of the same contact with 98.6% accuracy on the 63% of emails whose sender has been seen before. Fourth, per-email accuracy is not the product metric: composed into actions, the baseline ignores 18% of the emails that needed a reply or a task, against 4% for the best generative model; a cost-chosen threshold cuts that rate to 6% and makes the classifier the cheapest system at every price on the unbiased corpus. A third, adversarially sampled round (722 emails in total) with eleven pre-registered hypotheses confirms seven and refutes four. Two blind annotators agree with the reference at kappa = 0.81 and 0.86 on type. We release the protocol, the harness and the pre-registration; the emails stay private.
作者备注：v2: adds Jev, cost curves over error-cost ratios, composition policies, two blind annotators. 10 pages
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-02 15:45 · 最新观测版本
Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox
Personal email agents decide, for every incoming message, its type and urgency and whether to draft a reply, create a task or ignore it. In production these decisions are delegated to a generative LLM or to rules, and their error rate is rarely measured. We describe Foyer, a self-hosted personal inbox agent, and a measurement protocol (stratified sampling, a frozen snapshot of the model input, thread-grouped splits, secret redaction, and human labels collected in the agent's own review loop) that produces ground truth before any training. On 231 real emails in Italian and English we compare the production rules, a TF-IDF + logistic-regression baseline, five typed decision models, local and hosted (Tev1, Nimble, Jev), and three hosted generative LLMs. Four results. First, the production rules for message type score 28% against a 58% majority class. Second, with about two hundred labels a logistic regression has the highest accuracy on every field (type 0.79, urgency 0.74, needs-reply 0.83, is-noise 0.90) at a latency three orders of magnitude lower; the best zero-shot system reaches 0.74 on type. Third, the sender's relationship to the owner is recovered from other emails of the same contact with 98.6% accuracy on the 63% of emails whose sender has been seen before. Fourth, per-email accuracy is not the product metric: composed into actions, the baseline ignores 18% of the emails that needed a reply or a task, against 4% for the best generative model; a cost-chosen threshold cuts that rate to 6% and makes the classifier the cheapest system at every price on the unbiased corpus. A third, adversarially sampled round (722 emails in total) with eleven pre-registered hypotheses confirms seven and refutes four. Two blind annotators agree with the reference at kappa = 0.81 and 0.86 on type. We release the protocol, the harness and the pre-registration; the emails stay private.
查看此版本解读 →查看此版本原文 ↗