← 研踪 PaperTrailarXiv:2610.17205 · v2
Releasing the Inner Jev: A Simple Recipe for Open Decision Models
首次提交 2026-10-13 01:59
版本更新 2026-11-19 20:16
雷达观测 2026-11-20 16:53
摘要
LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026). Given a typed input state, a question, and a set of options, they return a calibrated probability for each option in a single forward pass, without text generation. We show that open instruction-tuned LLMs can achieve performance comparable to Jev without any additional training, by reading out answers from letters (A, B, C, etc.) in the model's output logits at the last token position. Next, we propose a simple supervised LoRA fine-tuning recipe that can improve their performance further. We evaluate four instruction-tuned LLMs: Gemma 4 31B, Gemma 4 12B, Qwen3.6-27B, and Qwen3.5-4B. Furthermore, we introduce OpenDecisions, a novel 2,352-question validation dataset designed to reflect the task composition of JevBench (JevBench maintainers, 2026) and which computes a Capability score based on accuracy and calibration. Without fine-tuning, Gemma 4 31B scores 79.6 on OpenDecisions, compared to Jev's 80.1. After fine-tuning, it scores 89.0, outperforming Jev by 8.9 Capability points and on par with the strongest open decision model we evaluated. The same recipe improves the three other models by 7.8 to 15.7 points. On a suite of public decision datasets, our models achieve accuracy and calibration comparable to Jev. Our results show that open LLMs already possess System One capabilities, and with simple fine-tuning they outperform Jev on a wide range of decision tasks. We release our models, training and serving code, OpenDecisions, and DecisionBank, our training dataset consisting of 261,561 decisions.
作者备注:27 pages, 8 figures
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-19 20:16 · 最新观测版本
Releasing the Inner Jev: A Simple Recipe for Open Decision Models
LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026). Given a typed input state, a question, and a set of options, they return a calibrated probability for each option in a single forward pass, without text generation. We show that open instruction-tuned LLMs can achieve performance comparable to Jev without any additional training, by reading out answers from letters (A, B, C, etc.) in the model's output logits at the last token position. Next, we propose a simple supervised LoRA fine-tuning recipe that can improve their performance further. We evaluate four instruction-tuned LLMs: Gemma 4 31B, Gemma 4 12B, Qwen3.6-27B, and Qwen3.5-4B. Furthermore, we introduce OpenDecisions, a novel 2,352-question validation dataset designed to reflect the task composition of JevBench (JevBench maintainers, 2026) and which computes a Capability score based on accuracy and calibration. Without fine-tuning, Gemma 4 31B scores 79.6 on OpenDecisions, compared to Jev's 80.1. After fine-tuning, it scores 89.0, outperforming Jev by 8.9 Capability points and on par with the strongest open decision model we evaluated. The same recipe improves the three other models by 7.8 to 15.7 points. On a suite of public decision datasets, our models achieve accuracy and calibration comparable to Jev. Our results show that open LLMs already possess System One capabilities, and with simple fine-tuning they outperform Jev on a wide range of decision tasks. We release our models, training and serving code, OpenDecisions, and DecisionBank, our training dataset consisting of 261,561 decisions.
查看此版本解读 →查看此版本原文 ↗