arXiv:2609.24965 · v5

Jev for Scientific Decisions: Evaluating Semantic Choices and Their Consequences

首次提交 2026-09-22 01:51

版本更新 2026-11-02 13:57

雷达观测 2026-11-04 23:42

摘要

Scientific workflows often require choosing among known relations before a deterministic calculation can proceed. Whether observations share a culture, treatment or reference standard can change the scientific meaning of the resulting count or comparison. Once the evidence is prepared and the calculation is written as code, this choice reduces to one semantic judgment among listed candidates. We evaluate Jev on this step, using a harness that follows its documented guidance. The study compares twelve model configurations on twenty Choices in ten scientific cases built from published sources, each repeated five times. We measure semantic selections, downstream outputs and final claim labels separately. Jev tied with five other configurations at complete semantic correctness and achieved the lowest observed median latency and known mean cost among successful responses. On one question, whether samples shared a culture, three comparison models made seven wrong selections that changed downstream counts while preserving the correct final label. These results support using Jev to select relations when the evidence and candidates are prepared in advance, and show that evaluating it requires checking the relations and quantities that a workflow will reuse.

作者备注：8 pages, 1 figure, 5 tables. Code available at https://github.com/bydeng01/scientific-decision-eval