AI models' growing capacity for complex reasoning has made them attractive tools for forecasting, or the systematic prediction of the future based on past and present data. However, should AI forecasting systems be adopted, human builders will shoulder the work of building and evaluating them. In this paper, we explore what this work looks like with a qualitative interview study with 15 builders of forecasting bots, sampled from among successful participants in a recent AI forecasting competition. Our analysis surfaces the challenges around (1) tuning a bot to acquire factual research, (2) implementing coherent reasoning, and (3) understanding how well one's bot actually does. During their work, builders frequently introduced complexity that did not pay off, and tested in ways that were opportunistic and piecemeal. Our detailed account of their challenges helps clarify what problems should be targeted in upcoming innovations in process, tools, and human-AI teaming for forecasting.
Hita Kambhamettu, Alyssa Hwang, Philippe Laban, Andrew Head
原文摘要
AI answer engines are a relatively new kind of information search tool: rather than returning a ranked list of documents, they generate an answer to a search question with inline citations to sources. But reading the cited sources is costly, and citation links themselves offer little guidance about what evidence they contain. We present attribution gradients, a technique to boost the informativeness of citations by consolidating scent and information prey in place. Its first feature is bringing evidence counts, supporting/contradictory excerpts, links to sources, and contextual explanations into one place. Its second feature is the ability to unravel second-degree citations in place. In a lab study, we demonstrate usage of the full gradient in a critical reading task and its support for deep engagement that increased the depth of what readers took away from the sources versus a standard citation and document QA design.