FlinkEval：用于评估流式 SQL 自然语言接口的合成语料库
FlinkEval: A Synthetic Corpus for Evaluating Natural Language Interfaces to Streaming SQL
Alfio Massimiliano Gliozzo, Nandana Mihindukulasooriya, Gaetano Rossiello, Erick Lee, Lisa Amini
AI 速读 基于摘要
作者提出一个基于执行验证的合成基准 FlinkEval，用 3,903 个问题–SQL 对评估面向流式 SQL 的自然语言接口。
cs.DB
查看原文摘要
Existing natural-language-to-SQL benchmarks primarily evaluate queries over bounded relational data, leaving streaming semantics underexplored. We introduce FlinkEval, an execution-grounded synthetic benchmark comprising 3,903 question--SQL pairs across 45 industry domains and 45 semantic labels. We create diverse schemas and domain-plausible, temporally consistent data through iterative generation and validation. SQL patterns collected from the human-written NL2SQL pairs and vendor documentation drive quota-balanced generation, and execution with a repair loop on Confluent Cloud Apache Flink produces reference queries covering windowing, temporal joins, pattern matching, nested types, and data manipulation. The corpus includes paired schema-explicit and business-language requests, with annotations for analysis by query pattern, complexity, and execution mode. A seven-level Question Specification Ladder examines semantics-preserving reformulation and underspecified requests augmented with oracle-derived clues. Baseline evaluations combine execution outcomes with adjudicated LLM judgments, revealing challenges in pattern composition and request specification.