IndicDBBench - Benchmarking Multilingual Text-to-SQL Capabilities in Indian Languages
Aviral Dawar, Roshan Karanth, Vikram Goyal, Rakhee Kulshrestha, Sumanta Pasari, Dhruv Kumar
While Large Language Models (LLMs) have significantly advanced Text-to-SQL performance, existing benchmarks largely focus on Western contexts leaving a gap in real-world, non-Western applications. We introduce IndicDBBench, a comprehensive multilingual Text-to-SQL benchmark designed to evaluate cross-lingual semantic parsing across diverse Indic language families. The underlying schemas are derived from real-world open-data platforms (NDAP and IDP), ensuring realistic structural complexity. IndicDBBench comprises 20 databases spanning 237 tables. To construct complex relational structures from denormalized government data, we employ a three-agent pipeline (Architect, Auditor, Refiner), achieving high relational density (11.85 tables per database) and join depths of up to six. We generate 15,612 tasks spanning seven languages: English and six Indic languages, ensuring that queries are grounded in real database values and require multi-table reasoning. We evaluate state-of-the-art models, including DeepSeek V3.2, MiniMax M2.7, Llama 3.3, and Qwen3, across seven linguistic variants. Our results reveal a 9.0-point drop in execution accuracy from English to Indic languages, exposing an "Indic Gap" that, for the two models tested, persists even when gold tables and literal values are supplied, since this grounding raises accuracy in every language by similar amounts. The dataset and code are publicly available at https://anonymous.4open.science/r/multilingualText2Sql-Indic--DDCC/
尚无 AI 解读，展示原文摘要