arXiv:2610.11482 · v2
Evaluating Local Language Model Agents for Data Engineering: An Empirical Software Engineering Study of Mobility Workflows
摘要
Context: Large language model (LLM) agents are increasingly used as software and data-engineering assistants, yet evidence about locally deployable open-weight agents remains limited. Existing evaluations often emphasise textual responses or isolated code rather than complete engineering artifacts. Objectives: We evaluate whether local LLM agents can produce correct and reliable data-engineering artifacts, isolate the contribution of a closed-loop workspace and of input-structure knowledge, and examine trade-offs in runtime, cost, and failure behaviour. Methods: We introduce a benchmark of fifteen mobility data-engineering tasks with deterministic checkers for the scripts, tables, structured files, figures, and reports they produce. Sixteen configurations from six model families are evaluated in three conditions (chat-only, chat-schema, agent-with-tools) with five repetitions per model, condition, and task, yielding 3,600 scored attempts on a consumer-grade GPU. Results: A 4-billion-parameter model at 4-bit reaches 77.3%, a 9-billion-parameter model 80.0% within about 6.5 GB, and a 12-billion-parameter model from another family 81.3%, against 98.7% for the highest-scoring configuration, which solves 14 of 15 tasks in all five repetitions. Above two billion parameters the workspace loop raises pass rates by 24.0 to 65.3 points over uninformed one-shot generation, and its contribution exceeds that of the input-structure summary in eleven of fifteen tasks. Conclusion: Open-weight models that fit on one consumer GPU support a meaningful share of data-engineering work once given a workspace loop, although capability still separates usable from unusable configurations and high-stakes outputs need validation. Being mobility-specific and checker-based, the benchmark characterizes the evaluated configurations, not all data-engineering practice.
作者备注：34 pages, 4 figures, 10 tables. Revised version: new chat-schema condition, task-clustered statistics and pass^k reliability, five additional model families, title and abstract updated. Under review. Replication package (v1.1.2): https://doi.org/10.5281/zenodo.24115839