跳到正文
研踪PaperTrail

论文库

检索计算机领域论文,按研究方向、作者或关键词筛选。

部分采集异常
正在读取账号与研究记录…

全站更新概况

+1292新增论文
+1016其中 AI 相关
+166已有论文修订
+59新增 AI 导读

arXiv 最近完成扫描:2026-11-18 04:28 · 全站最近入库:2026-11-18 11:33各信源进度 →

统计口径与数据新鲜度

统计 2026-11-17 12:24 — 2026-11-18 12:24(北京时间) · 点击数字查看对应简报;AI 相关包含在新增论文内。

最近完成检查:2026-11-18 04:28最近发现新增版本:2026-11-18 11:33库内论文最新更新时间:2026-11-17 20:55

每分钟刷新已保存的数据。空检查不代表采集异常;新收录可能包含历史论文。AI 导读按新增版本计数。

25,964 已收录论文21,631 AI 相关7,801 有修订112 HF 社区发现

浏览论文库

时间范围跟随排序:版本更新、本站收录或首次发表。最新收录可能包含历史论文。部分信源异常,查看状态 →

找到 230 篇论文 · 按版本更新时间排序上轮扫描起点 2026-11-18 04:27(北京时间)
新论文 · v1arXiv:2611.15675

Optimal-Time Independent Subset Sampling over Acyclic Joins

Xuyang Liu, Zijian Zhang, Zhen Li

原文摘要

Join results can greatly exceed their inputs even when applications need only a small random subset. We study sampling that includes each result independently with a prescribed probability given by the product of its input-tuple weights (PRODUCT). For fixed full acyclic joins with static weights, we achieve input-linear preprocessing and index space together with output-linear expected sampling work, eliminating logarithmic factors from previous bounds. A shared join-tree index supports weighted draws over the entire join and locates high-probability results. Direct Bernoulli trials handle high-probability results, allowing Poisson thinning to generate the remaining independent inclusion decisions with bounded overhead. Our exact sampler uses expected $O(N)$ preprocessing, $O(N)$ index space, and expected $O(1+mμ)$ time per call including output construction. Here $N$ is the input size, $m$ counts relation occurrences, and $μ$ is the expected sample size. The guarantees hold on a real random-access machine with unit-cost exact arithmetic and logarithms, continuous uniform randomness, and expected-time hashing. The time bounds are optimal, matching input reading and explicit output writing. Our floating-point prototype has lower sampling latency than the fastest applicable baseline in 17 of 18 PRODUCT settings on three queries over the STATS benchmark. Construction savings on large joins can offset indexed materialization's faster sampling over a limited number of calls. On a real-data path at $μ=100$, our sampler achieves a $20.5\times$ speedup over indexed materialization for construction plus 300 calls, measured after common data preparation and loading.

cs.DB · cs.DS

版本更新 · v3arXiv:2602.02025

Hippasus: Accurate and Cost-Efficient Feature Augmentation over Relational Tables

Serafeim Papadias, Kostas Patroumpas, Dimitrios Skoutas

原文摘要

ML models critically depend on feature quality, yet in real-world settings, useful features are often distributed across multiple relational tables rather than a single dataset. Feature augmentation addresses this problem by automatically discovering and joining additional tables to enrich a base table with predictive features. However, scaling feature augmentation to complex schemas with many tables and multi-hop relationships is challenging. It requires exploring a large space of join paths, executing costly joins, and selecting useful features from noisy results. Existing approaches suffer from either limited effectiveness or efficiency. Restricting exploration to simple joins limits predictive performance, while more expressive methods rely on expensive training data, lack scalability, or fail to fully exploit schema-level semantics. We present Hippasus, a cost-aware, LLM-guided feature augmentation framework over relational schemas, that addresses these challenges. Hippasus ranks candidate join paths before executing any join, by combining lightweight statistical signals with adaptive semantic analysis, invoking an LLM only when necessary. It further introduces efficient multi-way join execution with cross-path feature consolidation, and a hybrid feature selection strategy that integrates statistical relevance with semantic refinement. Experiments on 13 real-world datasets show that Hippasus improves predictive performance by up to 15.7% on average over state-of-the-art methods, while running on par with the fastest of them and keeping its LLM token cost low.

cs.DB · cs.LG

版本更新 · v2arXiv:2610.11482

Evaluating Local Language Model Agents for Data Engineering: An Empirical Software Engineering Study of Mobility Workflows

Jorge García-Carrasco, Javier Sanchis, Alejandro Reina-Reina, Alejandro Maté, Juan Trujillo

原文摘要

Context: Large language model (LLM) agents are increasingly used as software and data-engineering assistants, yet evidence about locally deployable open-weight agents remains limited. Existing evaluations often emphasise textual responses or isolated code rather than complete engineering artifacts. Objectives: We evaluate whether local LLM agents can produce correct and reliable data-engineering artifacts, isolate the contribution of a closed-loop workspace and of input-structure knowledge, and examine trade-offs in runtime, cost, and failure behaviour. Methods: We introduce a benchmark of fifteen mobility data-engineering tasks with deterministic checkers for the scripts, tables, structured files, figures, and reports they produce. Sixteen configurations from six model families are evaluated in three conditions (chat-only, chat-schema, agent-with-tools) with five repetitions per model, condition, and task, yielding 3,600 scored attempts on a consumer-grade GPU. Results: A 4-billion-parameter model at 4-bit reaches 77.3%, a 9-billion-parameter model 80.0% within about 6.5 GB, and a 12-billion-parameter model from another family 81.3%, against 98.7% for the highest-scoring configuration, which solves 14 of 15 tasks in all five repetitions. Above two billion parameters the workspace loop raises pass rates by 24.0 to 65.3 points over uninformed one-shot generation, and its contribution exceeds that of the input-structure summary in eleven of fifteen tasks. Conclusion: Open-weight models that fit on one consumer GPU support a meaningful share of data-engineering work once given a workspace loop, although capability still separates usable from unusable configurations and high-stakes outputs need validation. Being mobility-specific and checker-based, the benchmark characterizes the evaluated configurations, not all data-engineering practice.

cs.SE · cs.AI · cs.DB · cs.LG

版本更新 · v2arXiv:2609.09854

When Does Low-Bit Quantization Preserve the Decisions of Vector Search?

Wenxuan Xiao, Xu Cao

原文摘要

An orthogonal rotation improves the distance rank correlation of two-bit Cohere codes from 0.75 to 0.91, yet increases nearest-neighbour comparison flips by 51%. We analyse quantization through the comparisons made by graph search. A comparison changes only when its quantization error exceeds its exact margin, giving a flip bound that separates close calls from large errors. Shared endpoints induce correlated distance errors that partly cancel within a comparison. An exact replay identity connects candidate-level actions to the output of neighbour selection. A Gaussian model gives coordinate-code errors in closed form, and a rare-spike construction exposes heavy tails in the population-radius error. Held-out blocks give model-free certificates for acceptance rules based on exact or quantized margins. Across 24 dataset-code configurations, calibration-estimated risk correlates with held-out ranking and pruning flips at 0.97 and 0.99, compared with 0.91 and 0.99 for a margin-to-noise summary and 0.74 and 0.05 for distance rank correlation. Paired builds of 200,000-vector graphs connect rotation, query scoring, and topology to end-to-end recall and search work.

cs.DB · cs.IR · cs.LG

版本更新 · v4arXiv:2605.02171

QuIVer: Efficient Graph Indexing with Two-Bit Construction and Navigation

Wenxuan Xiao, Peidong Zhu, Chengcheng Li, Xu Cao

原文摘要

High-dimensional vector indices must balance construction time, memory, and query throughput. Two-bit comparisons reduce computation and storage, but preserving useful graph connections and search candidates requires scores suited to each stage. We present QuIVer, a graph index that uses a training-free sign-magnitude representation for construction and navigation while reserving original vectors for bounded final reranking. Construction discovers candidates with Hamming scores, orders them with an integer proxy, and defers reverse-edge pruning across parallel insertions. Search uses reconstruction norms and bitwise inner products to order both the frontier and retained candidates, without query calibration or an exact anchor. Experiments cover 21 datasets and compare 30 external baseline modes on three core workloads. QuIVer's validation-selected 95% configurations lie on the measured Pareto frontiers of achieved recall, throughput, index preparation time, and memory, whether memory counts navigation state alone or also the original vectors. On Cohere-5M, QuIVer reaches 95.90% Recall@10 at 3,075 single-thread queries/s, $11.74\times$ the throughput of Lucene BBQ with FP32 reranking at 95.83% recall. Against SymphonyQG at 95.87%, it retains 81% of single-thread and 97% of 32-thread throughput while reducing index preparation time by 54%, total logical memory by 47%, and navigation storage by 87%. An SSD mode with INT8 reranking serves Cohere-5M at about 5 GiB of steady-state process RSS. Controlled crossovers and candidate-coverage diagnostics connect these results to construction and retention quality. Across 21 datasets, high recall holds for most text and image embeddings, while non-negative descriptors, cross-modal queries, and random data expose the limits of the representation.

cs.DB

版本更新 · v2arXiv:2607.03501

STRATOS: Bridging the Symbolic-to-Numeric Gap in Spatio-Temporal Text-to-SQL for Meteorological Data

Yi Zhang, Farhad Nooralahzadeh, Jonathan Fürst, Fabio Scherrer, Claude Lehmann, Antonis Bezes, Vassiliki Kotroni, Kurt Stockinger

原文摘要

Copernicus, the European Union's Earth observation program, produces petabytes of Earth observation and climate data, offering immense potential for research, policy, and applications. However, access to these datasets requires advanced programming skills and familiarity with domain-specific formats such as NetCDF or GRIB. Moreover, general-purpose Text-to-SQL systems fail when applied naively to the meteorological domain due to a profound ``Symbolic-to-Numeric'' gap. To overcome these limitations, we present an end-to-end Text-to-SQL framework specifically engineered for real-world, scalable meteorological data exploration. Our system intercepts natural language to resolve spatial and semantic ambiguities \textit{before} SQL generation. We design STRATOS, a Spatio-Temporal Resolution Agent for Text-to-SQL to dynamically bridge the symbolic-to-numeric gap by mapping fuzzy concepts to a localized ontology and resolving spatial entities via external knowledge bases. Further, our complexity-aware query rewriter rewrites expensive spatial predicates, reducing execution times from hours to seconds. Last, we introduce the STRATOS Evaluation Workload, comprising 7,520 complex query pairs explicitly designed by domain experts to test scalability and symbolic-to-numeric translation across challenging spatio-temporal dimensions previously unexplored by Text-to-SQL systems.

cs.DB · cs.AI · cs.MA

版本更新 · v2arXiv:2606.09323

TRL-Bench: Standardizing Cross-Paradigm Representation-Level Evaluation of Tabular Encoders

Wei Pang, Xiangru Jian, Hehan Li, Zhixuan Yu, Alex Xue, Jinyang Li, Zhengyuan Dong, Xinjian Zhao 等 13 位作者

原文摘要

Tabular encoders are usually evaluated inside task-specific end-to-end pipelines, so models from different training paradigms are difficult to compare directly even when they operate on similar tabular signals. We introduce TRL-Bench, a multi-granular tabular representation learning (TRL) benchmark that standardizes cross-paradigm representation-level evaluation: each encoder exports row-, column-, or table embeddings through its supported wrapper, and shared lightweight heads probe them across three suites: TRL-CTbench (column/table), TRL-Rbench (row), and TRL-DLTE (compositional Data-Lake Table Enrichment spanning all three granularities). To support this standardized setting, we release curated benchmark assets and task reformulations, including 50 OpenML tables with 123 verified targets, 16 row-pair linkage rewrites, and a 47,772-table DLTE lake derived from 1,379 parent tables. Across 20 models and 16 tasks, TRL-Bench shows that once downstream conditions are standardized, encoder quality is capability-specific rather than captured by a single leaderboard. In TRL-CTbench, generic text encoders often lead on tasks with strong surface-text signal, while tabular specialists win where their pretraining objective aligns with the task. In TRL-Rbench, within-table prediction and cross-table linkage favor different training regimes, with atomic linkage performance correlating strongly with the row-matching stage of DLTE pipelines. In TRL-DLTE, the strongest pipelines combine capability-matched specialists rather than reuse a single encoder, and top end-to-end quality depends on non-additive compositional fit rather than per-stage marginal rank alone. TRL-Bench provides a common protocol for measuring reusable signal in exported tabular representations under shared downstream conditions. Code: https://github.com/LOGO-CUHKSZ/TRL-Bench; data is released on Hugging Face.

cs.AI · cs.DB

版本更新 · v2arXiv:2602.06594

Machine Learning Practitioners' Views on Data Quality in Light of EU Regulatory Requirements: A European Online Survey

Yichun Wang, Kristina Irion, Paul Groth, Hazar Harmouch

原文摘要

Understanding how data quality aligns with regulatory requirements in machine learning (ML) systems presents a critical challenge for practitioners navigating the evolving EU regulatory landscape. To address this, we first propose a practical framework aligning established data quality dimensions with specific EU regulatory requirements. Second, we conducted a comprehensive online survey with over 180 EU-based data practitioners, investigating their approaches, key challenges, and unmet needs when ensuring data quality in ML systems that align with regulatory requirements. Our findings highlight crucial gaps between current practices and regulatory expectations, underscoring practitioners' need for more integrated data quality tools and better collaboration between technical and legal practitioners. These insights inform recommendations for bridging technical expertise and regulatory compliance, ultimately fostering responsible and trustworthy ML deployments.

cs.DB

新论文 · v1arXiv:2611.15263

Probe-Based Routing for Join Optimizer Selection

Vladimir Burlakov, Alena Rybakina, Sergey Kudashev, Ilgam Latypov, Konstantin Gilev, Alexander Demin, Denis Ponomaryov, Yuriy Dorn

原文摘要

In optimizers that choose a join-order solver by a size rule, join ordering is an algorithm-selection problem, because dynamic programming (DP), genetic search (GEQO), and Monte Carlo tree search (MCTS) each win on different queries. PostgreSQL nevertheless chooses among them by relation count alone, although queries with the same relation count can favor different solvers. This paper introduces two landscape features that a probe computes from random join orders in microseconds per query. The tail spacing $α$ describes how the costs of the best sampled orders are spaced, and the crossover surprise $θ$ describes how far the cost of a crossover child deviates from the mean cost of its parents. The router clusters training queries by these features and the relation count, assigns a solver to every cluster that this solver wins in total end-to-end time by a margin, and leaves all other queries to PostgreSQL's default rule. On JOB and IMDb-CEB, the router lowers mean end-to-end time relative to the default rule, and on IMDb-CEB the probe features nearly double the gain obtained from the relation count alone, so they carry information about the queries that the relation count lacks.

cs.DB

新论文 · v1arXiv:2611.15185

Beyond Symmetry: Asymmetric Matching with Transformers for Collective Entity Resolution

Andrei Olar

原文摘要

A core assumption of entity resolution (ER) is its symmetric nature, which rests on a principle so self-evident it borders on tautology: if a refers to the same object as b, then b must also refer to the same object as a. However, this global property of symmetry does not necessarily extend to each of the complex decision processes within ER systems. In cases involving vague or ambiguous information, enforcing symmetry can be detrimental to performance, an issue exemplified by the ``bad triplet problem''. This paper investigates the effects of input-order asymmetry within a collective ER framework that integrates distinct matching and clustering stages. The input order-aware matching stage utilizes pre-trained transformers to explicitly model and exploit asymmetry, making it possible to evaluate its effect on clustering performance across multiple benchmark datasets. Our results demonstrate that asymmetry is not only measurable but also introduces a significant skew in the final entity clusters, highlighting the need to re-evaluate supervised training methodologies for matchers, particularly the development of direction-aware training paradigms that can preserve, rather than neutralize, the inherent asymmetry in matching tasks.

cs.DB

版本更新 · v2arXiv:2611.07579

Per-Query Recall Guarantees for Approximate Nearest Neighbor Search in Dynamic Vector Databases

Woong-Kee Loh, Suan Lee

原文摘要

Vector search now runs as an operator inside database systems, where a missed neighbor makes a query answer wrong, yet approximate nearest neighbor (ANN) indexes are tuned only for a target average recall. Tuned to an average recall of 0.99, indexes left 5-19% of in-distribution queries and, without a query log, up to 99.6% of out-of-distribution queries below it (more under updates) and changed a top-5-tag aggregate over the results for up to 27% of queries. In the spirit of approximate query processing, we present PQR-Search, a $k$-ANN operator that guarantees $\Pr[\mathrm{Recall}(q) < τ] \le δ$ for every query and, for $τ< 1$, $\mathbb E[\mathrm{Recall}(q)] \ge τ$. The probability is over a random rotation inside the index alone, so no distributional assumption is needed, only that queries and updates do not depend on it, and the guarantee holds after insertions and deletions without retuning. PQR-Search prunes 1-bit codes of rotated IVF residuals with exact error quantiles and predicts its cost without data-fitted constants. On 20 datasets of up to 100M vectors, failure rates stayed at most $δ$ in every audit, including the NeurIPS'23 streaming runbook, and the aggregate changed in at most 0.04% of trials. The guarantee costs 2.5$\times$ the single-query latency of HNSW tuned as above and 1.3$\times$ lower batch throughput (geometric means). PQR-Search was 26$\times$ faster than FAISS exhaustive search in batch, faster than IVF with $n_{\rm probe}$ chosen afterwards to meet the target on every test query, and 27-76$\times$ faster than PUFFINN, which also guarantees per-query recall.

cs.DB

新论文 · v1arXiv:2611.15105

VecDBLens: An Evaluatology-Based Framework for Vector Database Diagnosis

Yutong Zhou, Guoxin Kang, Wenxin Zhou, Haoyu He, Yuedong Zhu, Lei Wang, Wanling Gao, Fanda Fan 等 9 位作者

原文摘要

Vector databases are increasingly deployed as retrieval pipelines in which representation, indexing, and query-time execution jointly determine accuracy, throughput, and cost. Existing vector database evaluations primarily focus on indexing while treating embeddings as fixed inputs and ignoring the role of query-time execution, making it difficult to decide whether a bottleneck should be addressed by changing the embedding model, the index, or the query-processing functions. To address the above limitation, we introduce VecDBLens, an Evaluatology-based framework that treats the function under diagnosis as an evaluated object and all remaining dataset, workload, parameter, and runtime factors as matched evaluation conditions. VecDBLens attributes performance differences to individual evaluated objects through comparisons under matched evaluation conditions, summarizing their performance and stability across multiple datasets and workloads. It further analyzes interactions among objects using analysis of variance (ANOVA), thereby turning vector database benchmarking from leaderboard ranking into a reproducible diagnostic procedure. VecDBLens reveals two key insights. First, retrieval performance is fundamentally upper-bounded by embedding quality, revealing that many apparent indexing failures are therefore representation-side limitations. Second, stronger index parameterization is not always beneficial: overly aggressive configurations can reshape neighborhood topology and cause recall regression. These results challenge the common practice of evaluating vector databases primarily through indexing and show that real-world limitations arise from constraints across retrieval functions rather than any single function alone. VecDBLens provides a principled basis for identifying the true performance drivers of vector databases and for guiding their design in practical applications.

cs.DB

版本更新 · v2arXiv:2611.12169

Transactional Data Change for Real-Time Analytics in Lakehouses

Dongyang Geng, Haoqiong Bian, Zinuo Li, Haoyue Li

原文摘要

Real-time data analytics has become a key demand for lakehouses. It requires propagating data changes from the data source to the lakehouse in real time, while ensuring ACID guarantees and minimizing resource consumption and impact on analytical query performance. This is highly challenging for a data system according to the RUM conjecture, especially given the read-optimized immutable file storage layer in lakehouses. Existing lakehouses offer only git-style catalog version control on a single table, and they trade data freshness, query performance, and/or resource efficiency to achieve a barely acceptable data-change throughput. In this paper, we propose a novel data synchronization framework, Retina, that enables real-time, cost-efficient, query-performant, and fully transactional data changes in lakehouses. We implement and open-source Retina. Evaluations show that Retina delivers 10-millisecond-level data freshness and over 3.2 million row/s linear-scalable data-change throughput with low resource consumption and query-performance overhead, significantly outperforming state-of-the-art lakehouse systems and lake-compatible HTAP databases.

cs.DB · cs.AI · cs.DC · cs.PF

新论文 · v1arXiv:2611.14945

Auriga: Harness-Governed LLM Agents for Effective Throughput Optimization in Execute-Order-Validate Permissioned Blockchains

Xiao Zhang, Tianyang Wang, Fangtao Gu, Haoxiang Sun, Puwei Wang, Jinchuan Chen

原文摘要

Effective throughput, the product of raw throughput and transaction success rate, determines the useful capacity of Execute-Order-Validate blockchains such as Hyperledger Fabric. Most prior work either tunes configurations to raise raw throughput, requiring many costly redeployments, or controls conflicts to raise the success rate, repairing them only after execution with kernel modifications. The few joint approaches tune only a small set of parameters and still rely on kernel changes to resolve conflicts. To address these challenges, we present Auriga, a non-invasive framework that jointly optimizes both factors through sample-efficient LLM-guided configuration search and pre-execution conflict prevention. A harness-governed multi-agent architecture coordinates LLM agents for parameter exploration, saturation-aware evaluation, and experience consolidation, with the harness controlling tool and memory access and retaining termination authority. For transaction scheduling, an LLM extracts symbolic AST templates from smart contracts offline. At runtime, the scheduler predicts read-write sets without executing them and partitions transactions into conflict-free groups. Across three benchmarks and conflict rates up to 70%, Auriga improves effective throughput by up to 6.0 times over default Fabric and 2.0 times over the strongest baseline. The tuning agent also discovers undocumented parameter dependencies, and ablation studies show the independent contributions of both subsystems.

cs.DB

新论文 · v1arXiv:2611.14916

CuEVM: Scaling Smart Contract Execution on GPUs

Nhut-Minh Ho, Tien Tuan Anh Dinh, Quang Trung Ta, Fredrik Svantes, Xiaokui Xiao, Beng Chin Ooi

原文摘要

Smart contracts on Ethereum blockchain process transactions worth billions of US dollars daily. They run on top of Ethereum Virtual Machine (EVM) which executes transactions sequentially. This execution model creates a performance bottleneck, especially for new blockchain workloads with large numbers of transactions, such as security testing. We present CuEVM, a GPU-native and semantically complete EVM execution engine that treats transaction execution as a massively parallel data processing problem. It employs novel data structures and GPU-aware scheduling to accelerate execution. Unlike prior works that transpile the contract and rely on a host CPU for many operations, CuEVM supports complete EVM semantics natively on the GPU. It passes 100% of the Ethereum VMTests and 96.2% of the full execution-test suite. On a single commodity GPU, CuEVM executes over 8 million ERC-20 transfers per second, outperforming existing baselines by up to two orders of magnitude. We build a smart contract fuzzer, called CuEVM-fuzz, on top of CuEVM. The fuzzer achieves over a million end-to-end executions per second, up to 23x and 4.8x higher than the state-of-the-art CPU-based and GPU-based baselines, respectively.

cs.DC · cs.CR · cs.DB

新论文 · v1arXiv:2611.14765

SynthCorp: A Platform for Creating Benchmarks for Data Agents

Jiani Zhang, Manav Garg, Eugenie Lai, Cheng Zhen, Eva Sharma, Yang Pan, Yaming Lin, Alon Halevy

原文摘要

Current benchmarks for LLM-driven data agents saturate quickly and cannot target specific capabilities. We present SynthCorp, a platform that creates customizable data estates and benchmarks by simulating how business events produce and modify corporate data over time. Each data estate combines relational databases, a document store, and a data catalog, paired with evaluation tasks and executable gold programs. First, given a scenario and target complexity dimensions, SynthCorp synthesizes reusable event types and compiles each into a program over predefined simulation primitives. Second, SynthCorp schedules event instances on a virtual calendar, interleaving routine and disruption events so that tables, documents, and catalog entries remain consistent. Third, SynthCorp executes the calendar to materialize the data estate and records every operation in a simulation log. Fourth, SynthCorp generates analytical tasks and derives verified gold programs from the log. We use SynthCorp to generate initial evaluation benchmarks comprising seven data estates (up to 1K operational tables) and 256 tasks across five domains. The data estates pass deterministic pipeline checks (such as foreign-key violations, undeclared writes, recurrence violations, and database execution failures) with a near-zero error rate, introduce cross-asset complexities, and cover all requested dimensions. Evaluating three models across four data-context settings shows that the benchmarks separate data contexts and model tiers. Auditing candidate queries against causal provenance in the simulation log verifies gold programs without manual annotation, including on multi-event composite tasks that expose reasoning failures in state-of-the-art agents. We hope the community uses SynthCorp to generate more challenging and useful data estates and benchmarks that foster the development of enterprise data agents.

cs.AI · cs.DB

版本更新 · v2arXiv:2504.00001

HistogramTools for Efficient Data Analysis and Distribution Representation in Large Data Sets

Shubham Malhotra

原文摘要

Histograms provide a powerful means of summarizing large data sets by representing their distribution in a compact, binned form. The HistogramTools R package enhances R built-in histogram functionality, offering advanced methods for manipulating and analyzing histograms, especially in large-scale data environments. Key features include the ability to serialize histograms using Protocol Buffers for distributed computing tasks, tools for merging and modifying histograms, and techniques for measuring and visualizing information loss in histogram representations. The package is particularly suited for environments utilizing MapReduce, where efficient storage and data sharing are critical. This paper presents various methods of histogram bin manipulation, distance measures, quantile approximation, and error estimation in cumulative distribution functions (CDFs) derived from histograms. Visualization techniques and efficient storage representations are also discussed alongside applications for large data processing and distributed computing tasks.

cs.DB · cs.PF

新论文 · v1arXiv:2611.14706

Beyond Foreign-Key Connectivity: Semantic Relationship Representation for Relational Deep Learning

Noy Cohen-Shapira

原文摘要

Relational databases encode information through tables and primary--foreign key constraints. Although recent Relational Deep Learning (RDL) approaches represent these databases as heterogeneous graphs, a foreign-key connection primarily specifies structural connectivity between records. Its semantic interpretation may involve additional information that is not explicitly represented by connectivity alone. This paper introduces a conceptual framework for constructing relationship-aware representations from relational databases. The framework distinguishes three complementary aspects: (i) schema-defined relationship semantics, including endpoint types and foreign-key roles; (ii) association entities and explicitly available relationship-level attributes; and (iii) contextual information describing the temporal and structural circumstances of a connection. We formulate a Relationship Representation Construction (RRC) protocol that maps relational schema elements and database records to node representations, relationship descriptors, and contextual representations. A key principle of the protocol is semantic ownership: attributes are assigned to records, relationships, or context according to their meaning, rather than merely their physical location in a database table. Ambiguous attributes are not automatically promoted to relationship features. The resulting representations can be incorporated into existing relational graph learning architectures through relationship-conditioned message passing. The proposed framework is complementary to recent work on relational message passing, graph transformers, and task-dependent graph construction. We discuss its conceptual foundations, illustrate the construction protocol using a multi-table database, and identify research questions and empirical evaluations required to establish its potential benefits.

cs.DB · cs.LG

新论文 · v1arXiv:2611.14317

MammothDP:面向可信硬件与无泄漏超参数的私有提升决策树

MammothDP: Private Boosted Decision Trees Ready for Trusted Hardware and Leakage-free Hyperparameters

Moritz Kirschte, Thorsten Peinemann, Kari Kostiainen, Jan Wichelmann, Thomas Eisenbarth, Esfandiar Mohammadi

AI 速读 基于摘要

作者提出 MammothDP,将 S-BDT 梯度提升决策树与隐私过滤器、时序侧信道加固和 TEE 部署结合,并引入无隐私开销的早停准则与改进的回滚保护。

cs.CR · cs.DB · cs.LG

查看原文摘要

Data spaces like the European Health Data Space (EHDS) require the protection of sensitive data in studies during the data's processing and the study's release alike. In this work, we investigate how to implement this mandate using established trusted execution environments (TEEs) and timing side-channel hardening to protect the processing even under a compromised server, and the state-of-the-art differential privacy (DP) notion to protect the release of studies on tabular data. This combined setup could seem to complete the picture, but in fact raises unexpected security challenges that we mitigate by introducing MammothDP: a significant milestone towards a holistic machine learning system based on the recent gradient boosted decision tree learner S-BDT that is provably DP if used within a privacy filter, timing side-channel hardened, and deployed on a TEE. To achieve a strong privacy-utility tradeoff, we introduce two novel components: an early stopping criterion that adaptively selects the number of trees without any privacy overhead, and an improved rollback protection that is no longer vulnerable under timing side-channel attacks.

新论文 · v1arXiv:2611.14601

VASTRA: Efficient Agentic Top-K Interactive Video Analysis

Tushar Tyagi, Aditya Raghavan, Haoran Qiu, Francisco Romero

原文摘要

Vision-language models (VLMs) enable natural language video retrieval, bypassing traditional hand-engineered pipelines. Integrating agentic reasoning frameworks (e.g., ReAct) further advances this paradigm by allowing models to dynamically inspect video content rather than relying on a single visual pass. Applying this pattern naively to video retrieval inherits problems that video systems cannot afford: results vary from run to run under identical conditions, and cost and latency scale with every reasoning step the agent takes rather than with the work the query actually requires. We present VASTRA, an agentic video retrieval system whose execution is dictated by query complexity rather than an open-ended reasoning loop: cheap, reliably answerable queries are ranked directly from pre-ingested summaries, while harder queries trigger targeted verification only where relevance is genuinely uncertain. This keeps control flow external to the model rather than latent in a growing chain of thought, establishing deterministic control flow by construction and end-to-end rank reproducibility under greedy serving. Across 40 benchmark queries, VASTRA exposes explicit presets: VASTRA-Fast runs 12.2x faster (4.2 s vs. 51.2 s) and 10.4x cheaper ($0.0044 vs. $0.0458) than best performing ReAct agent at a 5.0-point mAP reduction, while VASTRA-Max matches peak ReAct agent accuracy (0.653 vs. 0.649 mAP) at comparable latency, trading higher spend ($0.1022 vs. $0.0458) for zero run-to-run variance.

cs.DB

版本更新 · v2arXiv:2611.10398

冷缓存下表格数据格式的测量:基于真实数据集的序列化、扫描与索引性能复现基准

Measuring Tabular Data Formats Cold: A Replicated Cold-Cache Benchmark of Serialization, Scan, and Index Performance on Real Datasets

Asra Fathima, Kristoffer Roberts

AI 速读 基于摘要

该研究在显式冷缓存、随机执行顺序和三次独立会话条件下,对两种真实数据集上的十五种表格数据格式与访问路径配置进行基准测试,并用统计检验分析性能差异。

cs.DB · cs.PF · cs.SE

查看原文摘要

Choosing a file format for tabular data is a routine engineering decision, yet published comparisons often time reads served from the operating-system page cache, use one session, or rely on a single synthetic dataset. This article benchmarks fifteen format and access-path configurations (CSV, JSON Lines, Pickle, Feather, Parquet, ORC, DuckDB, and SQLite, with several access paths) on two real public datasets, New York City taxi trips and United States airline on-time records, at 50,000 to 3 million rows. Every measurement uses an explicitly cold cache, randomized execution order, and three independent sessions (with restarts on the laptop), and is analyzed with bootstrap intervals and Holm-adjusted permutation tests. Parquet answered a filtered aggregation 2.6 to 11 times faster than CSV, while JSON Lines was 10 to 27 times slower. A secondary index made SQLite queries 2.6 to 4.8 times slower on randomly ordered data and helped only on data sorted by the filtered column, at low selectivity. A row filter inside the Parquet reader was significantly faster than a projected scan on flight data from one million rows and on both datasets at three million, and row-group skipping needs several row groups, which the default writer produced only at 3 million rows. A 4 GB Linux host ran out of memory on configurations that a 24 GB laptop completed and reversed some orderings, such as Pickle versus Parquet. Code, derived data, and raw timings are released.

版本更新 · v2arXiv:2611.07342

Selective Retrieval Routing for Multi-Violation Knowledge Graph Repair

Tung-Wei Lin, Gabe Fierro, Han Li, Tianzhen Hong, Pierluigi Nuzzo, Alberto Sangiovanni-Vincentelli

原文摘要

Knowledge graphs often contain missing or inconsistent facts that undermine the applications built on them. Schema validation detects such violations but does not identify which of several valid repairs recovers the intended graph. External corpora can supply the evidence needed to identify the intended repair, but retrieving from them is costly and not always successful. We propose a framework for selective retrieval in iterative graph repair. An LLM repair agent explores and edits the graph, while a router controls its access to a researcher agent that retrieves external evidence. We extend a violation-injection framework to generate cases with multiple compatible violations and known reference repairs. The agent's repair outcomes on these cases provide supervision for training the deployment-specific router. On two building-automation datasets, researcher access improves repairs for some violations and degrades them for others, with effects differing across deployments. The router uses fewer repair-time tokens than always enabling access in all six evaluation suites. It achieves higher repair $F_1$ than always disabling access in five of six suites.

cs.DB · cs.AI

版本更新 · v2arXiv:2605.15790

Fairness-Aware Retrieval Optimization for Retrieval-Augmented Generation

Yingqi Zhao, Vasilis Efthymiou, Jyrki Nummenmaa, Kostas Stefanidis

原文摘要

Retrieval-Augmented Generation (RAG) grounds large language models (LLMs) in external knowledge, but the retrieved documents can systematically skew generated answers toward one group. This effect is difficult to analyze and control in top-k settings, where several documents at different context positions jointly shape the output. We present a three-stage framework for fairness-aware retrieval in top-k RAG. First, a probabilistic reranker sets the group composition of the retrieved context at each position, enabling controlled perturbation of retrieval bias without modifying the retriever. Second, we show that output bias is well approximated by a position-weighted linear function of position-wise retrieval bias, revealing model-specific positional sensitivities. Third, we use this model to formulate retrieval as an optimization problem that trades off relevance against predicted output bias, and propose FARO, a scalable solution to that problem which decomposes the globally coupled quadratic objective into independent per-query assignment problems through a Fenchel--Legendre dual reformulation. Experiments with four open LLMs, and political and gender bias as use cases show that the linear model predicts output bias on unseen configurations in 17 of 24 settings. FARO substantially reduces output bias in most settings, while, compared with a linear-programming baseline, FARO achieves similar relevance, remains feasible where the baseline does not, and scales better to larger query sets and candidate pools.

cs.DB · cs.IR

版本更新 · v3arXiv:2607.25891

Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation

Stefan Krsteski, Charlotte Meyer, Guillaume Allegre, Tony O'Halloran, Alexandre Sallinen

原文摘要

Comprehensively evaluating AI agents across interactive environments is difficult due to fragmented tasks, scaffolds, verifiers, and scoring rules. Unfortunately, existing efforts to unify these evaluations are limited in scale and domain, making costly reruns necessary and leaving available data incomparable. We introduce Messier, a unified corpus of 923,158 records spanning 30 benchmarks, 745 agents, 11,891 tasks, and 71,969 verifiers. Messier combines public evaluation results with new runs on six underrepresented professional and scientific benchmarks, standardizing their heterogeneous components into a common schema. Using this corpus, we show that frontier progress is uneven across benchmark groups, with function-calling evaluations largely saturated, programming improving fastest, and enterprise workflows remaining most challenging. Counterfactual rescoring further shows that reported performance can be highly sensitive to how verifier results are aggregated. Finally, we derive capability scores from our corpus that correlate with Epoch's Evaluation Capability Index rankings at Spearman \r{ho} = 0.84. The scores can also be estimated for subsets defined by domain, occupation, action space, or verifier type. In essence, Messier is a reusable resource for studying agent performance at scale, and a basis for designing better evaluations.

cs.AI · cs.DB

新论文 · v1arXiv:2611.13856

SAIL: Self-Adaptive Index Learning with Hybrid Models (Extended Version)

Kasun Amarasinghe, Farhana Choudhury, Jianzhong Qi

原文摘要

Learned index structures have demonstrated a significant improvement in query performance over traditional index structures. The majority of learned indexes use a uniform node structure throughout their index structures. A few studies show that combining multiple types of node structures can lead to indexes that better adapt to local key-set distributions, hence yielding query efficiency gains. Following this line of work, we propose a learned index structure named SAIL that supports a wide range of different candidate node structures for both leaf and inner nodes, selected via cost models based on data driven approximations. We design an efficient construction algorithm that recursively partitions the key-set to be fitted by the candidate node types. Experiments demonstrate that SAIL achieves 2x improvement over existing learned indexes in query time, with similar memory consumption and index build times.

cs.DB

新论文 · v1arXiv:2611.13765

Beyond Interaction Scores: Selecting Feature Interactions via Independent Replication

Jincheng Gao, Shuo Sun

原文摘要

Learning feature interactions is essential for accurate prediction and interpretable modeling, yet a high interaction score may reflect fitting noise rather than a reproducible pattern. Ranking methods score feature pairs but leave the cutoff to the user; testing methods set the cutoff but rely on repeated model refits or a model of the joint feature distribution. We present LENS, a model-agnostic framework for selecting feature interactions through independent replication. LENS trains two models with the same learning algorithm on disjoint data and accesses them only through predictions. For each feature pair, it removes main effects from the joint-effect surface to isolate an interaction pattern, ranks pairs by its strength, and tests whether the second model reproduces it at a meaningful strength. A single multiple-testing correction determines which pairs to retain, using only two model fits regardless of the number of features. We prove that a replication ratio alone can retain arbitrarily weak patterns shared by both fits, whereas a strength floor restores selection consistency, and we characterize when normalization in Friedman's $H^2$ reverses interaction rankings. On 16 synthetic designs, LENS matches leading model-specific ranking methods (e.g., TreeSHAP) and recovers 91% of material interactions, against 45% for the knockoff-based DIAMOND, without selecting false pairs; with 1,225 candidate pairs and no interactions, it makes substantially fewer false discoveries. On large-scale French motor insurance data, it identifies interactions documented in the actuarial literature and recovers 90% of the deviance gain obtained using all pairs. On two credit datasets, LENS selects few pairs while maintaining or slightly improving predictive performance over the additive baseline.

cs.LG · cs.DB · stat.ML

新论文 · v1arXiv:2611.13209

ARoute: Task-Aligned Batch and Streaming Model Routing

Daofu Zhang, Qingcheng Zeng, Chunwei Liu, Fuheng Zhao

原文摘要

Language models increasingly power data-intensive applications, from semantic data processing to question answering and agentic workflows. Using frontier models for every request is prohibitively expensive, while relying only on smaller models reduces accuracy. We study model routing under a user-specified cost budget, aiming to maximize accuracy under task-specific correctness criteria. Existing routers fall short in two ways: pretrained routers rely on generic preference signals that may not reflect a task's own correctness criteria or a new model pair, and model cascades run the weaker model first on every request, adding sequential latency. We present ARoute, a task-aligned router that learns cost and accuracy signals to select a model for each request before inference. ARoute has two variants: ARoute-Batch operates on a batch of requests known in advance, routing requests with the greatest expected benefit to the stronger model subject to the budget, whereas ARoute-Stream processes requests online, using a threshold calibrated on a small validation set to satisfy the cost quota and dynamically adjusting it as the budget is consumed. For ARoute-Stream, we provide a finite-sample guarantee on expected normalized spending. We evaluate ARoute on MMLU-Pro, LiveCodeBench, SuperGPQA, and two value-filling tasks built from SWAN semantic query benchmark against other state-of-the-art approaches. At matched cost, ARoute-Batch achieves the highest mean accuracy in all 15 task-cost settings, improving over the strongest baseline by 4.1 percentage points on average and by up to 11.9 points. ARoute also completes a batch up to 2.8x faster than cascades.

cs.DB

版本更新 · v2arXiv:2610.00038

BuildGraph: A Synthetic Multi-Archetype Building Knowledge Graph Dataset

Wooyoung Jung

原文摘要

Semantic querying of building knowledge graphs (KGs) underpins the integration of artificial intelligence into building operations, from natural-language access to cross-building analytics, but such KGs are rarely public owing to proprietary, security, and cost barriers. BuildGraph is a synthetic building KG dataset of 120 buildings in the Brick Schema ontology, grounded in U.S. Department of Energy prototype models and sensor-placement patterns from real buildings. It spans eight commercial building types across three ASHRAE energy-code vintages, with five realizations per archetype varying URI naming, sensor-attachment predicates, and topology. A 75-query SPARQL benchmark confirms structural completeness: BuildGraph reaches 91.5% Query Answerability Rate versus 35.8% for 59 real-world Brick files. Independently, it reproduces real buildings' sensor-type proportions on their shared vocabulary (cosine 0.86-0.94), evidence of realistic instrumentation where measurable. A downstream text-to-SPARQL experiment with Gemma 4 (26B) reaches 27.3% Row-Matching F1 (+17.5 pp over zero-shot) on 12 held-out buildings. BuildGraph gives facility managers and digital-twin developers a testbed for portable analytics and natural-language interfaces, and its dataset, generator, and benchmark are openly available at https://github.com/humanbuildingsynergy/BuildGraph.

cs.DB

新论文 · v1arXiv:2611.13138

Answering Conjunctive Queries over Linear Existential Rules with Transitivity

Cristina Feier, Tomasz Gogacz, Filip Murlak

原文摘要

We study Conjunctive Query Entailment Problem over ontologies con- sisting of Linear Tuple Generating Dependencies and transitivity asser- tions. A special case of this problem has been shown to be decidable in [1]. In particular, it covers the case of linear single head TGDs with bi- nary signature. In This paper we show that The problem for linear multi head TGDs with binary signature is Ackermanian hard

cs.DB

新论文 · v1arXiv:2611.12817

LUCQUE:用于可靠AI评估的魁北克肺癌筛查数据集

LUCQUE: A lung cancer screening dataset from Québec for reliable AI evaluation

Johanna Topalis, Yannick Lemaréchal, Gabriel Couture, Hanife Ersay, Loran Nas, Balthasar Schachtner, Bastian Oliver Sabel, Michael Ingrisch 等 10 位作者

AI 速读 基于摘要

作者提出并构建了来自魁北克LDCT肺癌筛查的大型、全面标注数据集LUCQUE,并通过模型到数据的方式支持隐私保护、可复现的AI评估。

cs.DB · cs.AI

查看原文摘要

BACKGROUND: AI-based tools are expected to play a key role in lung cancer screening (LCS). The evaluation of these tools often relied on private datasets or fragmented public datasets with inconsistent availability of clinical data and expert nodule annotations. This has hindered reproducibility, comparability, and independent assessment of AI tools. To address these gaps, we present LUCQUE, a novel, large, comprehensively annotated dataset derived from LDCT lung cancer screening in Québec, which is accessible for reliable AI evaluation. METHODS: LUCQUE was curated and is hosted on PARADIM, a digital infrastructure supporting DICOM data de-identification, annotation, long-term storage and analysis. LDCT scans and clinical information were combined to a curated collection and complemented by voxel-level, expert-derived pulmonary nodule and cyst segmentations, and Lung-RADS-based lesion characteristics. Key dataset characteristics and statistics were reported, including participant demographics, imaging parameters, radiological findings, and segmented lesion characteristics. Dataset analyses were conducted according to PARADIM's model-to-data approach, in which user code is executed on the platform such that data remains on premise. RESULTS: LUCQUE currently comprises 1,678 LDCT baseline studies from 1,678 participants. Most studies were classified as Lung-RADS 2. A total of 1,776 pulmonary nodules and 60 cysts were annotated, with a median (interquartile range) volume of $89\,(56,\,159)\,\mathrm{mm}^3$ and $1,384\,(589,\,3793)\,\mathrm{mm}^3$ respectively. CONCLUSIONS: Compared to publicly available LCS datasets, LUCQUE currently represents the largest accessible annotated real-world LCS dataset. Access to LUCQUE is provided via a model-to-data approach, enabling privacy-preserving, reproducible, and independent real-world evaluation of AI tools in LCS.