Minwoo Park, Junwoo Chang, Jongeun Choi, Roberto Horowitz
Equivariant diffusion policies (EDPs) combine the expressivity of diffusion models with the generalization and sample efficiency afforded by geometric symmetries. While steering these policies with reinforcement learning (RL) offers a promising fine-tuning mechanism beyond demonstration data, directly applying standard (non-equivariant) RL ignores the symmetries that EDPs are designed to exploit, leading to sample inefficiency and instability. We theoretically establish that the diffusion process of an EDP is equivariant, inducing a group-invariant latent-noise MDP which is well-suited for equivariant diffusion steering. Building on this, we introduce a principled symmetry-aware steering framework and compare standard, equivariant, and approximately equivariant RL strategies across tasks with varying degrees of symmetry. While we identify the practical boundaries of strict equivariance under symmetry breaking, exploiting symmetry during the steering yields substantial benefits---enhancing sample efficiency, preventing value divergence, and achieving strong policy improvements even when EDPs are trained from extremely limited demonstrations.
Panagiotis Georgios Pennas, Konstantinos Papaioannou, Marco Guarnieri, Thaleia Dimitra Doudali
Large Language Models (LLMs) rely on optimizations like Automatic Prefix Caching (APC) to accelerate inference. APC works by reusing previously computed states for the beginning part of a request (prefix), when another request starts with the same text. While APC improves throughput, it introduces timing side channels: cache hits are faster than misses, creating observable latency differences. In multi-tenant systems, attackers can exploit these differences to infer sensitive information, e.g., by incrementally reconstructing another user's request by observing hit/miss patterns. Current defenses take a sledgehammer approach: they disable APC and cache sharing, isolating users, and sacrificing efficiency for regular users. This paper presents CacheSolidarity, a system that secures multi-tenant LLM serving systems against APC side channels without sacrificing performance and efficiency. CacheSolidarity monitors cache reuse across users, flags suspicious sharing, and selectively isolates prefixes, restricting their reuse only when necessary. Evaluation shows that CacheSolidarity enables up to 70% higher cache reuse and 30% lower inference latency compared to existing user-isolation defenses. CacheSolidarity's lightweight design demonstrates how LLM serving can be secured without sacrificing performance or incurring unbearable overheads.
Rong Fu, Chunlei Meng, Haoyu Zhao, Kun Liu, Simon James Fong
The pursuit of optimal trade-offs in high-dimensional search spaces under stringent computational constraints poses a fundamental challenge for contemporary multi-objective optimization. We develop NeuroPareto, a cohesive architecture that integrates rank-centric filtering, uncertainty disentanglement, and history-conditioned acquisition strategies to navigate complex objective landscapes. A calibrated Bayesian classifier estimates epistemic uncertainty across non-domination tiers, enabling rapid generation of high-quality candidates with minimal evaluation cost. Deep Gaussian Process surrogates further separate predictive uncertainty into reducible and irreducible components, providing refined predictive means and risk-aware signals for downstream selection. A lightweight acquisition network, trained online from historical hypervolume improvements, guides expensive evaluations toward regions balancing convergence and diversity. With hierarchical screening and amortized surrogate updates, the method maintains accuracy while keeping computational overhead low. Experiments on DTLZ and ZDT suites and a subsurface energy extraction task show that NeuroPareto consistently outperforms classifier-enhanced and surrogate-assisted baselines in Pareto proximity and hypervolume.
Programming the static friction of mechanical interfaces is critical for soft robotics, haptics, and precision gripping. Static friction is governed by the real contact area, and standard rough surfaces exhibit a linear area-load scaling inherent to classical models, severely restricting their functional range. Here, we propose a framework for the inverse design of tribological metainterfaces engineered for programmable contact behaviors. By utilizing general axisymmetric asperities, we unlock nonlinear macroscopic responses unattainable by standard Hertzian contacts. To solve the inverse problem, we embed a fully differentiable contact mechanics engine within a neural network and a quadratic optimizer. We leverage regularized physical gradients to automatically discover non-standard topographies that reproduce complex target contact laws, with only a few asperities in unit cells. The predicted designs are validated against high-fidelity Boundary Element Method (BEM) simulations. By front-loading computational costs during training, the neural surrogate yields near-instantaneous design inference, bypassing the bottlenecks of traditional multi-start optimization. This framework bridges data-driven optimization and rigorous physics, offering a scale-invariant pathway for the high-throughput discovery and real-time control of functional tribological surfaces.
The universal approximation property uniformly with respect to weakly compact families of measures is established for several classes of neural networks. To that end, we prove that these neural networks are dense in Orlicz spaces, thereby extending classical universal approximation theorems even beyond the traditional $L^p$-setting. The covered classes of neural networks include widely used architectures like feedforward neural networks with non-polynomial activation functions, deep narrow networks with ReLU activation functions and functional input neural networks.
The integration of Artificial Intelligence (AI) into corporate strategy has become critical for organizations seeking to maintain competitive advantage in the digital age. Although organizations increasingly rely on AI as a strategic and organizational resource, existing C-suite roles remain only partially equipped to govern, integrate, and leverage it coherently at the enterprise level. Organizations vary in their responses. Some create a dedicated Chief AI Officer (CAIO), others extend existing mandates into hybrid roles, and still others coordinate AI through federated structures. This paper develops a role-design theory to explain this variation. I identify three properties that distinguish AI systems from earlier cross-cutting enterprise technologies, distributed accountability for judgment, upstream governance, and non-stationarity, and three configurations through which organizations respond: concentrated extension, distributed extension, and role creation. Four propositions specify when a dedicated CAIO emerges, what form an organization's response takes, when the role is effective, and how configurations evolve. Drawing on the first archival census of executive AI leadership, I find that the role has diffused sharply since 2023, that role creation and mandate extension are advancing in near parity, that adoption concentrates where digital leadership structures already exist, and that executive tenures contradict practitioner accounts of a transient role. The paper contributes to research on executive leadership, AI governance, and the design of AI-driven organizations.
Rong Fu, Xiaowen Ma, Kun Liu, Wangyu Wu, Ziyu Kong, Jia Yee Tan, Tailong Luo, Xianda Li 等 10 位作者
Deploying expressive learning models directly on programmable dataplanes promises line-rate, low-latency traffic analysis but remains hindered by strict hardware constraints and the need for predictable, auditable behavior. Chimera introduces a principled framework that maps attention-oriented neural computations and symbolic constraints onto dataplane primitives, enabling trustworthy inference within the match-action pipeline. Chimera combines a kernelized, linearized attention approximation with a two-layer key-selection hierarchy and a cascade fusion mechanism that enforces hard symbolic guarantees while preserving neural expressivity. The design includes a hardware-aware mapping protocol and a two-timescale update scheme that together permit stable, line-rate operation under realistic dataplane budgets. The paper presents the Chimera architecture, a hardware mapping strategy, and empirical evidence showing that neuro-symbolic attention primitives can achieve high-fidelity inference within the resource envelope of commodity programmable switches.
Deep neural networks on manifold-valued representations have attracted growing interest, but many basic components remain tied to specific manifolds, rely on Euclidean approximations, or require costly and numerically fragile geometric operations. This thesis develops a unified framework for Riemannian deep learning from three complementary perspectives: reusable neural modules, manifold-specific network architectures, and the design of underlying geometries. It generalizes batch normalization from Euclidean spaces and individual manifolds to broad classes of Lie groups and gyrogroups, and extends multinomial logistic regression from Euclidean space to SPD manifolds and then to general Riemannian manifolds. It further develops neural networks for several important geometric representations, including an unconstrained model of hyperbolic space, Busemann-based hyperbolic learning, and full-rank correlation matrices. Finally, it introduces adaptive and computationally efficient Riemannian metrics on SPD manifolds, including learnable Log-Euclidean geometries and fast, stable Cholesky-based geometries. The proposed methods are supported by theoretical analysis and validated through numerical experiments and applications in vision, signal processing, graph learning, and genomics.
Most modern optimizers form their momentum as an exponential moving average (EMA) of past gradients, forgetting every direction at one fixed rate. However, the inputs a deep network sees during training can be highly anisotropic, with a few directions queried frequently while most are seen rarely. Preconditioning methods address this anisotropy by wrapping extra processing around this buffer and leave the momentum update itself unchanged. We propose Activation-Keyed Momentum (AK-Momentum), which builds direction-awareness into the momentum update rule. The gradient of a linear layer splits into an input activation that acts as a $\textit{key}$ and an output-side error that acts as a $\textit{value}$. Keying on that activation, AK-Momentum updates the momentum buffer by the canonical delta rule, so each direction is forgotten at a rate set by how often it appears. We prove that it is a valid momentum, that it applies the input-side curvature correction without matrix inversion, and that it clears stale directions faster than EMA under both a fixed and a drifting optimum. It is a drop-in replacement for the momentum buffer of any optimizer, its coefficient transfers across widths under $μ$P, and its extra compute stays between $22.2\%$ and $25.0\%$ of a gated-MLP block's linear cost with no persistent memory. In FineWeb-Edu pretraining, AdamW with AK-Momentum (AK-AdamW) reaches AdamW's validation loss in up to $46.64 \pm 4.52\%$ fewer steps at 67M and $22.14 \pm 0.81\%$ at 370M over three seeds, and the gain persists at 1B on a Chinchilla-optimal budget. A Muon baseline tuned under the same protocol sits above AK-AdamW at both language-model scales, and the gain holds for SGD, ResNet-18, and ViT-Tiny on CIFAR-10. Training-time diagnostics confirm the predicted mechanism, better gradient tracking and healthier input directions. Code is available at https://github.com/euijinh/AK-Momentum.
Roman Rausch, David Jansen, Sukhbinder Singh, Román Orús
Low-rank factorization of weight matrices is a promising approach to compress large language models (LLMs), but its practical effectiveness depends on allocating ranks across layers and efficiently representing the resulting factors. We introduce three complementary methods for improving SVD-based LLM compression. First, FermiGrad formulates global layer-wise rank allocation as a differentiable optimization problem by replacing discrete singular-value truncation with continuous Fermi gates. Because FermiGrad optimizes only rank parameters over precomputed low-rank factors, it avoids repeated activation decompositions and backpropagation through SVD operations. Second, the Targeted Compressor extends FermiGrad with a task-specific objective that directs limited model capacity toward a selected domain. At 60% parameter density for Llama-3.1-8B-Instruct, SciQ-targeted compression achieves 60% MMLU accuracy, exceeding the reported quantized Dobi-SVD result by 10 percentage points, while deliberately trading performance on unrelated tasks for domain specialization. Third, PivGa provides a lossless secondary compression of low-rank factors by exploiting the gauge freedom in their parametrization. The proposed framework is modular, computationally efficient, and applicable at scale, as demonstrated by compressing Llama-3.3-70B-Instruct to 50% parameter density.
Learning a compact model of the world from interaction data is central to sample-efficient deep reinforcement learning. Spectral representation methods have become the leading paradigm for representation learning in continuous control by taking a matrix view of the transition kernel, with state-action pairs on one side and next states on the other, and learning a low-rank factorization through self-supervised contrastive objectives. We take this view one step further. The transition kernel is naturally a three-mode tensor over states, actions, and next states, and a CP decomposition gives one feature map per mode. We propose FaStR, which fits this decomposition with a noise contrastive objective, producing separate state, action, and next-state encoders that together form a single spectral representation. The factored form yields a smaller hypothesis class, and the sample size needed for representation learning shrinks by a factor that scales with the smaller of the state and action dimensions. Empirically, FaStR delivers its largest gains on high-dimensional locomotion tasks whose dynamics align with the factored structure, and the learned state encoder transfers intact across actuator shift while only the action encoder is retrained.
Route choice models are one of the most important foundations for transportation research. Traditionally, theory-based models have been utilized for their great interpretability, such as logit models and Recursive logit models. More recently, machine learning approaches have gained attentions for their better prediction accuracy. In this study, we propose novel hybrid models that integrate the Recursive logit model with Graph Neural Networks (GNNs) to enhance both predictive performance and model interpretability. To the authors' knowldedge, GNNs have not been utilized for route choice modeling, despite their proven effectiveness in capturing road network features and their widespread use in other transportation research areas. We mathematically show that our use of GNN is not only beneficial for enhancing the prediction performance, but also relaxing the Independence of Irrelevant Alternatives property without relying on strong assumptions. This is due to the fact that a specific type of GNN can efficiently capture multiple cross-effect patterns on networks from data. By applying the proposed models to one-day travel trajectory data in Tokyo, we confirmed their higher prediction accuracy compared to the existing models.
Generative AI can convert uncertainty into authoritative-seeming verdicts, displacing the justificatory work on which democratic epistemic agency depends. As a corrective, I propose a Brouwer-inspired assertibility constraint for responsible AI: in high-stakes domains, systems may assert or deny claims only if they can provide a publicly inspectable and contestable certificate of entitlement; otherwise they must return Undetermined. This constraint yields a three-status interface semantics (Asserted, Denied, Undetermined) whose statuses mark entitlement to categorical speech rather than truth values of the underlying world-claim. The framework distinguishes internal entitlement from public standing while connecting them through the certificate as a boundary object. Philosophically, it develops a form of applied intuitionism: resistance to unwarranted completeness becomes a norm of public machine speech, under which categorical commitment requires exhibited and contestable entitlement. I operationalize the constraint through structured certificates, decision-layer gates for threshold and argmax predicates, and a reason-coded output contract. A design lemma shows that, under a refutation-soundness condition on the contractually specified negative side, certificate-sound binary totality presupposes witnessed decidability on the declared scope. Under the proposed normative output contract, Undetermined is not a tunable reject option but a mandatory status whenever the available certificates do not uniquely license assertion or denial. By making outputs answerable to contestable warrants rather than confidence alone, the paper aims to preserve epistemic agency where automated speech enters public justification.
Reasoning distillation aims to transfer multi-step reasoning capabilities from large language models to smaller, efficient ones. While recent methods have shown promising gains, they typically rely on static teacher-student hierarchies and frame distillation as trajectory imitation. This is misaligned with the structure of reasoning, where intermediate steps are locally under-specified: global correctness constrains the final answer, but does not uniquely determine each intermediate move. We propose validity-calibrated reasoning distillation, a framework that treats reasoning distillation as a problem of local learning-signal allocation rather than path alignment. Instead of enforcing token-level imitation, we compare the student's and teacher's proposed next-step actions under the same prefix and use their relative local validity to modulate the strength of the distillation update. This yields a dynamic, context-dependent supervision that preserves the teacher's structural guidance while adapting update strength to local reasoning quality. Across math, code, and instruction-following benchmarks, our method outperforms strong distillation baselines. These results indicate that effective LLM reasoning distillation is governed not by rigid trajectory imitation, but by locally calibrated allocation of learning signal.
James Chappell, Sophie Luise Arana, Jack Bowyer, Matthew Burnand, Alejandro Coca-Castro, Evangeline Corcoran, Eric G. Daub, Tom Dunstan 等 18 位作者
Preserving fine-scale structure without degrading spatial accuracy is a central challenge for kilometre-scale machine-learning weather prediction (MLWP). We introduce a loss based on the dual-tree complex wavelet transform that constrains atmospheric fields jointly by scale, location, and orientation. We evaluate its performance using an experimental joint global--regional FastNet configuration, which produces \SI{1.5}{km} forecasts over the UK alongside $\sim$\SI{31}{km} global forecasts. Relative to both mean-squared error (MSE) and an FFT-based spectral objective, the wavelet loss substantially improves spectral fidelity. Compared with MSE training, it maintains similar pointwise performance, improves neighbourhood-based verification, and produces sharper precipitation fields with more realistic intensity distributions. These results demonstrate that spatially localised wavelet constraints can preserve multiscale atmospheric structure while largely avoiding the loss of pointwise accuracy associated with globally supported spectral objectives.
Larger models and longer contexts make the KV cache a memory and bandwidth bottleneck. We introduce Block-GTQ, a framework that uses the two-dimensional frequency blocks of rotary position embeddings (RoPE) to allocate cached-key precision. A blockwise logit-error bound motivates a label-free query-key energy score, which we combine with codec-dependent rate penalties. Under equal-cost budgets and diminishing marginal gains, largest-marginal-gain allocation exactly minimizes the separable integer proxy. We instantiate Block-GTQ with TurboQuant-MSE (TQ-MSE), grouping equal-rate blocks to support attention over packed caches without materializing a decoded fp16 KV cache. Across ten model configurations, K-only diagnostics reduce layer-averaged logit mean absolute error by about 32-80% relative to uniform TQ-MSE, improving all 367 layers at both 2- and 3-bit average logical key rates. At K2V2 (2 average logical bits per K/V coordinate) on Llama-3.1-8B-Instruct, Block-GTQ raises the six-task needle-in-a-haystack (NIAH) average from 70.6 to 97.4 and LongBench-EN from 33.7 to 44.9; K2V2 gains also hold on the tested 32B and 70B models. On DeepSeek-R1-Distill-Qwen-7B at K3V2, without fp16 sink or recent-token buffers, it achieves 49.6/38.8% avg@8 on AIME 2024/2025 versus fp16's 54.2/37.9%, where uniform TQ-MSE scores zero on both. On Qwen2.5-3B-Instruct, packed K3V3 serving on one H800 reduces resident KV footprint by a factor of 3.24 and retains perplexity close to fp16 at 128K, while uniform TQ-MSE degrades severely.
Tareque Mohmud Chowdhury, Abu Raihan Mostofa Kamal
Random Forest--based selectors such as Boruta depend on tree depth, yet Boruta's most used implementation, BorutaPy, recommends depths of 3 to 7 without published evidence. We tested the effect of depth using pan-cancer data from The Cancer Genome Atlas (10,469 tumors, 33 cancer types) and breast cancer PAM50 subtyping (981 tumors, 5 subtypes). Label-dependent steps were fitted inside training folds, effects were measured against seed-to-seed variation, and each task had a locked test set. In pan-cancer data, depth 5 changed the selection far more than a new seed did: the top-500 Jaccard overlap with unlimited depth was 0.37, against 0.84 (depth 5) and 0.93 (unlimited) between seeds. Depth-5 trees confirmed 472 instead of 1,659 genes, covered fewer large-class markers (16% vs 62%) and gave less accurate 50-gene panels (balanced accuracy 0.809 vs 0.889). The effect vanished once the cap reached 61 to 76% of the median depth of unrestricted trees, a depth set mainly by the number of classes, not sample size. In PAM50 subtyping (median depth 12) the effect was small, and leaving depth uncapped cost no accuracy in either task. The effect replicated in a second cross-validation repeat and under permutation importance. About a third of pre-filtered candidates correlated within class with the number of detected genes, a sample-quality measure; a within-class filter removed them without loss of test accuracy (0.943 vs 0.942 balanced accuracy, 1,000 genes). We recommend uncapped trees, or a cap above 75% of the realized median depth, and a within-class quality filter before Boruta.
Deep generative models that can tractably compute input likelihoods, including normalizing flows, often assign unexpectedly high likelihoods to out-of-distribution (OOD) inputs. We mitigate this paradox from an entropic perspective with Semantic Proportional Entropy Manipulation (SPEM), a training-free procedure that applies stronger perturbations to inputs less similar to an in-distribution (ID) memory bank. We derive theoretical lower bounds showing that this entropy gap increases the expected log-likelihood difference in favor of the ID, and experiments on standard benchmarks show that this manipulation realigns the likelihood ordering with human intuition. Strikingly, this realignment persists even when the flow is given only visually meaningless noise instead of the original input, and we analyze this phenomenon theoretically to characterize the conditions under which it occurs.
Explaining opaque Machine Learning (ML) models has become an increasingly important challenge. However, current eXplanation in AI (XAI) methods suffer several shortcomings, including insufficient abstraction, limited user interactivity, and inadequate integration of symbolic knowledge. We propose ReasonX, an explanation tool based on expressions (or, queries) in a closed algebra of operators over theories of linear constraints. ReasonX provides declarative and interactive explanations for decision trees, which may represent the ML models under analysis or serve as global or local surrogate models for tabular-data black-box classifiers. Users can express background or common sense knowledge as linear constraints. This allows for reasoning at multiple levels of abstraction, ranging from fully specified examples to under-specified or partially constrained ones. ReasonX leverages Mixed-Integer Linear Programming (MILP) to reason over the features of factual and contrastive instances. We present here the architecture of ReasonX, which consists of a Python layer, closer to the user, and a Constraint Logic Programming (CLP) layer, which implements a meta-interpreter of the query algebra. The capabilities of ReasonX are demonstrated through qualitative examples, and compared to other XAI tools through quantitative experiments.
Pekka Malo, Lauri Viitasaari, Antti Suominen, Eeva Vilkkumaa, Olli Tahvonen
In many control problems, actions consume a resource budget, such as a wear allowance, and the probability of exhausting it within a given horizon must stay below a tolerance. A simple, widely used approach is to augment the state with the remaining budget, subtract an interior barrier from the utility and penalize exhaustion, so that policy-gradient methods can be applied unchanged. We clip the combined one-step reward from below at a level m to keep the returns of exploratory policies finite. The clip level is a price of failure risk, since the clipped value decomposes exactly into the utility before failure, the cost of the interior margin and m times a geometric failure risk. Consequently, an approximately optimal policy maximizes utility net of margin cost among policies with no greater risk. Its risk exceeds that of any comparison policy by at most order 1/m and can be certified from its value or from validation data. Our main result proves exponential convergence of the population policy-gradient flow for every parameter regularizer that is convex along generalized geodesics and sufficiently strong relative to the price. This extends the mean-field analysis from relative entropy to weight decay and to finite networks. Relative entropy adds parameter noise and a pull toward a prior, whereas weight decay provides only the pull. Action regularization bounds changes in failure risk by the relative entropy to a reference policy. The bounds are evaluated on an exact maintenance benchmark, a fuel-tank controller, a solvable flow and a benchmark-suite task.
Out-of-distribution (OOD) detection is recognized as a critical problem in AI safety. SCALE is an effective post-hoc OOD detection method in the activation-shaping family that requires no training data and rescales penultimate activations using a percentile-based magnitude ratio. However, this ratio is ill-defined when negative activations are present, since positive and negative contributions cancel within the underlying sum, causing SCALE to degenerate on backbones such as ViT. We address this with an Absolute Fix that computes the ratio over absolute-valued activations rather than discarding negative activations through ReLU. Building on this corrected scale, we propose Anti-coherence regularization, which penalizes the scaling factor by the fraction of a sample's max-logit contributions to the max logit that are sign-mismatched and thereby suppresses unreliable evidence for the predicted class. Despite requiring no access to training data or training labels, our method ties or outperforms activation-shaping methods that rely on training data in terms of AUROC and FPR@95 on CIFAR-10 and ImageNet under the OpenOOD v1.5 protocol.
Piping and Instrumentation Diagrams (P&IDs) are the authoritative maps of process plants: isolation, maintenance, and HAZOP decisions depend on what connects to what. Vision-language models describe these sheets fluently, yet they often invent or miss process connections---and an invented or missed link can reverse an isolation or reachability call, so a plant decision cannot trust a fluent answer that was never checked against the linework. We instead recover an explicit graph of the drawing---its symbols, the process connections between them, and the tags that name them---and then require the model to answer only by querying that graph through seven read-only operators, so a topology claim is returned only when it cites the query results that support it. On TopoPID-VQA, a new suite of 3000 topology questions over these sheets, Graph-Grounded Harness (Ours) raises exact match accuracy from 36.7--41.3% under image-only prompting to 74.3--76.0% for Qwen3-VL-4B, Qwen3-VL-8B, and Gemma-4-E4B. It does so on an imperfect substrate: on Digitize-PID dataset the recovered graph scores F1 0.742 on exact process connections, and 0.801 once symbols and tags are pooled in. The residual errors track that gap---grounding pays off where the recovered graph is right, and perception error still breaks topology questions where it is not.
LLM agents are increasingly deployed in multi-agent systems, where they can collude while keeping their actions benign. Output monitors designed to detect such collusions can be fooled by obfuscation and steganography, motivating the use of probes trained on internal activations. However, these probes are usually evaluated on agents that do not know they are being watched. We study how activation-based detection changes when agents are explicitly informed that their internal activations are being monitored, and when they additionally receive feedback from the monitor. We keep the models, probes, and thresholds fixed and change only what the agents are told: nothing (baseline), that an activation monitor is present (aware), or that a monitor is present together with the previous round's score (feedback). We test two games, a four-agent blackjack game and a two-agent Simmons prisoners game, using Qwen3-32B-AWQ and GPT-OSS-20B in homogeneous and mixed pairings. Telling agents about the monitor does not hide them. The best probes stay accurate in all three conditions, and the agents keep colluding.
Deep generative models with tractable likelihoods, exemplified by normalizing flows, offer an effective basis for anomaly detection through likelihood-based scoring. We demonstrate that, unlike in the image domain where deep generative models frequently assign higher likelihoods to anomalous data, such counterintuitive behavior occurs rarely in tabular settings. We first introduce a domain-agnostic formulation that enables consistent detection and evaluation of the counterintuitive phenomenon, addressing the lack of a precise definition in prior work. Through extensive experiments on 47 tabular datasets benchmarked against 13 baseline models and on 10 CV/NLP embedding datasets from ADBench, we show that the phenomenon is consistently rare in general tabular data and that normalizing flow likelihood achieves competitive anomaly detection performance. We further provide theoretical and empirical analyses attributing this disparity to differences in dimensionality and feature correlation between tabular and image data. Therefore, our results indicate that likelihood-based detection with normalizing flows is a practical approach for tabular anomaly detection.
Out-of-distribution (OOD) detection is a critical component for ensuring the reliability of deep neural networks in safety-critical applications. In this work, we present a key empirical observation: for in-distribution (ID) samples, class-wise Mahalanobis distances exhibit a pronounced sharp minimum structure, where the distance to the nearest class is small while distances to all other classes remain large, resulting in high variance across classes. In contrast, OOD samples tend to exhibit a less pronounced sharp minimum structure, producing comparatively lower variance across classes. We further provide a theoretical analysis grounding this observation in Neural Collapse geometry: under relaxed Neural Collapse assumptions on within-class compactness and inter-class separation, ID samples are shown to structurally exhibit high class-wise distance variance, offering a theoretical basis for its use as an OOD score. Motivated by this observation and its theoretical backing, we propose MahaVar, a simple and effective post-hoc OOD detector that augments the Mahalanobis distance with a class-wise distance variance term. Following the OpenOOD v1.5 benchmark protocol, MahaVar achieves state-of-the-art performance on CIFAR-100 and ImageNet, with consistent improvements in both AUROC and FPR@95 over existing Mahalanobis-based methods across all benchmarks.
Object interaction tasks have been a focus of advances in imitation learning. End-to-end methods, dominated by generative variants have shown leaps in performance but tend to be black boxes. Recent indications hint at demonstration lookup as the operating process underneath generative policy performance. In this paper, we revisit a few modern imitation learning benchmarks for object interactions, with the aim of composing a simple framework from first principles. We repurpose theories of manipulation modes and apply them to retrieve and transform demonstration datasets, creating a minimal strategy that preserves performance and robustness while remaining interpretable. We identify pose estimation as a key module that governs information about the object. This informs a manipulation mode-aware retrieval of a demonstration, a mode-aware transformation, a replan step that connects to the retrieval point while preserving mode constraints, returning a transformed demonstration for rollout. These compose four key steps of the proposed RE4 framework, evaluated over state-based and image-based benchmarks in simulated Push-T, Robomimic, and MimicGen as well as a real robot experiment. Robustness is measured in an adversarial study that evaluates sparse data regimes, a study with visual distractors, and the addition of an obstructing collision geometry. RE4 outperforms baselines, while notably preserving interpretability. The results overwhelmingly motivate the benefits of identifying simple interpretable building blocks to learn manipulation skills for object interactions.
Hyunwoo Yu, Jongbeom Lee, Hyunbin Kim, Sungjun Park, Jaemin Choi, Jincheol Yang, Seunghun Moon, Bongjoon Hyun 等 11 位作者
Sparse attention reads only a small subset of the key-value cache, and how to treat the unread tail remains a key challenge. Discarding the tail biases the output, and sampling it as previous approaches do fails at reads of 5% and below, where the tail likely holds heavy hitters the selector missed. Our key observation is that the simplest possible proxy, one token that folds the entire tail into its count-aware mean, already yields lower output error than a sample at every read behind a practical selector. We further show in closed form that at low reads sampling nearly doubles the weight error of discarding while a single proxy stays below it. Building on these findings, we explore approximating the dense attention distribution with a few proxies that target the tail. We propose OpenAttention, a pipeline of clustering, ErrorGate and top-k selection followed by a count-aware mean proxy. To implement clustering and selection both efficiently and effectively, we design training-free Query-Geometry Clustering (QGC) and a Query-Geometry Selector (QGS), which compress the keys while respecting the query directions. Additionally, unlike prior methods that rank clusters by centroid alone, ErrorGate allocates the exact read by the estimated weight error of each cluster. On RULER-32K HARD with Llama-3.1-8B, OpenAttention retains 99% of dense accuracy at a 1% read. It also achieves state-of-the-art accuracy across models, context lengths and datasets. Our customized attention kernel for OpenAttention runs the decode attention stage 10$\times$ faster than dense attention at a 1% read.
Exhaustively evaluating many large language models (LLMs) on a large suite of benchmarks is expensive. We cast benchmarking as finite-population inference and, under a fixed query budget, seek tight confidence intervals (CIs) for model accuracy with asymptotic frequentist coverage. We propose Factorized Active Querying (FAQ), which (a) leverages historical information through a Bayesian factor model; (b) adaptively selects questions using a hybrid variance-reduction/active-learning sampling policy; and (c) maintains validity through Pro-Active Inference---a finite-population extension of active inference (Zrnic & Candès, 2024) that enables direct question selection while preserving asymptotic frequentist coverage. In our experiments, FAQ delivers up to $5\times$ effective sample size gains over strong baselines on two benchmark suites, across varying historical-data missingness levels, with negligible computational overhead: this means that it matches the CI width of uniform sampling while using up to $5\times$ fewer queries. We release our source code and our curated datasets to support reproducibility and future research.
We show that established performance metrics in binary classification, such as Matthews' correlation coefficient (MCC), Cohen's $κ$, the F-score or the Jaccard similarity coefficient are not robust to class imbalance in the sense that if the proportion of the minority class tends to $0$, the true positive rate (TPR) of the Bayes classifier under these metrics {becomes very small or even tends to $0$}. Thus, in imbalanced classification problems, these metrics favour classifiers which ignore the minority class. To alleviate this issue we introduce robustified modifications of the MCC, of Cohen's $κ$ and of the F-score with an additional tuning parameter which allows to adapt the amount of robustness against class imbalance. As theoretical guarantee we show that the Bayes-optimal classifier for these robustified performance metrics, when expressed in terms of the density ratio $f_1/f_0$ of the class-conditional densities $f_i$, has a threshold parameter which is upper-bounded in terms of the tuning parameters. Therefore, even in strongly imbalanced settings, the TPR associated to this classifier will be bounded away from $0${, provided that $f_1/f_0$ exceeds this bound with positive probability under the positive class}. We numerically illustrate the behaviour of the various performance metrics and the effect of the tuning parameters in simulations as well as on a credit default data set. We also discuss connections to the receiver operating characteristic and precision-recall curves, which provide an alternative perspective on the proposed notion of robustness, and give recommendations on how to combine their usage with performance metrics.
Neural operators are widely used as fast surrogates for numerical PDE solvers, mapping input functions to solution functions. However, their generalizability and robustness are not yet clearly defined in the operator-learning setting, which differs from traditional adversarial robustness definitions. This paper studies the generalizability and robustness of a learned neural operator from a solver-integrated perspective, addressing the challenge that the output of a learned operator and a numerical solver tends to change in tandem under input perturbation. First, we formalize the definition of generalization and robustness through a model-solver error operator, identifying fixed-input model-solver loss as generalization metric, and norm-bounded adversarial attack loss increase and Jacobian-error function norm as robustness metric. Second, we identify the solver-integrated adversarial attack as appropriate for PDE operator learning and show why model-only or fixed-ground-truth attacks can be insufficient when the solver output also changes with the input. Third, we develop solver-integrated adversarial training methods for neural operators. Experiments on representative PDE benchmarks show that this solver-integrated adversarial training clearly improves both generalizability and robustness. Deeper solver integration yields more effective attacks, more informative samples, and more efficient training than less integrated alternatives. These results provide a general framework for robust operator training and automatic sample selection without heavy manual intervention. More broadly, the formulation applies to adversarial regression whenever a ground-truth oracle can evaluate, and ideally differentiate, the true input-output map; PDE operator learning is one such case.