Jalaj Bhandari, Dhawal Gupta, Hongli Zhan, Zhenting Wang, Varad Pimpalkhute, Zhengzhong Liu, Eric Xing, Rupesh Srivastava
In this work, we evaluate a simple inference-time scaling approach based on \textit{turn-level best-of-N sampling} (\method) for large language model (LLM) agents on complex, long-horizon tasks. We instantiate this using a \textit{generator-selector} framework with two LLMs; at each turn, a \emph{generator} model generates $N$ candidate actions and a \emph{selector} model selects one of them, which is then executed in the environment. This paper is closely related to the work of \cite{zainullina2025guided} who used critic models for guided search at inference time. Different from their work, we evaluate if off-the-shelf, open-weight LLMs can be effective at comparing different candidate actions directly. Experiments on long-horizon terminal tasks show that \method can lead to significant performance improvements, both in terms of accuracy (5--10\%) and consistency across repeated attempts. Analysis of agent trajectories reveal a shift in agent behaviour towards desirable patterns such as reduction in number of attempts to solve the task, improved task instruction following, debugging and testing.
Zhong Li, Qi Huang, Yuxuan Zhu, Mohammad Mohammadi Amiri, Niki van Stein, Thomas Bäck, Matthijs van Leeuwen, Zaiwen Wen 等 9 位作者
Optimization modeling translates real decision-making problems into mathematical optimization models and solver-executable implementations. Although language models are increasingly used to generate optimization formulations and solver code, existing benchmarks for this capability are almost entirely text-only. This omits many optimization-modeling tasks that arise in operational practice, where requirements are described in text but essential instance information is often conveyed through visual artifacts such as tables, graphs, maps, schedules, and planning dashboards. We introduce multimodal optimization modeling, a benchmark setting where models must construct both a mathematical formulation and executable solver code from a text-and-visual problem specification. To evaluate this setting, we develop a solver-grounded framework that generates structured optimization instances, verifies each instance with an exact solver, and builds both model-facing inputs and hidden reference files from the same verified source. We instantiate it as MM-OptBench, a benchmark of 780 solver-verified instances spanning 6 optimization families, 26 subcategories, and 3 structural difficulty levels. We evaluate 9 multimodal large language models (MLLMs), including 6 frontier general-purpose and 3 math-specialized models, with aggregate, family-level, difficulty-level, and failure-mode analyses. The task remains far from solved: the best two models reach 52.1% and 51.3% pass@1, while on average across the six general-purpose MLLMs, pass@1 is 43.4% on easy and 15.9% on hard instances. All three math-specialized MLLMs solve 0/780 instances. Post-hoc extraction diagnostics and oracle-reading evaluation reveal difficulties in both multimodal grounding and downstream solver-artifact generation. MM-OptBench provides a concrete testbed for solver-grounded, decision-oriented multimodal intelligence.
Research taste, the judgment of what to read, which problems to pursue, and which methods to try, increasingly shapes the work of AI scientists, yet it remains far less understood than their ability to execute research. We introduce IdeaTaste, a benchmark that scores an LLM by how closely the research proposals it writes from a literature snapshot frozen at 2025 match the work the same communities published in 2026. Our experiments span six contemporary language models and four research areas from established to emerging fields, covering more than 90{,}000 proposals in total. LLM agents fall well short of human communities in both the literature they cite and the ideas they state, reaching about 26\% of the human level on citation matching and only 18\% on idea matching. Their populations can cite as diversely as a community and take up its tasks and methods, yet rarely arrive at the concrete approaches and research plans it pursues. Neither richer field context nor larger populations substantially close this gap. In blinded expert review, AI proposals trail the 2026 papers least in presentation and most in expected success, echoing their gap in concrete approaches, and those whose ideas no human paper shares fare worst, suggesting weaker directions rather than overlooked opportunities. These results indicate that the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue. We release IdeaTaste as an open, continuously updated leaderboard (https://cxcscmu.github.io/IdeaTaste) that benchmarks contemporary AI proposals against future publication cycles.
Lincen Yang, Qi Huang, Niki van Stein, Thomas Bäck, Matthijs van Leeuwen, Zhong Li
Subgroup discovery aims to identify regions of the feature space where a target variable deviates from its marginal distribution. A fundamental challenge in this problem is the trade-off between a subgroup's deviation and its support: small subgroups can be highly deviating but statistically meaningless, while large ones cannot deviate much by construction. Existing methods tackle this challenge by collapsing the two into a single scalar quality measure, implicitly committing to one particular trade-off and making it difficult to recover the most deviating subgroup under a user-specified support constraint. We instead provide the first theoretical and structural characterization of optimal subgroups with maximum Kullback-Leibler (KL) divergence under arbitrary support constraints. Under a partition-based generative model, where the feature space decomposes into regions of homogeneous conditional distributions of the target variable called atoms, we prove that every point on the deviation-support Pareto front is a union of whole atoms together with at most one fractional atom. This reduces an uncountable search to a finite combinatorial problem. Guided by this theory, we propose a simple two-step recipe: learn a homogeneous partition, then select atoms greedily. Across 19 benchmark datasets, it finds the most deviating subgroups at a range of support thresholds, and reliably improves subgroups returned by existing methods when applied as a post-processing step. These results empirically validate the sufficiency and necessity of our theoretical contributions.
In physical backpropagation, users return loss derivatives over the time-division duplex uplink, giving the base station the downlink gradient without channel reconstruction. Reciprocity calibration conventionally prioritizes base-station chains, as column-normalized zero-forcing reduces a per-user factor to a stream rotation. This priority can reverse. With ideal base-station chains and no noise, per-user amplitude errors make the returned field a weighted sum-rate gradient. With zero-forcing full service at both optima, the loss is independent of array size. Otherwise the selection penalty, serving the wrong users, carries 86 to 94 percent of the amplitude-induced loss in five of six configurations. Per-user phase errors mainly slow the iteration. Base-station chain errors scale the returned field per antenna, and at 3 dB and 25 degrees they cost one user 1.3 dB of signal-to-noise ratio (SNR) that no array size removes. With 80 users, such errors on both sides, and mismatched pilots, oracle per-user and base-station calibration leave 6.2 and 12.3 percent loss at 256 antennas on clustered channels. This ordering holds at 30 dB SNR and reverses for 1 dB, 10 degree user errors. A capped one-echo estimator comes within 2.4 percentage points of oracle per-user calibration but underperforms no calibration at 10 dB SNR.
Over-the-air backpropagation returns a loss gradient through the reciprocal channel, which requires the return port to see the channel of the forward port. If the terminal moves in between, a fluid antenna system can return through the port nearest to the forward position. In ray-traced Chicago streets, with a pedestrian displacement of about half a wavelength and a street-level base station, this rule reverses 20 percent of the gradients at a port spacing of 0.7 wavelengths and 49 percent at 0.9 wavelengths, while one of the four nearest ports stays aligned in all but 2 and 9 percent. We derive a certificate for the sign of the mean correlation from the second moment of a planar angular spectrum. It reduces to a linear program with three variables and infinitely many constraints, is tight, and holds over a region of second moments. In a static Gaussian model, observations collected along a line cannot tell an angular spectrum from its mirror image, so the sign of the cross moment must come from other information. At unsurveyed positions of the same city, a knowledge map that transports surveyed paths through virtual sources and fits them to the causal channel history lowers the reversal rate at 0.9 wavelengths from 49 to 27 percent. Its gain over the nearest port survived every tested error, while its gain over a geometry rule stayed significant at the tested single-error levels of 5 degrees of phase drift per step and 2 mm of position error but not at the next tested levels. Without any survey, across five cities and ten base stations, the nearest port reverses 37 percent of the returns at 0.9 wavelengths and a rule based on the direction and elevation of the base station 20.5 percent. The multilayer perceptrons we tested did not beat these physical rules by the margin fixed in advance in the primary settings.
In reciprocity-based over-the-air split learning, channel aging rotates and attenuates the derivative signal returned to the transmitter. Seven case studies combine numerical forecasts, mechanism checks, and competing predictions. Near the correlation zero, an archived SGD comparison finds truncated inversion 0.000813 lower in cross-entropy than polar correction, with a paired 95% interval for the difference from -0.001853 to 0.000060, and a retained-mode quadratic proxy slightly favors truncation. Selecting the return port with the largest received power falls short of the predicted five-point alignment advantage over random selection at both tested spacings, gaining at most 0.97 points, while the local inverse relation between amplitude and phase gradient is supported. Seven specified interference-induced accuracy-collapse predictions are rejected, and independent zero-mean interference preserves the conditional mean under a fixed linear chain. Per-subcarrier correction and spherical-wavefront compensation do not deliver their predicted accuracy gains. Focusing leaves both methods near chance in a single-mode model and adds no observed gain in a model with a line-of-sight component plus scattering. A successful port assignment under nonsymmetric motion refutes an exact-symmetry requirement. The competing positioning-error prediction is supported. Gaussian rail jitter with a standard deviation of 0.383 wavelengths yields high terminal test accuracy, whereas a fixed bias of the same magnitude yields near-chance accuracy. These cases distinguish valid local physical relations from unsupported training forecasts through task support, noise injection, and update moments.
A Gaussian random wave field is sampled on a lattice within a bounded aperture. After displacement, re-indexing leaves a residual in the Voronoi cell. We study whether the empirical lag correlation has positive real part throughout the cell. For each realization in a single-output reciprocal link with noiseless over-the-air return, this is equivalent to the negative returned gradient being a descent direction for all residuals and differentiable losses with nonzero error signals. Return noise is treated exactly. For isotropic fields, ball-shaped apertures, unshifted lattice points, and dilations approaching the critical covering radius, every minimizer is a deep hole except with exponentially small probability. A Gaussian vector on the deep holes determines the limiting failure probability, with explicit covariance and lattice-independent limiting normalization. Square, hexagonal, simple cubic, and body-centered cubic lattices have two, three, four, and twelve sign classes. At fixed target failure probabilities below the critical limiting value, this covariance determines the exact leading aperture constant. When the margin and target failure probability vanish jointly, that constant is common to lattice shapes within each dimension. A cubic correction yields vanishing relative tail error in a moderate-deviation regime. A non-asymptotic two-sided aperture law has explicit constants. With a Rayleigh-faded specular path and positive diffuse margin, a failure floor occurs at every positive path power exactly when the cosine of its phase change is negative somewhere in the cell. The floor vanishes for all directions exactly when the covering radius is at most a quarter wavelength.
In over-the-air split learning over an aging single-antenna Rayleigh channel, the mean returned gradient equals the antenna-domain gradient scaled by the temporal correlation. Uncorrected training collapses in negative-correlation lobes. Near a correlation zero, no nonzero scalar calibration improves the ratio of mean power to innovation power for an individual transmission, and, on the tested grid, using sixteen times as many pilots recovers only 7 percent of the training gap. Returning at the first correlation extremum with phase correction, 1.3 ms later at 3.5 GHz and 15 m/s, achieves 90.2 percent accuracy, compared with 91.8 percent with CSI. In the tested line-of-sight setting, the near-zero gap becomes negligible.
Over-the-air split learning returns the receiver's loss gradient through the reciprocal channel, so terminal motion between the forward and backward transmissions corrupts the backward pass. We repair the aged return at a fixed receive array using only second-order statistics. Two numbers taken from the forward-backward cross-covariance, the alignment reachable by the best common phase and by the best unitary correction, bound the main term of every repair. In the 3GPP tapped-delay-line channels TDL-A/B/C, where every antenna ages by the same scalar, the two numbers coincide, and at a correlation zero all tested fixed corrections stay between 58% and 62% accuracy, whereas a return delayed to the next correlation lobe reaches 98.4%. In 3GPP clustered-delay-line channels a statistical unitary correction stays within 0.02 in mean cross-entropy of ideal channel knowledge in all six tested profile and speed combinations, while common-phase correction falls to 69% for CDL-A at 3 km/h and the tested delay loses up to 10 points. In a two-cluster model spatial correction is decisive only where the common-phase alignment nearly vanishes (64% to 95%). With a ResNet-18 on CIFAR-10, only the delayed return trains at a separable correlation zero (86% against at most 16%), and at 30 km/h in a clustered-delay-line channel the unitary correction reaches 80% where the common phase reaches 60%. Given geometry, displacement and angular statistics, a port permutation chosen by linear assignment reaches 99.5-99.7% accuracy for hexagonal and square arrays in oblique motion without pilots or digital feedback during training, where nearest-position matching drops to 89%.
Yuheng Zha, Yilei Wang, Qiyue Gao, Junrong Chen, Yujia Wu, Zhengfeng Lai, Zhengzhong Liu, Eric P. Xing
Humans often solve spatial problems by mentally simulating visual transformations. In contrast, conventional vision-language models (VLMs) reason primarily through language. We investigate whether VLMs can solve spatial problems by reasoning with both text and generated visual states. To this end, we introduce WM-VLM, which equips a pretrained VLM with a lightweight world model branch for generating intermediate visual states. Our two-stage training first teaches the model to generate the next visual state and then to use that state for reasoning. We programmatically construct spatial reasoning tasks with verifiable intermediate visual states. These tasks allow us to evaluate how well the model generates visual states and how much it relies on them to answer the question. On 2D and 3D mental rotation tasks, WM-VLM consistently outperforms the supervised fine-tuned backbone, with gains of up to 39.25 percentage points. Ablations suggest that these gains depend on the generated visual states, as removing or corrupting them sharply reduces performance. Together, these results suggest that internal world models offer a promising path toward VLMs that reason in both language and visual space. Code is publicly available at: https://github.com/yuh-zha/WM-VLM
Kejuan Yang, Yizhuo Zhang, Mingyuan Du, Yue Zhang, Dixin Zheng, Kaili Zhao, Yang Xiao, Hanzhong Liang 等 9 位作者
Global-scale video moderation faces a dual challenge: the need for fine-grained multimodal reasoning and the demand for interpretable outputs to support downstream enforcement. Traditional moderation systems often rely on fragmented black-box classifiers that are difficult to maintain and lack transparency.In this paper, we present UNIVID, a Unified Vision-Language model for Video Moderation. Unlike standard classification models, UNIVID generates policy-aware captions that serve as an interpretable intermediate representation, enabling human-verifiable decisions and multi-task reusability. While existing open-source and commercial VLMs often suffer from safety-guardrail refusals and lack fine-grained policy alignment, we develop a specialized training data recipe that combines expert human-refined labels with synthetic data to align the model with our safety guidelines.By integrating UNIVID as the core captioner, we design a novel end-to-end video moderation system that reduces violation leakage by 42.7% and overkill rate by 37.0% relatively. Meanwhile, by replacing over 1,000 policy-specific models with a single UNIVID backbone, we recycle extensive computational resources while significantly reducing engineering maintenance overhead. To our knowledge, this is one of the first reports of a high-efficiency captioning VLM successfully supporting industrial-scale moderation and cross-functional business.
尚无 AI 解读，展示原文摘要
cs.CL · cs.CV
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track) · 2026 · 正式发表
Yanbin Yin, Kun Zhou University of California, San Diego, Zhen Wang University of California, San Diego, Xiangdong Zhang New York University, Yifei Shao, Shibo Hao University of California, San Diego, Yi Gu University of California, San Diego, Jieyuan Liu 等 14 位作者
The recent explosion of large language models (LLMs), each with its own general or specialized strengths, makes scalable, reliable benchmarking more urgent than ever. Standard practices nowadays face fundamental trade-offs: closed-ended question-based benchmarks (MMLU) struggle with saturation as newer models emerge, while crowd-sourced leaderboards (Chatbot Arena) rely on costly and slow human judges. Recently, automated methods (LLM-as-a-judge) shed light on the scalability, but risk bias by relying on one or a few “authority” models. To tackle these issues, we propose Decentralized Arena (), a fully automated framework leveraging collective intelligence from all LLMs to evaluate each other. It mitigates single-model judge bias by democratic, pairwise evaluation, and remains efficient at scale through two key components: (1) a coarse-to-fine ranking algorithm for fast incremental insertion of new models with sub-quadratic complexity, and (2) an automatic question selection strategy for the construction of new evaluation dimensions. Across extensive experiments across 66 LLMs, attains up to 97% correlation with human judgements, while significantly reducing the cost.
尚无 AI 解读，展示原文摘要
cs.CL
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2026 · 正式发表
Weikang Yuan, Kaisong Song, Zhuoren Jiang Zhejiang University, Junjie Cao, Yujie Zhang, Jun Lin, Kun Kuang Zhejiang University, Ji Zhang Alibaba Group 等 9 位作者
Legal consultation is essential for safeguarding individual rights and ensuring access to justice, yet remains costly and inaccessible to many individuals due to the shortage of professionals. While recent advances in Large Language Models (LLMs) offer a promising path toward scalable, low-cost legal assistance, current systems fall short in handling the interactive and knowledge-intensive nature of real-world consultations. To address these challenges, we introduce LeCoDe, a multi-turn benchmark dataset constructed from publicly available real-world legal consultation content and carefully processed into a de-identified, structured research resource for evaluating and advancing research on LLMs in legal consultation settings. LeCoDe contains 3,696 multi-turn consultation cases with 110,008 dialogue turns. The dataset is further enriched through expert annotation, including key facts, fact importance, and advice summaries. Furthermore, we propose a comprehensive evaluation framework that assesses LLMs’ consultation capabilities in terms of (1) clarification capability and (2) professional advice quality. This unified framework incorporates 12 metrics across two dimensions. Through extensive experiments on various general and domain-specific LLMs, our results reveal significant challenges in this task, with even state-of-the-art models like GPT-4 achieving only 35.9% recall for clarification and 59.1% overall score for advice quality, highlighting the complexity of professional consultation scenarios. Based on these findings, we further explore several strategies to enhance LLMs’ legal consultation abilities. Our benchmark contributes to advancing research in legal domain dialogue systems, particularly in simulating more real-world user-expert interactions. The resource is available at https://github.com/PiLab-ZJU/LeCoDe.
尚无 AI 解读，展示原文摘要
cs.CL
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2026 · 正式发表
Yang Wu Axon, Jinhong Yu Worcester Polytechnic Institute, Jingwei Xiong, Zhimin Tao Jiangsu university, Xiaozhong Liu Worcester Polytechnic Institute
The integration of Large Language Models (LLMs) into scientific workflows presents exciting opportunities to accelerate biomedical discovery. However, the reactive nature of LLMs, which respond only when prompted, limits their effectiveness in collaborative settings that demand foresight and autonomous engagement. In this study, we introduce CoLabScience, a proactive LLM assistant designed to enhance biomedical collaboration between AI systems and human experts through timely, context-aware interventions. At the core of our method is PULI (Positive-Unlabeled Learning-to-Intervene), a novel framework trained with a reinforcement learning objective to determine when and how to intervene in streaming scientific discussions, by leveraging the team’s project proposal and long- and short-term conversational memory. To support this work, we introduce BSDD (Biomedical Streaming Dialogue Dataset), a new benchmark of simulated research discussion dialogues with intervention points derived from PubMed articles. Experimental results show that PULI significantly outperforms existing baselines in both intervention precision and collaborative task utility, highlighting the potential of proactive LLMs as intelligent scientific assistants.
尚无 AI 解读，展示原文摘要
cs.CL
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2026 · 正式发表
Zhiyuan Yu, Shijian Xiao, Cam-Tu Nguyen, Zhangyue Yin, Lekai Xing, Wenzhong Li Nanjing University, Sanglu Lu Nanjing University
Improving the exploration of reasoning is essential for advancing Large Language Models’ (LLMs) problem-solving performance. Current methods primarily rely on output-level stochasticity, which decode within fixed reasoning patterns of LLM and suffer from insufficient exploration. In this paper, we introduce adjusting attention temperature to directly modulate the model’s internal focus during reasoning, which enables a dynamic shift between exploratory and focused processing. We reveal that moderate adjustments preserve LLM’s reasoning capability while producing problem hardness-dependent benefits: higher temperatures facilitate solving complex tasks by encouraging wider exploration, whereas lower temperatures mitigate overthinking on simpler problems. Leveraging this insight, we propose a two-stage inference strategy: first, attention temperature scaling modulates the LLM’s reasoning patterns to diversify the reasoning traces; then, a difficulty-aware aggregation scheme is introduced to effectively identify the most reliable solution from the generated candidates. Extensive evaluations show that our method improves Pass@10 by 6.78–14.20% and aggregation accuracy by 9.74% across 7 reasoning benchmarks.
尚无 AI 解读，展示原文摘要
cs.CL
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2026 · 正式发表
Yiqing Zhang, Xiaozhong Liu Worcester Polytechnic Institute, Fabricio Murai Worcester Polytechnic Institute
Trustworthy biomedical question answering (QA) systems must not only provide accurate answers but also justify them with current, verifiable evidence. Retrieval-augmented approaches partially address this gap but lack mechanisms to iteratively refine poor queries, whereas self-reflection methods kick in only after full retrieval is completed. In this context, we introduce PubMed Reasoner, a biomedical QA agent composed of three stages: self-critic query refinement evaluates MeSH terms for coverage, alignment, and redundancy to enhance PubMed queries based on partial (metadata) retrieval; reflective retrieval processes articles in batches until sufficient evidence is gathered; and evidence-grounded response generation produces answers with explicit citations. PubMed Reasoner with a GPT-4o backbone achieves 78.32% accuracy on PubMedQA, slightly surpassing human experts, and showing consistent gains on MMLU Clinical Knowledge. Moreover, LLM-as-judge evaluations prefer our responses across: reasoning soundness, evidence grounding, clinical relevance, and trustworthiness. By orchestrating retrieval-first reasoning over authoritative sources, our approach provides practical assistance to clinicians and biomedical researchers while controlling compute and token costs.
尚无 AI 解读，展示原文摘要
cs.CL
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2026 · 正式发表