Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a promising approach for enhancing the reasoning capabilities of large language models. However, it still remains unclear whether RLVR can extend reasoning capabilities beyond what is already encoded in the base model. In this paper, we study this question through the lens of primacy bias: the tendency of models to over-rely on already-learned representations, disproportionately improving performance on problems they already handle well while underexploring harder ones that require departing from those representations. Through our controlled experiments and large-scale analysis, we show that RLVR training on mixed data tends to further sharpen performance on easier problems while yielding only marginal gains on harder ones. We find that although hard problems lead to modest gains overall, they are actually the ones that help most in expanding the model's reasoning capabilities and pushing the representations. However, prolonged training on these hard problems may also hurt and lead to forgetting on easy problems. These findings help to better understand some of the prior conflicting observations about RLVR, specifically, why on (OOD) domains where the base model performs poorly, improvements on hard problems can expand reasoning; while on well-trained domains, forgetting can reduce performance and mask genuine gains obtained from hard problems.
Interactive LLM agents increasingly work with users across tasks, yet remembering a previous correction does not ensure that the agent changes its behavior in the future. In a diagnostic constructed from anonymized real-world agent interactions, agents using Mem0 still violate 57.5% of applicable preference checks. We introduce TellOnce, a drop-in personalization layer that turns conversational corrections into persistent, scoped, and revisable behavioral rules without updating the underlying agent model. TellOnce identifies which requirements should persist, determines when they apply, reconciles subsequent feedback with existing rules, and prepares reusable checks when reliable verification is possible. We evaluate whether learned corrections improve future behavior by freezing the learned state and testing on subsequent tasks without additional user corrections. Across three models on ClawArena and MemoryArena, TellOnce achieves the lowest preference violation rate in all six settings, reducing violations relative to PGMem by 19.3-42.3 percentage points on ClawArena and 3.0-22.9 points on MemoryArena, while maintaining broadly comparable task performance. On SWE-Together, TellOnce further improves task quality on new coding tasks by learning from feedback received during earlier tasks. Experiment code is available at https://github.com/YujunZhou/TellOnce_exp, and the deployable skill at https://github.com/YujunZhou/tellonce.
Mateusz Bystroński, Doheon Han University of Notre Dame, Nitesh V. Chawla University of Notre Dame, Tomasz Jan Kajdanowicz Wroclaw University of Science and Technology
Starting from the observation that conditioning a poetry-writing prompt with a pancake recipe leads an LLM to produce a coherent poem incorporating pancake-related content and, more broadly, that such contexts arrange themselves into a structured semantic vector space, we argue that this renders the space explorable. By sampling it and using the resulting continuous representations to condition an LLM’s generation distribution, we can systematically expand the model’s reachable semantic range.We introduce a framework that requires no modification of LLM parameters and operationalizes this idea by constructing a conditioning distribution from a small set of diverse anchor generations. This distribution conditions LLM’s generation via an xRAG-style projector.Our experiments demonstrate that this manifold-based conditioning substantially increases generative diversity, with direct benefits for enhancing divergent thinking, a core facet of creativity, in language models.
尚无 AI 解读，展示原文摘要
cs.CL
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop) · 2026 · 正式发表