跳到正文
研踪PaperTrail

论文库

检索计算机领域论文,按研究方向、作者或关键词筛选。

部分采集异常
正在读取账号与研究记录…

全站更新概况

+479新增论文
+361其中 AI 相关
+63已有论文修订
+93新增 AI 导读

arXiv 最近完成扫描:2026-11-16 01:34 · 全站最近入库:2026-11-16 13:44各信源进度 →

统计口径与数据新鲜度

统计 2026-11-15 16:18 — 2026-11-16 16:18(北京时间) · 点击数字查看对应简报;AI 相关包含在新增论文内。

最近完成检查:2026-11-16 01:34最近发现新增版本:2026-11-16 13:44库内论文最新更新时间:2026-11-14 02:59

每分钟刷新已保存的数据。空检查不代表采集异常;新收录可能包含历史论文。AI 导读按新增版本计数。

23,827 已收录论文19,941 AI 相关6,921 有修订112 HF 社区发现

浏览论文库

时间范围跟随排序:版本更新、本站收录或首次发表。最新收录可能包含历史论文。部分信源异常,查看状态 →

找到 23,827 篇论文 · 按版本更新时间排序上轮扫描起点 2026-11-16 01:33(北京时间)
版本更新 · v2arXiv:2106.09685Google Research

LoRA: Low-Rank Adaptation of Large Language Models

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen

原文摘要

An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA.

cs.CL · cs.AI · cs.LG

新论文 · v1arXiv:2110.06169Berkeley BAIR

基于隐式Q学习的离线强化学习

Offline Reinforcement Learning with Implicit Q-Learning

Ilya Kostrikov, Ashvin Nair, Sergey Levine

AI 速读 基于摘要

提出隐式Q学习(IQL),通过状态条件上期望分位数估计最优动作价值,无需查询数据集外动作即可改进策略。

cs.LG

查看原文摘要

Offline reinforcement learning requires reconciling two conflicting aims: learning a policy that improves over the behavior policy that collected the dataset, while at the same time minimizing the deviation from the behavior policy so as to avoid errors due to distributional shift. This trade-off is critical, because most current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy, and therefore need to either constrain these actions to be in-distribution, or else regularize their values. We propose an offline RL method that never needs to evaluate actions outside of the dataset, but still enables the learned policy to improve substantially over the best behavior in the data through generalization. The main insight in our work is that, instead of evaluating unseen actions from the latest policy, we can approximate the policy improvement step implicitly by treating the state value function as a random variable, with randomness determined by the action (while still integrating over the dynamics to avoid excessive optimism), and then taking a state conditional upper expectile of this random variable to estimate the value of the best actions in that state. This leverages the generalization capacity of the function approximator to estimate the value of the best available action at a given state without ever directly querying a Q-function with this unseen action. Our algorithm alternates between fitting this upper expectile value function and backing it up into a Q-function. Then, we extract the policy via advantage-weighted behavioral cloning. We dub our method implicit Q-learning (IQL). IQL demonstrates the state-of-the-art performance on D4RL, a standard benchmark for offline reinforcement learning. We also demonstrate that IQL achieves strong performance fine-tuning using online interaction after offline initialization.

版本更新 · v2arXiv:2103.14030Google Research

Swin Transformer:使用移位窗口的层次化视觉Transformer

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo

AI 速读 基于摘要

提出一种以移位窗口计算表示的层次化视觉Transformer,可作为通用视觉骨干网络,并在分类、检测和分割任务上取得作者报告的良好结果。

cs.CV · cs.LG

查看原文摘要

This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with \textbf{S}hifted \textbf{win}dows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures. The code and models are publicly available at~\url{https://github.com/microsoft/Swin-Transformer}.

版本更新 · v2arXiv:2102.01330Google Research

AI4VIS: Survey on Artificial Intelligence Approaches for Data Visualization

Aoyu Wu, Yun Wang, Xinhuan Shu, Dominik Moritz, Weiwei Cui, Haidong Zhang, Dongmei Zhang, Huamin Qu

原文摘要

Visualizations themselves have become a data format. Akin to other data formats such as text and images, visualizations are increasingly created, stored, shared, and (re-)used with artificial intelligence (AI) techniques. In this survey, we probe the underlying vision of formalizing visualizations as an emerging data format and review the recent advance in applying AI techniques to visualization data (AI4VIS). We define visualization data as the digital representations of visualizations in computers and focus on data visualization (e.g., charts and infographics). We build our survey upon a corpus spanning ten different fields in computer science with an eye toward identifying important common interests. Our resulting taxonomy is organized around WHAT is visualization data and its representation, WHY and HOW to apply AI to visualization data. We highlight a set of common tasks that researchers apply to the visualization data and present a detailed discussion of AI approaches developed to accomplish those tasks. Drawing upon our literature review, we discuss several important research questions surrounding the management and exploitation of visualization data, as well as the role of AI in support of those processes. We make the list of surveyed papers and related material available online at ai4vis.github.io.

cs.HC · cs.GR

版本更新 · v2arXiv:2107.03374Google Research

Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda 等 58 位作者

原文摘要

We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.

cs.LG

新论文 · v1arXiv:2107.03312Google Research

SoundStream: An End-to-End Neural Audio Codec

Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, Marco Tagliasacchi

原文摘要

We present SoundStream, a novel neural audio codec that can efficiently compress speech, music and general audio at bitrates normally targeted by speech-tailored codecs. SoundStream relies on a model architecture composed by a fully convolutional encoder/decoder network and a residual vector quantizer, which are trained jointly end-to-end. Training leverages recent advances in text-to-speech and speech enhancement, which combine adversarial and reconstruction losses to allow the generation of high-quality audio content from quantized embeddings. By training with structured dropout applied to quantizer layers, a single model can operate across variable bitrates from 3kbps to 18kbps, with a negligible quality loss when compared with models trained at fixed bitrates. In addition, the model is amenable to a low latency implementation, which supports streamable inference and runs in real time on a smartphone CPU. In subjective evaluations using audio at 24kHz sampling rate, SoundStream at 3kbps outperforms Opus at 12kbps and approaches EVS at 9.6kbps. Moreover, we are able to perform joint compression and enhancement either at the encoder or at the decoder side with no additional latency, which we demonstrate through background noise suppression for speech.

cs.SD · cs.LG · eess.AS

版本更新 · v3arXiv:1912.04977Google Research

联邦学习的进展与开放问题

Advances and Open Problems in Federated Learning

Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles 等 59 位作者

AI 速读 基于摘要

一篇综述性论文,梳理联邦学习的研究进展并系统列出开放问题与挑战。

cs.LG · cs.CR · stat.ML

查看原文摘要

Federated learning (FL) is a machine learning setting where many clients (e.g. mobile devices or whole organizations) collaboratively train a model under the orchestration of a central server (e.g. service provider), while keeping the training data decentralized. FL embodies the principles of focused data collection and minimization, and can mitigate many of the systemic privacy risks and costs resulting from traditional, centralized machine learning and data science approaches. Motivated by the explosive growth in FL research, this paper discusses recent advances and presents an extensive collection of open problems and challenges.

新论文 · v1arXiv:2103.00564Google Research

An Introduction to Johnson-Lindenstrauss Transforms

Casper Benjamin Freksen

原文摘要

Johnson--Lindenstrauss Transforms are powerful tools for reducing the dimensionality of data while preserving key characteristics of that data, and they have found use in many fields from machine learning to differential privacy and more. This note explains what they are; it gives an overview of their use and their development since they were introduced in the 1980s; and it provides many references should the reader wish to explore these topics more deeply.

cs.DS · cs.LG

版本更新 · v3arXiv:1905.10615Berkeley BAIR

对抗策略:攻击深度强化学习

Adversarial Policies: Attacking Deep Reinforcement Learning

Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, Stuart Russell

AI 速读 基于摘要

作者展示在零和博弈中,仅通过选择对抗性策略制造自然观测,就能稳定击败经自博弈训练、对对手鲁棒的深度强化学习智能体。

cs.LG · cs.AI · cs.CR · stat.ML

查看原文摘要

Deep reinforcement learning (RL) policies are known to be vulnerable to adversarial perturbations to their observations, similar to adversarial examples for classifiers. However, an attacker is not usually able to directly modify another agent's observations. This might lead one to wonder: is it possible to attack an RL agent simply by choosing an adversarial policy acting in a multi-agent environment so as to create natural observations that are adversarial? We demonstrate the existence of adversarial policies in zero-sum games between simulated humanoid robots with proprioceptive observations, against state-of-the-art victims trained via self-play to be robust to opponents. The adversarial policies reliably win against the victims but generate seemingly random and uncoordinated behavior. We find that these policies are more successful in high-dimensional environments, and induce substantially different activations in the victim policy network than when the victim plays against a normal opponent. Videos are available at https://adversarialpolicies.github.io/.

版本更新 · v3arXiv:1908.07125Berkeley BAIR

用于攻击和分析NLP的通用对抗触发器

Universal Adversarial Triggers for Attacking and Analyzing NLP

Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, Sameer Singh

AI 速读 基于摘要

提出与输入无关的通用对抗触发器,通过梯度引导搜索短token序列,能触发特定预测并迁移到其他模型,同时用于分析模型全局行为。

cs.CL · cs.LG

查看原文摘要

Adversarial examples highlight model vulnerabilities and are useful for evaluation and interpretation. We define universal adversarial triggers: input-agnostic sequences of tokens that trigger a model to produce a specific prediction when concatenated to any input from a dataset. We propose a gradient-guided search over tokens which finds short trigger sequences (e.g., one word for classification and four words for language modeling) that successfully trigger the target prediction. For example, triggers cause SNLI entailment accuracy to drop from 89.94% to 0.55%, 72% of "why" questions in SQuAD to be answered "to kill american people", and the GPT-2 language model to spew racist output even when conditioned on non-racial contexts. Furthermore, although the triggers are optimized using white-box access to a specific model, they transfer to other models for all tasks we consider. Finally, since triggers are input-agnostic, they provide an analysis of global model behavior. For instance, they confirm that SNLI models exploit dataset biases and help to diagnose heuristics learned by reading comprehension models.

新论文 · v1arXiv:2009.13081Google Research

What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams

Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, Peter Szolovits

原文摘要

Open domain question answering (OpenQA) tasks have been recently attracting more and more attention from the natural language processing (NLP) community. In this work, we present the first free-form multiple-choice OpenQA dataset for solving medical problems, MedQA, collected from the professional medical board exams. It covers three languages: English, simplified Chinese, and traditional Chinese, and contains 12,723, 34,251, and 14,123 questions for the three languages, respectively. We implement both rule-based and popular neural methods by sequentially combining a document retriever and a machine comprehension model. Through experiments, we find that even the current best method can only achieve 36.7\%, 42.0\%, and 70.1\% of test accuracy on the English, traditional Chinese, and simplified Chinese questions, respectively. We expect MedQA to present great challenges to existing OpenQA systems and hope that it can serve as a platform to promote much stronger OpenQA models from the NLP community in the future.

cs.CL · cs.AI

版本更新 · v4arXiv:2005.14165Google Research

Language Models are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam 等 31 位作者

原文摘要

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

cs.CL

新论文 · v1arXiv:2003.04974Google Research

Transformer++

Prakhar Thapak, Prodip Hore

原文摘要

Recent advancements in attention mechanisms have replaced recurrent neural networks and its variants for machine translation tasks. Transformer using attention mechanism solely achieved state-of-the-art results in sequence modeling. Neural machine translation based on the attention mechanism is parallelizable and addresses the problem of handling long-range dependencies among words in sentences more effectively than recurrent neural networks. One of the key concepts in attention is to learn three matrices, query, key, and value, where global dependencies among words are learned through linearly projecting word embeddings through these matrices. Multiple query, key, value matrices can be learned simultaneously focusing on a different subspace of the embedded dimension, which is called multi-head in Transformer. We argue that certain dependencies among words could be learned better through an intermediate context than directly modeling word-word dependencies. This could happen due to the nature of certain dependencies or lack of patterns that lend them difficult to be modeled globally using multi-head self-attention. In this work, we propose a new way of learning dependencies through a context in multi-head using convolution. This new form of multi-head attention along with the traditional form achieves better results than Transformer on the WMT 2014 English-to-German and English-to-French translation tasks. We also introduce a framework to learn POS tagging and NER information during the training of encoder which further improves results achieving a new state-of-the-art of 32.1 BLEU, better than existing best by 1.4 BLEU, on the WMT 2014 English-to-German and 44.6 BLEU, better than existing best by 1.1 BLEU, on the WMT 2014 English-to-French translation tasks. We call this Transformer++.

cs.CL · cs.LG · stat.ML

首次观测 · 记录 #1软件学报:jos:5618软件学报

一种基于协同上下文关系学习的同城活动推荐算法

赖奕安,张玉洁,杜雨露,孟祥武

原文摘要

新兴的基于活动的社交网络以活动为核心,结合线上关系与线下活动促进用户真实、有效的社交关系的形成,但过多的活动信息会使用户难以分辨和选择.结合上下文进行个性化同城活动推荐,是解决活动信息过载问题的一种有效手段.然而大部分现有的同城活动推荐算法都是从用户参与活动记录中间接统计用户对上下文信息的偏好,忽略了两者之间潜在的交叉影响关系,从而影响了推荐结果的有效性.为了解决用户参与活动偏好与上下文信息潜在交叉影响关系利用不足的问题,提出了一种基于协同上下文关系学习的同城活动推荐算法(colletivecontextual relation learning,简称CCRL).首先,对用户参与活动记录和活动主办方、活动内容、活动地点、举办时间等相关上下文信息进行关系建模;然后,采用多关系贝叶斯个性化排序学习方法进行协同上下文关系学习及同城活动推荐.Meetup数据集上的实验结果表明,该算法在多项指标上均优于现有的主流活动推荐算法.

cs.*

软件学报 · 2020 · 正式发表

新论文 · v1arXiv:2001.08361Google Research

Scaling Laws for Neural Language Models

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford 等 10 位作者

原文摘要

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.

cs.LG · stat.ML

首次观测 · 记录 #1软件学报:jos:5545软件学报

基于超图随机游走标签扩充的微博推荐方法

马慧芳,张迪,赵卫中,史忠植

原文摘要

向微博用户推荐对其有价值和感兴趣的内容,是改善用户体验的重要途径.通过分析微博特点以及现有微博推荐算法的缺陷,利用标签信息表征用户兴趣,提出一种结合标签扩充与标签概率相关性的微博推荐方法.首先,考虑到大部分微博用户未给自己添加任何标签或添加标签过少,视用户发布微博为超边,微博中的词视为超点来构建超图,并以一定的加权策略对超边和超点进行加权,通过在超图上随机游走,得到一定数量的关键词,对微博用户标签进行扩充;然后,采用相关性标签权重加权方案构建用户-标签矩阵,利用标签之间的概率相关性,构造标签相似性矩阵,对用户-标签矩阵进行更新,使该矩阵既包含用户兴趣信息,又包含标签与标签之间的关系.以新浪微博公开API抓取的微博信息作为实验数据进行了一系列的实验和分析,结果表明,该推荐算法具有较好的效果.

cs.*

软件学报 · 2019 · 正式发表

版本更新 · v5arXiv:1805.12152Berkeley BAIR

鲁棒性可能与准确率相冲突

Robustness May Be at Odds with Accuracy

Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, Aleksander Madry

AI 速读 基于摘要

作者论证对抗鲁棒性与标准泛化之间存在内在张力,训练鲁棒模型可能导致标准准确率下降,但鲁棒模型学到的特征表示更符合人类感知。

stat.ML · cs.CV · cs.LG · cs.NE

查看原文摘要

We show that there may exist an inherent tension between the goal of adversarial robustness and that of standard generalization. Specifically, training robust models may not only be more resource-consuming, but also lead to a reduction of standard accuracy. We demonstrate that this trade-off between the standard accuracy of a model and its robustness to adversarial perturbations provably exists in a fairly simple and natural setting. These findings also corroborate a similar phenomenon observed empirically in more complex settings. Further, we argue that this phenomenon is a consequence of robust classifiers learning fundamentally different feature representations than standard classifiers. These differences, in particular, seem to result in unexpected benefits: the representations learned by robust models tend to align better with salient data characteristics and human perception.

新论文 · v1arXiv:1909.00512Google Research

How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings

Kawin Ethayarajh

原文摘要

Replacing static word embeddings with contextualized word representations has yielded significant improvements on many NLP tasks. However, just how contextual are the contextualized representations produced by models such as ELMo and BERT? Are there infinitely many context-specific representations for each word, or are words essentially assigned one of a finite number of word-sense representations? For one, we find that the contextualized representations of all words are not isotropic in any layer of the contextualizing model. While representations of the same word in different contexts still have a greater cosine similarity than those of two different words, this self-similarity is much lower in upper layers. This suggests that upper layers of contextualizing models produce more context-specific representations, much like how upper layers of LSTMs produce more task-specific representations. In all layers of ELMo, BERT, and GPT-2, on average, less than 5% of the variance in a word's contextualized representations can be explained by a static embedding for that word, providing some justification for the success of contextualized representations.

cs.CL

新论文 · v1arXiv:1905.07830Google Research

HellaSwag: Can a Machine Really Finish Your Sentence?

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi

原文摘要

Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as "A woman sits at a piano," a machine must select the most likely followup: "She sets her fingers on the keys." With the introduction of BERT, near human-level performance was reached. Does this mean that machines can perform human level commonsense inference? In this paper, we show that commonsense inference still proves difficult for even state-of-the-art models, by presenting HellaSwag, a new challenge dataset. Though its questions are trivial for humans (>95% accuracy), state-of-the-art models struggle (<48%). We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers. AF proves to be surprisingly robust. The key insight is to scale up the length and complexity of the dataset examples towards a critical 'Goldilocks' zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models. Our construction of HellaSwag, and its resulting difficulty, sheds light on the inner workings of deep pretrained models. More broadly, it suggests a new path forward for NLP research, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges.

cs.CL

版本更新 · v2arXiv:1812.00740Berkeley BAIR

解耦对抗鲁棒性与泛化性

Disentangling Adversarial Robustness and Generalization

David Stutz, Matthias Hein, Bernt Schiele

AI 速读 基于摘要

论文通过假设低维数据流形,论证对抗鲁棒性与泛化性并非必然冲突,并区分流形外与流形上对抗样本。

cs.CV · cs.CR · cs.LG · stat.ML

查看原文摘要

Obtaining deep networks that are robust against adversarial examples and generalize well is an open problem. A recent hypothesis even states that both robust and accurate models are impossible, i.e., adversarial robustness and generalization are conflicting goals. In an effort to clarify the relationship between robustness and generalization, we assume an underlying, low-dimensional data manifold and show that: 1. regular adversarial examples leave the manifold; 2. adversarial examples constrained to the manifold, i.e., on-manifold adversarial examples, exist; 3. on-manifold adversarial examples are generalization errors, and on-manifold adversarial training boosts generalization; 4. regular robustness and generalization are not necessarily contradicting goals. These assumptions imply that both robust and accurate models are possible. However, different models (architectures, training strategies etc.) can exhibit different robustness and generalization characteristics. To confirm our claims, we present extensive experiments on synthetic data (with known manifold) as well as on EMNIST, Fashion-MNIST and CelebA.

首次观测 · 记录 #1软件学报:jos:5681软件学报

因子分解机模型的宽度和深度扩展研究

燕彩蓉,周灵杰,张青龙,李晓林

原文摘要

因子分解机(factorization machine,简称FM)模型因为能够有效解决高维数据特征组合的稀疏问题且具有较高的预测精度和计算效率,在广告点击率预测和推荐系统领域被广泛研究和应用.对FM及其相关模型的研究进展进行综述,有利于促进该模型的进一步改进和应用.通过比较FM模型与多项式回归模型和因子分解模型之间的关联关系,阐述FM模型的灵活性和普适性.从特征的高阶交互、特征的场交互、特征的分层交互以及基于特征工程的特征提取、合并、智能选择和提升等角度,总结模型在宽度扩展方面的方法、策略和关键技术.比较和分析了FM模型与其他模型的集成方式和特点,尤其是与深度学习模型的集成,为传统模型的深度扩展提供了思路.对FM模型的优化学习方法和基于不同并行与分布式计算框架的实现进行概括、比较和分析.最后,对FM模型中有待深入研究的难点、热点及发展趋势进行展望.

cs.*

软件学报 · 2019 · 正式发表

版本更新 · v2arXiv:1811.03604Google Research

面向移动键盘预测的联邦学习

Federated Learning for Mobile Keyboard Prediction

Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon 等 9 位作者

AI 速读 基于摘要

作者在智能手机虚拟键盘的下一词预测任务上,用联邦平均算法在客户端设备上训练循环神经网络语言模型,并与服务器端随机梯度下降训练进行比较,报告联邦方法取得更好的预测召回率。

cs.CL

查看原文摘要

We train a recurrent neural network language model using a distributed, on-device learning framework called federated learning for the purpose of next-word prediction in a virtual keyboard for smartphones. Server-based training using stochastic gradient descent is compared with training on client devices using the Federated Averaging algorithm. The federated algorithm, which enables training on a higher-quality dataset for this use case, is shown to achieve better prediction recall. This work demonstrates the feasibility and benefit of training language models on client devices without exporting sensitive user data to servers. The federated learning environment gives users greater control over the use of their data and simplifies the task of incorporating privacy by default with distributed training and aggregation across a population of client devices.

版本更新 · v4arXiv:1812.08466Google Research

Fréchet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms

Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, Matthew Sharifi

原文摘要

We propose the Fréchet Audio Distance (FAD), a novel, reference-free evaluation metric for music enhancement algorithms. We demonstrate how typical evaluation metrics for speech enhancement and blind source separation can fail to accurately measure the perceived effect of a wide variety of distortions. As an alternative, we propose adapting the Fréchet Inception Distance (FID) metric used to evaluate generative image models to the audio domain. FAD is validated using a wide variety of artificial distortions and is compared to the signal based metrics signal to distortion ratio (SDR), cosine distance and magnitude L2 distance. We show that, with a correlation coefficient of 0.52, FAD correlates more closely with human perception than either SDR, cosine distance or magnitude L2 distance, with correlation coefficients of 0.39, -0.15 and -0.01 respectively.

eess.AS · cs.SD

版本更新 · v3arXiv:1711.05101Google Research

Decoupled Weight Decay Regularization

Ilya Loshchilov, Frank Hutter

原文摘要

L$_2$ regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is \emph{not} the case for adaptive gradient algorithms, such as Adam. While common implementations of these algorithms employ L$_2$ regularization (often calling it "weight decay" in what may be misleading due to the inequivalence we expose), we propose a simple modification to recover the original formulation of weight decay regularization by \emph{decoupling} the weight decay from the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam and (ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter). Our proposed decoupled weight decay has already been adopted by many researchers, and the community has implemented it in TensorFlow and PyTorch; the complete source code for our experiments is available at https://github.com/loshchil/AdamW-and-SGDW

cs.LG · cs.NE · math.OC

首次观测 · 记录 #1软件学报:jos:5322软件学报

基于信任机制下概率矩阵分解的用户评分预测

杜东舫,徐童,鲁亚男,管楚,刘淇,陈恩红

原文摘要

互联网的蓬勃发展,在为用户提供便利的同时,其海量信息也为用户选择造成了困难,基于用户理解的信息推荐服务正成为应时之需.相较于面向单个用户信息的传统推荐技术,基于社交信息的推荐技术通过引入影响力建模,可以更真实地还原用户属性及行为.然而,已有的社交推荐技术往往停留于对用户影响的笼统归纳,并没有对其内在机制进行清晰分类和量化.针对这一问题,通过对用户评分行为中的信任关系进行分析,着重研究了信任用户间接影响用户偏好和直接影响用户评分两种不同机制,进而提出了基于用户间信任关系融合建模的概率矩阵分解模型TPMF,从而实现对上述两种机制的有效融合.在此基础之上,针对不同用户受两种机制影响权重不同的问题,通过借助评分相关性对用户进行聚类并映射到相应权重,实现了用户模型参数的个性化选择.公开数据集的多项实验结果表明:提出的TPMF及其衍生算法在各项指标上优于现有代表性算法,验证了所提出的影响机制及技术框架的有效性.

cs.*

软件学报 · 2018 · 正式发表

首次观测 · 记录 #1软件学报:jos:5648软件学报

系统软件新洞察

王怀民,毛晓光,丁博,沈洁,罗磊,任怡

原文摘要

系统软件是计算学科的基本概念之一,从系统软件的本质特征、时代特点和发展趋势这3个方面给出了关于系统软件的新洞察.洞察1认为,通用图灵机和存储程序思想是系统软件的理论源头和技术源头,其本质特征是"操纵计算系统执行",编码加载和执行管控是两种主要的操纵方式.洞察2认为,系统软件在互联网时代的时代特点是持续在线提供基础服务,为"软件即服务"的新型应用模式奠定了基础.洞察3认为,系统软件的发展趋势是持续在线演化,在计算系统创新、信息物理融合和智能技术的推动下,将成为未来软件生态的核心.

cs.*

软件学报 · 2018 · 正式发表

首次观测 · 记录 #1软件学报:jos:5437软件学报

一种融合节点先验信息的图表示学习方法

温雯,黄家明,蔡瑞初,郝志峰,王丽娟

原文摘要

图表示学习是实现各类图挖掘任务的基础.现实中的图数据不仅包含复杂的网络结构,还包括多样化的节点信息.如何将网络结构和节点信息更加有效地融入图的表示学习中,是一个重要的问题.为了解决这一问题,基于深度学习,提出了融合节点先验信息的图表示学习方法.该方法将节点特征作为先验知识,要求学习到的表示向量同时保持图数据中的网络结构相似性和节点特征相似性.该方法的时间复杂度为O(|V|),其中,|V|为图节点数量,表明该方法适用于大规模图数据分析.同时,在多个数据集上的实验结果表明:所提出的方法相比目前流行的几种基线方法,在分类任务上能够获得良好而稳定的优势.

cs.*

软件学报 · 2017 · 正式发表

首次观测 · 记录 #1软件学报:jos:2017008软件学报

AtexRace: across thread and execution sampling for in-house race detection

郭郁,蔡彦,扬子江

原文摘要

Data race is a major source of concurrency bugs. Dynamic data race detection tools (e.g., FastTrack) monitor the execu-tions of a program to report data races occurring in runtime. However, such tools incur significant overhead that slows down and perturbs executions. To address the issue, the state-of-the-art dynamic data race detection tools (e.g., LiteRace) ap-ply sampling techniques to selectively monitor memory access-es. Although they reduce overhead, they also miss many data races as confirmed by existing studies. Thus, practitioners face a dilemma on whether to use FastTrack, which detects more data races but is much slower, or LiteRace, which is faster but detects less data races. In this paper, we propose a new sam-pling approach to address the major limitations of current sampling techniques, which ignore the facts that a data race involves two threads and a program under testing is repeatedly executed. We develop a tool called AtexRace to sample memory accesses across both threads and executions. By selectively monitoring the pairs of memory accesses that have not been frequently observed in current and previous executions, AtexRace detects as many data races as FastTrack at a cost as low as LiteRace. We have compared AtexRace against FastTrack and LiteRace on both Parsec benchmark suite and a large-scale real-world MySQL Server with 223 test cases. The experiments confirm that AtexRace can be a replacement of FastTrack and LiteRace.

cs.*

软件学报 · 2017 · 正式发表

首次观测 · 记录 #1软件学报:jos:2017009软件学报

Adaptively generating high quality fixes for atomicity violations

蔡彦,曹玲微,赵靖

原文摘要

It is difficult to fix atomicity violations correctly. Existing gate lock algorithm (GLA) simply inserts gate locks to serialize exe-cutions, which may introduce performance bugs and deadlocks. Synthesized context-aware gate locks (by Grail) require complex source code synthesis. We propose ?Fixer to adaptively fix ato-micity violations. It firstly analyses the lock acquisitions of an atomicity violation. Then it either adjusts the existing lock scope or inserts a gate lock. The former addresses cases where some locks are used but fail to provide atomic accesses. For the latter, it infers the visibility (being global or a field of a class/struct) of the gate lock such that the lock only protects related accesses. For both cases, ?Fixer further eliminates new lock orders to avoid introducing deadlocks. Of course, ?Fixer can produce both kinds of fixes on atomicity violations with locks. The experi-mental results on 15 previously used atomicity violations show that: ?Fixer correctly fixed all 15 atomicity violations without introducing deadlocks. However, GLA and Grail both intro-duced 5 deadlocks. HFix (that only targets on fixing certain types of atomicity violations) only fixed 2 atomicity violations and introduced 4 deadlocks. ?Fixer also provides an alternative way to insert gate locks (by inserting gate locks with proper visibility) considering fix acceptance.

cs.*

软件学报 · 2017 · 正式发表

首次观测 · 记录 #1软件学报:jos:5391软件学报

融合社交信息的矩阵分解推荐方法研究综述

刘华锋,景丽萍,于剑

原文摘要

随着社交网络的发展,融合社交信息的推荐成为推荐领域中的一个研究热点.基于矩阵分解的协同过滤推荐方法(简称矩阵分解推荐方法)因其算法可扩展性好及灵活性高等诸多特点,成为研究人员在其基础之上进行社交推荐模型构建的重要原因.围绕基于矩阵分解的社交推荐模型,依据模型的构建方式对社交推荐模型进行综述.在实际数据上,对已有代表性社交推荐方法进行对比,分析各种典型社交推荐模型在不同视角下的性能(如整体用户、冷启动用户、长尾物品).最后,分析了基于矩阵分解的社交推荐模型及其求解算法存在的问题,并对未来研究方向与发展趋势进行展望.

cs.*

软件学报 · 2017 · 正式发表