论文概要
研究领域: ML 作者: Tiberiu Musat, Tiago Pimentel, Nicholas Zucchet, Thomas Hofmann 发布时间: 2026-07-13 arXiv: 2607.11875
中文摘要
本文提出一个理论框架解释Transformer语言模型中归纳推理能力的涌现。先前关于Transformer学习动态的研究大多局限于特定任务，本文研究一类广义的归纳任务，统一了文献中多个已知合成任务，包括上下文n-gram和多跳推理。在此类任务中，理论上证明注意力模型的训练动态可被限制在一个高度可解释的低维不变流形上。在此流形上，学习动态由少数可解释坐标而非数百万参数捕获，使理论和实证分析更易于处理。利用该框架，作者刻画了数据统计如何调控上下文学习与权重内学习之间的竞争，研究随机初始化如何决定存在多个解时的”获胜”回路，并证明与流形相关的坐标框架可用于自动检测训练模型中学到了哪些回路。将回路形成视为低维动态现象，向Transformer如何学习的预测理论迈进一步。
原文摘要
We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models. While previous works on Transformer learning dynamics have so far been mostly tied to specific tasks, we study a generalized class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning. In this class, we theoretically prove that the training dynamics of attention models can be confined to a highly interpretable, low-dimensional invariant manifold. On this manifold, the learning dynamics are captured by a handful of interpretable coordinates rather than millions of parameters, making both theoretical and empirical analysis more tractable. Using this framework, we characterize how data stat…
— 自动采集于 2026-07-15
#论文 #arXiv #ML #小凯