Super Weights in LLMs and the Failure of Selective Training

论文概要

研究领域: LLM训练 作者: Shreyas Subramanian, Adewale Akinfaderin, Akarsha Sehwag 发布时间: 2025-07-12 arXiv: 2507.08699

中文摘要

最近的研究发现了超级权重（Super Weights），即单个参数的移除会使模型性能下降数个数量级。我们表明这种性能下降并不普遍适用于所有LLM。单独训练超级权重（100到8,192个参数）会使OLMo-1B和OLMo-7B的准确率降至随机猜测水平。这种失败是超级权重坐标特有的：在相同的down_proj层中训练相同数量的随机选择位置反而比基线更好。普通LoRA通过低秩结构更新注意力权重矩阵中的每个位置，仅使用0.16%的参数就成功了。这些发现确立了参数重要性并不意味着单独训练该参数的可行性，有效的微调依赖于整个层的结构化分解。

原文摘要

Recent work identified Super Weights, individual parameters whose removal degrades model performance by orders of magnitude. We show that this degradation does not universally apply to all LLMs. Training Super Weights in isolation (100 to 8,192 parameters) drops accuracy to random-guessing levels on both OLMo-1B and OLMo-7B. The failure is specific to Super Weight coordinates: training an equal number of randomly chosen positions in the same down_proj layers instead improves over the baseline. Vanilla LoRA, updating every position in attention weight matrices through low-rank structure, succeeds with only 0.16% of parameters. These findings establish that parameter importance does not imply parameter trainability in isolation, and that effective fine-tuning relies on structured decompositi…

— 自动采集于 2025-07-13

#论文 #arXiv #LLM训练 #小凯