论文概要
研究领域: ML 作者: Jianghui Wang, Silong Yong, Francesco Orabona, Marco Canini, Katia P. Sycara, Yaqi Xie 发布时间: 2026-07-24 arXiv: 2607.22489
中文摘要
低秩适配(LoRA)已成为广泛采用的高效神经网络微调技术，将模型更新分解为低秩矩阵。然而，LoRA仍然计算成本高昂，因为它均匀地更新所有矩阵，而不考虑它们对适配的实际贡献。这一成本对具有数十亿参数的大规模模型以及边缘部署和设备端微调等资源受限设置尤其不可接受。本文首次表明并非所有LoRA矩阵都同样值得微调：条件数较小（最大与最小奇异值之比）的矩阵在各方向上已经平衡良好，对适配的贡献仅为边际性的，而条件数较大的矩阵包含未充分发展的方向，跨越更丰富的子空间并驱动大部分性能提升。这一观察本身就是本文的关键贡献，并激发了更具选择性的微调方法。基于这一洞察，本文提出κ-LoRA，一种通过专注于条件数最大的矩阵来优化LoRA的方法，这些矩阵捕获了最具信息量的变化方向。通过将LoRA更新限制在按条件数排名的前50%权重矩阵，κ-LoRA将可训练参数量减半并相应减少计算和内存成本。跨多个基准测试的大量实验表明，该设计平均减少了16.2%的微调时间，同时匹配标准LoRA的准确性并减少4.5%的内存成本。进一步分析揭示，所选矩阵的条件数在训练过程中持续下降，表明κ-LoRA的有效性源于有针对性的谱重平衡而非仅靠参数选择。
原文摘要
Low-Rank Adaptation (LoRA) has become a widely adopted technique for efficient neural network fine-tuning, decomposing model updates into low-rank matrices. However, LoRA remains computationally costly because it updates all matrices uniformly, regardless of their actual contribution to adaptation. This cost is especially prohibitive for large-scale models with billions of parameters and for resource-constrained settings such as edge deployment and on-device fine-tuning. We show for the first time that not all LoRA matrices are equally worth tuning: matrices with smaller condition numbers (the ratio of largest to smallest singular value) are already well-balanced across directions and contribute only marginally to adaptation, whereas matrices with larger condition numbers contain underdeve…
— 自动采集于 2026-07-28
#论文 #arXiv #ML #小凯