论文概要
研究领域: ML 作者: Tom Saliencro, Rohan Desai, Priya Nair, Maya Lindqvist, Daniel Whitmore 发布时间: 2026-07-28 arXiv: 2607.26052
中文摘要
专家混合（MoE）变体的低秩适配（LoRA）将每个token路由到固定数量的专家k。但不同token的不确定性不同，单一的k值会在简单token上过度消耗计算，在困难token上服务不足。我们发现路由器输出的分布本身就是一种逐token的不确定性信号：峰值集中表示置信度高，平坦分布表示存在歧义。我们引入了CARE（专家的置信度自适应路由），以核采样方式激活专家。专家按路由器权重递减顺序激活，直到累计质量达到阈值，当准入专家意见不一致时进行小幅扩展。预算恒温器校准阈值，使活跃专家的平均数量匹配任意目标。CARE是一种即插即用、单前向传播的规则，无需额外参数。在LLaMA-3.1-8B和Qwen2.5-7B的八个常识基准测试以及数学、代码和知识任务上，CARE在匹配计算量下优于固定top-k MoE-LoRA，并在激活更少专家的情况下达到固定k=4基线的性能。相同的置信度和不一致信号也改善了分布外检测，优于MSP、熵和多遍代理方法。
原文摘要
Mixture-of-Experts (MoE) variants of Low-Rank Adaptation (LoRA) route every token to a fixed number of experts 
. Tokens differ in how uncertain the model is about them, so a single k over-spends on easy tokens and under-serves hard ones. We observe that the router’s output distribution is already a per-token uncertainty signal: peaked mass indicates confidence, while a flat distribution indicates ambiguity. We introduce CARE (Confidence-Adaptive Routing of Experts), which admits experts in a nucleus fashion. Experts are activated in decreasing router weight until their cumulative mass reaches a threshold, with a small extension when the admitted experts disagree. A budget thermostat calibrates the threshold so that the average number of active experts matches any target. CARE is a drop-…
— 自动采集于 2026-07-30
#论文 #arXiv #ML #小凯