论文概要
研究领域: CV 作者: Runhui Huang, Qihui Zhang, Zhe Liu, Yu Gao, Jie Wu 发布时间: 2026-07-13 arXiv: 2607.11886
中文摘要
本文提出 SpectraReward，一种无需训练即可将预训练多模态大语言模型（MLLM）转化为图像生成强化学习奖励模型的方法。不同于让MLLM评判生成图像或回答分解验证问题，SpectraReward 通过单次图像条件化的教师强制前向传播，测量原始提示词从生成图像中被恢复的程度。它以图像条件化提示词的平均对数似然作为奖励，直接复用MLLM预训练的图文对齐能力，无需偏好标签或奖励模型微调。此外还提出 Self-SpectraReward，让统一多模态模型自身的理解分支作为其生成分支的奖励模型，形成无需外部奖励模型或外部知识的闭环自改进框架。大量实验覆盖两种扩散模型、三种RL算法、四个MLLM家族的九个骨干网络（4B到235B参数）和五个分布外文生图基准，验证 SpectraReward 和 Self-SpectraReward 均显著且持续地提升生成性能，超越先前基于MLLM的奖励训练方法。进一步分析揭示更大的奖励MLLM并非总是更好，而 Self-SpectraReward 可匹配或超越更大的外部奖励模型，表明奖励-策略对齐是有效图像生成RL的关键因素。
原文摘要
In this paper, we propose SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM’s pretrained image-text alignment ability without preference labels, reward-model fine-tuning. We further introduce Self-SpectraReward, a special case for unified multimodal models where the policy’s own understanding branch serves as the reward model for its gen…
— 自动采集于 2026-07-15
#论文 #arXiv #CV #小凯