Learning & methods · November 2026学习与方法 · 2026 年 11 月
Does a Higher Reward Mean a Better Model?奖励涨了,模型真的变强了吗?
Read reward, KL, entropy, policy loss, and validation scores as different measurements—and learn what evidence is needed to connect them.把奖励、KL、熵、策略损失与验证分数放回各自的测量对象,理解一条曲线能回答什么,以及还缺什么证据。
First ask what this reward includes先问:这条 reward 到底加减了什么?
一个模型的验证正确率上升,训练面板里的 reward 却下降。此时立刻判断“过拟合”或“训练坏了”,都可能看错了对象。更有用的第一步是把指标展开:谁生成了答案,谁给分,是否扣除了 KL,按 token、答案还是问题平均?名称相同,不代表统计量相同。
例如 TRL v0.25.1 把 objective/scores 与 objective/rlhf_reward 分开记录;后者扣除 KL 相关代价。这个约定不能自动套用到另一个项目的 train/reward。必须沿着日志字段找到实际计算式。为便于讨论,下面明确使用:
取一个教学账本,不是实测曲线:固定评测问题、评分规则、KL 汇总单位与 。早期原始得分 0.60、KL 代价 0.02,合成目标是 0.58;后期得分 0.66、代价 0.12,目标变为 0.54。原始表现提高与受约束目标降低可以同时成立。至于实际训练是否应接受这个交换,要看约束的用途与独立评测,不能只挑更好看的一列。
Validation accuracy improves, yet the training dashboard shows a falling reward. Calling this overfitting or a broken run may confuse different measurements. First expand the metric: which policy generated the answers, which evaluator scored them, whether a KL cost was subtracted, and whether the reduction is over tokens, responses, or prompts. Matching names do not establish matching quantities.
For example, TRL v0.25.1 distinguishes objective/scores from objective/rlhf_reward; the latter subtracts a KL-related cost. This convention does not automatically apply to another project's train/reward. Trace the logging field to its computation. Here we explicitly use:
Consider a teaching example, not measured training data. Hold evaluation prompts, scoring rules, KL units, and fixed. Initially, a raw score of 0.60 minus a cost of 0.02 gives 0.58. Later, a score of 0.66 minus a cost of 0.12 gives 0.54. Better raw performance and a lower regularized objective can coexist. Whether that trade-off is acceptable depends on the purpose of the constraint and independent evaluation, not on selecting the more attractive column.
Static diagram静态图
若 KL 是每 token 平均而奖励是整条答案得分,直接相减还缺少长度和系数的定义。比较两次运行时,题目组成、答案长度、采样温度、缺失 EOS 的处理、奖励版本也应保持可比。奖励增加本身并不能排除模型学会了评分器的漏洞。TRL 指标定义提供的是一个可核对的实例,而非所有框架的统一命名标准。
If KL is averaged per token while reward scores a whole answer, subtracting them also requires a specified length reduction and coefficient. Comparisons across runs need compatible prompt mixtures, answer lengths, sampling temperatures, missing-EOS handling, and reward versions. A rising reward alone does not rule out exploitation of an evaluator. The TRL definitions are a checkable example, not a universal naming standard.
Two KL measurements have different anchors两种 KL,量的是两段不同的距离
参考策略 通常长时间固定;采样策略 则随着 rollout 批次刷新。当前策略离参考策略多远,与它在这轮更新里离旧策略多远,是两个问题。连续许多小步可以走得很远,所以 reference KL 上升而 update KL 很小,并不矛盾。
| 问题 | 比较对象 | 常见用途 |
|---|---|---|
| 离起点多远? | current 与 reference | 控制相对初始行为的偏移 |
| 这一轮走多远? | old 与 current | 观察更新幅度、复用 rollout 的程度 |
方向和采样分布也必须写清。固定一批旧策略访问的前缀,令 ,动作由旧策略采样。具有共同支持集时,。CleanRL 的近似 KL 使用这一形式;有限批次仍只是估计,而且它不等同于对整个新策略轨迹分布计算 KL。
因此,不存在脱离单位的“KL 小于某个数就正常”。每 token 平均、整序列求和、prompt 平均的尺度不同。KL 接近零可能是更新很小、实现记录了别的量,或策略确实停滞;先确认定义,再结合梯度、参数变化与任务结果判断。CleanRL 的实际计算代码也把 KL、比率越界比例和梯度范数裁剪分成了不同操作。
A reference policy often stays fixed for a long interval; the rollout policy is refreshed across batches. Distance from the reference and movement within the current update answer different questions. Many small steps can travel far, so increasing reference KL and small update KL are compatible.
| Question | Compared policies | Typical purpose |
|---|---|---|
| How far from the starting behavior? | Current and reference | Control drift from a behavioral anchor |
| How far in this update? | Old and current | Monitor update size and rollout reuse |
State the direction and sampling distribution too. At prefixes visited by a fixed old-policy batch, let , with actions sampled from the old policy. Under common support, . CleanRL uses this form for approximate KL. A finite batch remains an estimate, and the quantity is not the KL of the complete new-policy trajectory distribution.
There is therefore no unit-free rule that KL below a particular number is healthy. Per-token averages, sequence sums, and prompt averages have different scales. Near-zero KL could mean a small update, a differently defined logger, or genuine stagnation. Establish the definition before comparing gradients, parameter movement, and task outcomes. CleanRL's implementation also separates KL, the ratio-outside-range fraction, and gradient-norm clipping.
Lower entropy can mean accuracy or collapse熵下降,既可能更准确,也可能更单一
在一个固定前缀上,token 熵是分布的不确定性:。自然对数对应 nat。二选一的概率从 变成 ,熵从 0.693 降到 0.325;但如果第一个选项正确,这是变得更可靠,若第二个才正确,则是变得更自信地犯错。熵的数值不包含正确答案标签。
词面的多样也不等于解法的多样。十种同义表达可以有较高 token 熵,却采用同一条错误思路;两种措辞朴素的答案也可能使用完全不同的正确方法。把熵当作探索的线索,需要同时看去重后的解法、分支覆盖和结果。聚合方式同样重要:一个 10-token 回答平均熵为 1,一个 90-token 回答为 0,先按回答平均得到 0.5,合并所有 token 则得到 0.1。
pass@k 回答的是“给 k 次机会,至少成功一次的概率”。在同一题上假设独立同分布采样、单次成功率为 ,则 。取 k=4,p 从 0.5 升到 0.9,pass@4 从 0.9375 升到 0.9999,但它与 pass@1 的差距从 0.4375 缩至 0.0999。差距缩小完全可能是单次成功率提高后的饱和效应,不能单独诊断多样性塌缩。
真实题目难度不同,应先按题估计再平均,不能把总体平均 p 直接塞进非线性公式。有限样本评测还要使用合适的 pass@k 估计器;本文算例只是已知 p 的理想模型。类似地,GRPO 组内奖励方差为零,可能全对,也可能全错;两者都不给相对优势提供组内区分,但训练含义截然不同。HumanEval 论文 §2 与附录 A给出了 pass@k 的定义和有限样本估计。
At one fixed prefix, token entropy measures distributional uncertainty: , in nats when using natural logarithms. Moving a binary distribution from to lowers entropy from 0.693 to 0.325. If the first option is correct, reliability improves; if the second is correct, the model becomes more confidently wrong. Entropy does not contain the answer label.
Lexical diversity is not diversity of reasoning. Ten paraphrases can produce high token entropy while following the same faulty approach; two plainly worded answers can use different correct methods. Treat entropy as an exploration signal alongside distinct solution strategies, branch coverage, and outcomes. Reduction matters too: if a 10-token response has average entropy 1 and a 90-token response has average entropy 0, averaging responses gives 0.5, whereas averaging all tokens gives 0.1.
Pass@k asks for the probability of at least one success in k attempts. For a fixed question with independent, identically distributed attempts and single-attempt success probability , . With k=4, increasing p from 0.5 to 0.9 raises pass@4 from 0.9375 to 0.9999, while its gap from pass@1 shrinks from 0.4375 to 0.0999. A shrinking gap can simply reflect saturation after improving single-attempt success; it does not by itself diagnose diversity collapse.
Real questions have different difficulties. Estimate per question before averaging instead of inserting the overall mean p into a nonlinear formula. Finite-sample evaluation also requires an appropriate pass@k estimator; this example assumes p is known. Similarly, zero within-group GRPO reward variance can mean every answer is correct or every answer is wrong. Neither supplies within-group discrimination for relative advantages, but their training implications differ sharply. The HumanEval paper, §2 and Appendix A, defines pass@k and its finite-sample estimator.
A zero policy loss can have a nonzero gradient策略损失等于零,仍然可以推动更新
监督学习里,我们习惯在固定数据上观察 loss 是否下降。在线 RL 却不断更换采样策略、轨迹和优势;即使代码中的公式没改,数值对应的样本分布也变了。将所有训练步的 policy loss 当成同一张固定试卷上的分数,容易产生误解。
一个两动作算例足以说明。旧策略各以 1/2 采样两个动作,冻结优势为 ,当前第一个动作的概率是 。暂不考虑处于阈值外的裁剪:
在 z=0 时,p=1/2,L=0;但 。梯度下降仍会增加第一个动作的概率。对优势中心化后,零附近的 loss 很常见,它不意味着没有学习。
还应区分两个“裁剪”:PPO 对概率比构造裁剪代理目标;梯度范数裁剪约束的是反向传播后梯度向量的大小。CleanRL 的 clipfrac 统计 的样本比例,甚至不完全等于该样本在给定优势符号下选择了裁剪分支。它既不是“多少参数被剪”,也不保证所有真实概率比落在区间内。是否异常,应结合比率分布、优势符号、更新轮数和任务评测判断,而不是追求所有曲线单调下降。
In supervised learning, we often monitor loss on fixed data. Online RL continually changes the sampling policy, trajectories, and advantages. Even with the same formula, the sampled distribution changes. Treating policy loss across all training steps as scores on one fixed exam is misleading.
A two-action example is enough. The old policy samples both actions with probability 1/2; freeze their advantages at . Let the current probability of the first action be . In the region where clipping is inactive:
At z=0, p=1/2 and L=0, yet . Gradient descent still increases the first action's probability. A loss near zero is common after centering advantages; it does not imply an absence of learning.
Also distinguish two kinds of clipping. PPO constructs a clipped surrogate using probability ratios; gradient-norm clipping acts on the gradient vector after backpropagation. CleanRL's clipfrac counts samples with . This is not even exactly the fraction selecting the clipped branch for the observed advantage sign. It neither counts clipped parameters nor guarantees that all actual probability ratios remain inside the interval. Diagnose it together with ratio distributions, advantage signs, update epochs, and task evaluation instead of demanding that every curve decrease monotonically.
Evaluation can influence selection without backpropagation没有反向传播,也可能参与了训练选择
训练集影响参数梯度;验证集影响 checkpoint、提示词、奖励权重和超参数的选择;最终测试集用于评估冻结后的方案。这三个角色由信息如何流动决定,不由文件夹叫什么决定。
假设比较 20 个 checkpoint,挑验证分数最高者。选择会同时利用真实能力差异与这批题目的偶然噪声;再用同一最高分声称最终能力,就可能乐观。验证集并非因此不可用——它本来就服务于选择——但需要另一份未参与选择的评测来量化选中方案。若每看一次“test”都修改训练,它实际上也承担了验证角色。重复运行一个预先冻结的评测协议则不自动构成这种适应性选择,应记录实际做过哪些决策。
Training data affect parameter gradients; validation data affect checkpoint, prompt, reward-weight, and hyperparameter choices; final test data evaluate a frozen procedure. These roles are defined by information flow rather than directory names.
Suppose we compare 20 checkpoints and select the highest validation score. Selection uses both genuine capability differences and noise peculiar to those questions. Reporting that same maximum as final capability can be optimistic. Validation remains useful—it is meant to support selection—but evaluating the selected procedure requires another set not used for selection. If every inspection of a “test” score leads to a training change, that set is serving as validation too. Repeating a prespecified frozen evaluation does not automatically create this adaptive-selection problem; record which decisions actually used the results.
Static diagram静态图
对生成模型,还要固定采样温度、每题尝试次数、最大长度和判题器版本。题目重复、同模板变体、工具返回内容泄漏,会让名义上的不同 split 仍共享信息。按任务、时间或用户来源分组划分是否必要,要由预期部署场景决定。Cawley 与 Talbot 的研究解释了为什么模型选择过程本身也可能过拟合有限评测样本。
For generation, also fix temperature, attempts per question, maximum length, and evaluator version. Duplicated questions, template variants, and leaked tool outputs can make nominally different splits share information. Whether to split by task, time, or user source depends on the intended deployment. Cawley and Talbot explain why model selection itself can overfit finite evaluation samples.
Explain a run with complementary evidence用相互独立的证据解释一次训练
遇到“奖励升、熵降”,先看固定条件下的真实成功率与解法覆盖:可能只是更稳定地答对,也可能在重复讨巧答案。遇到“训练 reward 降、验证分数升”,先拆开原始得分与 KL 账本,再核对训练和验证的题目难度;这组曲线本身不能排除 reward hacking。遇到“loss 很平”,先确认优势的中心化和每轮数据变化,再检查梯度与更新是否实际发生。
一个够用的面板因此至少需要四种互补证据:可比的任务结果、明确口径的目标组成、策略变化的尺度、独立评测上的表现。再保留少量固定题目的实际回答,才能把数值变化联系到行为变化。指标的价值在于帮助提出可检查的解释,而不是替模型作出“已经学会了”的结论。
For “reward up, entropy down,” inspect task success and solution coverage under fixed conditions: the model may answer correctly more reliably, or repeat evaluator-friendly shortcuts. For “training reward down, validation score up,” separate raw scores from KL costs and compare prompt difficulty. Those curves alone cannot rule out reward hacking. For “flat loss,” check advantage centering and changing data before examining whether gradients and parameter updates actually occur.
A useful dashboard therefore needs four complementary kinds of evidence: comparable task outcomes, explicitly defined objective components, the scale of policy movement, and independent evaluation. Retain actual responses to a small fixed set of questions to connect numerical changes with behavior. Metrics help formulate checkable explanations; they do not themselves establish that a model has learned the intended skill.
Continue reading继续阅读
- PPO: Proximal Policy Optimization Algorithms
- TRL v0.25.1: PPO logged metrics
- CleanRL: PPO implementation
- Evaluating Large Language Models Trained on Code, §2 and Appendix A
- Cawley & Talbot: Over-fitting in Model Selection
- Original study notes原始学习笔记 · ppo grpo entropy
- Original study notes原始学习笔记 · rl metrics