{
  "slug": "grpo-update",
  "category": "Post-training",
  "title": [
    "一次 GRPO 更新，究竟学到了什么？",
    "Inside a GRPO Update",
    "GRPOの1回の更新で何を学ぶのか"
  ],
  "lead": [
    "同一道题，模型生成几种回答。有的通过验证，有的失败。GRPO 用这组回答的差异，调整下一次生成的概率。沿着一组具体数字，就能看懂这个过程。",
    "The model answers one question several ways. Some attempts pass verification; others fail. GRPO uses differences within that group to change future generation probabilities. A small numerical example makes the update tangible.",
    "同じ問題に複数の回答を生成し、検証に通るものと失敗するものを比較します。GRPOはその差を使って次回の生成確率を変えます。具体的な数値で更新を追います。"
  ],
  "sections": [
    {
      "title": [
        "把一次回答拆成策略的连续选择",
        "A response is a sequence of policy decisions",
        "回答を方策の連続した選択に分ける"
      ],
      "paragraphs": [
        [
          "对语言模型，状态可取问题q与已生成前缀o<ₜ，动作是下一个token，πθ(oₜ|q,o<ₜ)是该动作概率。一条回答的概率是各token条件概率之积：πθ(o|q)=∏ₜπθ(oₜ|q,o<ₜ)。因此log概率是和，整段结果奖励可以通过这些log概率反传到生成路径。工具Agent还包含工具观察与环境转移，不能把所有动作都当纯文本拼接。",
          "For a language model, use the question q and generated prefix o<ₜ as state, the next token as action, and πθ(oₜ|q,o<ₜ) as its probability. A response has probability πθ(o|q)=∏ₜπθ(oₜ|q,o<ₜ). Its log probability is a sum, allowing an outcome reward to train the sampled path through token log probabilities. Tool agents additionally include observations and environmental transitions.",
          "言語モデルでは問題qと生成済み接頭辞o<ₜを状態、次tokenを行動、πθ(oₜ|q,o<ₜ)をその確率とします。回答確率πθ(o|q)=∏ₜπθ(oₜ|q,o<ₜ)のlogは和なので、結果報酬をtokenのlog確率を通じて学習に使えます。ツールAgentには観測や環境遷移もあり、単なる文字列追加だけでは表せません。"
        ],
        [
          "结果奖励可以是最终答案的规则校验、程序测试或模型评审。必须先冻结“怎样评分”：解析失败、超时、截断、格式通过与业务通过不是同一标签。训练会放大被奖励的行为；若验证器只检查字段名，优化器不会自动补上它没检查的字段语义。",
          "An outcome reward may come from answer checks, program tests, or a model judge. Specify the scoring contract first: parse failure, timeout, truncation, valid format, and business success are different events. Training reinforces what receives reward. If a verifier only checks field names, the optimizer cannot be assumed to supply semantics the verifier ignores.",
          "結果報酬は正答判定、テスト、モデル評価などから得られます。解析失敗・タイムアウト・切詰め・形式妥当性・業務成功を先に区別します。訓練は報酬を与えた行動を強化します。検証器がフィールド名しか見なければ、未検証の意味を最適化器が自動で補う保証はありません。"
        ]
      ]
    },
    {
      "title": [
        "从期望奖励到 REINFORCE",
        "From expected reward to REINFORCE",
        "期待報酬からREINFORCEへ"
      ],
      "paragraphs": [
        [
          "固定问题q，目标J(θ)=Σₒπθ(o|q)R(q,o)。用∇π=π∇logπ得到 ∇J=E[R∇logπθ(o|q)]，再把logπ展开为Σₜlogπθ(oₜ|q,o<ₜ)。这就是用采样奖励加权生成token的策略梯度。奖励无需可微，反向传播经过的是模型概率；实现时奖励和优势作为固定权重，不沿评分器反传。",
          "For fixed q, J(θ)=Σₒπθ(o|q)R(q,o). Using ∇π=π∇logπ yields ∇J=E[R∇logπθ(o|q)], with logπ expanded into a token sum. This gives a policy gradient weighted by sampled reward. Rewards need not be differentiable; gradients flow through policy probabilities. Implementations treat rewards and advantages as fixed weights rather than differentiating through the evaluator.",
          "qを固定しJ(θ)=Σₒπθ(o|q)R(q,o)とします。∇π=π∇logπより∇J=E[R∇logπθ(o|q)]となり、logπはtokenごとの和です。報酬は微分可能でなくてもよく、勾配はモデル確率を通ります。実装では報酬と優位性を固定重みとして扱い、評価器へ逆伝播しません。"
        ],
        [
          "在一个固定状态，减去不依赖当前动作的基线b(s)不会改变期望梯度，因为Σₐπ(a|s)∇logπ(a|s)=∇Σₐπ(a|s)=0。合适基线能降低方差；优势Aπ(s,a)=Qπ(s,a)−Vπ(s)问的是“这个动作比当前状态通常的选择好多少”。这项不偏结论有动作独立条件，不可直接用来证明包含本样本的组均值及标准差归一化仍是同一个不偏估计器。",
          "At a fixed state, subtracting an action-independent baseline b(s) preserves the expected gradient because Σₐπ(a|s)∇logπ(a|s)=∇Σₐπ(a|s)=0. A suitable baseline reduces variance; Aπ(s,a)=Qπ(s,a)−Vπ(s) compares the action with typical behavior at that state. This argument requires action independence. It does not automatically prove that a group mean containing the same sample, followed by standardization, is the same unbiased estimator.",
          "固定状態で行動に依存しないb(s)を引いても、Σₐπ(a|s)∇logπ(a|s)=∇Σₐπ(a|s)=0なので期待勾配は変わりません。適切な基準は分散を減らし、Aπ=Qπ−Vπは通常の選択との差を示します。ただし行動非依存が条件です。自分を含むグループ平均と標準化にも同じ不偏性が成立すると即断できません。"
        ]
      ]
    },
    {
      "title": [
        "PPO 的价值模型解决了哪一部分",
        "What PPO’s value model estimates",
        "PPOの価値モデルが推定するもの"
      ],
      "paragraphs": [
        [
          "Actor–critic版本的PPO用V(sₜ)预测未来回报，TD残差为δₜ=rₜ₊₁+γV(sₜ₊₁)−V(sₜ)。GAE将后续残差按(γλ)ˡ加权：Âₜ=Σₗ≥₀(γλ)ˡδₜ₊ₗ。λ调节依赖长回报还是局部自举；终止状态的bootstrap应为0，长度截断则需要按实际任务定义处理。",
          "An actor–critic implementation of PPO estimates return with V(sₜ), giving TD residual δₜ=rₜ₊₁+γV(sₜ₊₁)−V(sₜ). GAE combines residuals as Âₜ=Σₗ≥₀(γλ)ˡδₜ₊ₗ. λ trades longer-horizon returns against local bootstrapping. True terminal states have zero bootstrap; length truncation requires treatment appropriate to the task.",
          "Actor–critic版PPOはV(sₜ)で収益を予測し、δₜ=rₜ₊₁+γV(sₜ₊₁)−V(sₜ)を使います。GAEはÂₜ=Σₗ≥₀(γλ)ˡδₜ₊ₗと残差をまとめます。λは長期収益と局所bootstrapの配分を変えます。真の終端はbootstrapを0とし、長さによる切詰めはタスク定義に従って扱います。"
        ],
        [
          "GRPO改用同题多答的相对奖励，省去单独训练这个critic，却增加了每题多条rollout的需求。节省多少显存取决于是否保存旧策略副本、优化器状态、参数共享、卸载和序列长度，不能仅数“三个模型对四个模型”就断言固定节省25%。",
          "GRPO replaces this separately learned critic with relative rewards from multiple responses to one prompt, while requiring grouped rollouts. Memory savings depend on old-policy storage, optimizer states, parameter sharing, offloading, and sequence lengths. Counting three versus four model roles does not establish a fixed 25% saving.",
          "GRPOは別学習のcriticを同じ問題への複数回答の相対報酬に置き換えますが、複数rolloutが必要です。メモリ削減率は旧方策の保存、optimizer状態、共有、offload、系列長に依存し、役割が4つから3つだから25%と固定できません。"
        ]
      ]
    },
    {
      "title": [
        "先生成一组，再比较",
        "Sample a group, then compare",
        "グループを生成して比較する"
      ],
      "paragraphs": [
        [
          "考虑只有结果奖励的设置。对同一问题，从采样策略生成 G 条回答，每条得到一个奖励。组内均值提供比较基准；常见的 GRPO 形式还除以组内标准差。它不需要另训练一个价值模型来给这些回答估计基线。",
          "In the outcome-reward setting, sample G responses to the same prompt and assign each a reward. The group mean supplies a baseline; the common GRPO formulation also divides by the group standard deviation. This avoids training a separate value model to estimate that baseline.",
          "結果報酬の設定では、同じ問題からG個の回答を生成し、それぞれに報酬を付けます。グループ平均を基準とし、一般的なGRPOでは標準偏差でも割ります。この基準を推定する別の価値モデルは不要です。"
        ],
        [
          "优势值 Aᵢ = (rᵢ − mean(r)) / (std(r) + ε)。高于平均值的回答得到正优势，低于平均值得到负优势。奖励是“本次回答如何”，优势是“相对于同组其他回答如何”。",
          "The advantage is Aᵢ = (rᵢ − mean(r)) / (std(r) + ε). Above-average responses receive positive advantages; below-average responses receive negative ones. Reward evaluates an attempt; advantage compares it with its group.",
          "優位性はAᵢ = (rᵢ − mean(r)) / (std(r) + ε)です。平均より良い回答は正、悪い回答は負になります。報酬は試行の評価、優位性はグループ内の相対評価です。"
        ]
      ],
      "figure": "loop"
    },
    {
      "title": [
        "四个回答，算一次优势",
        "Four answers, one calculation",
        "4つの回答で計算する"
      ],
      "paragraphs": [
        [
          "教学例题：3 × (4 + 2)。四次回答依次为 14、21、18、18，精确答案验证器给出奖励 [0, 0, 1, 1]。均值为 0.5；按总体标准差计算，标准差为 0.5。忽略很小的 ε，优势就是 [−1, −1, +1, +1]。",
          "Teaching example: 3 × (4 + 2). Four responses answer 14, 21, 18, and 18. An exact-answer verifier returns [0, 0, 1, 1]. The mean is 0.5 and the population standard deviation is 0.5. Ignoring a small ε, advantages are [−1, −1, +1, +1].",
          "説明用の問題は3 × (4 + 2)です。回答が14、21、18、18なら、正解検証器の報酬は[0, 0, 1, 1]。平均と母標準偏差はともに0.5なので、小さなεを無視すると優位性は[−1, −1, +1, +1]になります。"
        ],
        [
          "这还没有告诉模型“括号那一步错了”。在这个结果奖励版本里，同一回答的所有生成 token 共享一个优势值。参数更新会整体推动相应的采样路径；它不是逐步推理的正确性标注。",
          "This does not identify the incorrect arithmetic step. In this outcome-reward version, generated tokens in one response share an advantage. The update pushes on the sampled path as a whole; it is not a correctness label for each reasoning step.",
          "これだけでは計算のどの手順が誤ったかは分かりません。この結果報酬版では、同じ回答の生成tokenが1つの優位性を共有します。生成経路全体を更新する信号であり、推論の各段階への正誤ラベルではありません。"
        ]
      ],
      "figure": "example"
    },
    {
      "title": [
        "三个策略，各做什么",
        "Three policy roles",
        "3つの方策の役割"
      ],
      "paragraphs": [
        [
          "采样策略 πold 负责生成这批回答。当前策略 πθ 是正在更新的参数；在已经采样的 token 上，ρₜ = πθ / πold 比较新旧概率。参考策略 πref 用于 KL 约束，通常来自一个固定基线。采样策略会随训练刷新；参考策略承担的是另一个角色。",
          "The behavior policy πold generated this batch. The current policy πθ is being optimized; for sampled tokens, ρₜ = πθ / πold compares current and behavior probabilities. A reference policy πref supplies a KL anchor, often a fixed baseline. Refreshing the sampler and anchoring the policy are different roles.",
          "πoldはこのバッチを生成した方策、πθは更新中の方策です。サンプルしたtoken上でρₜ = πθ / πoldを計算します。πrefは通常、固定基準としてKL制約を与えます。サンプラーの更新と参照基準は別の役割です。"
        ],
        [
          "PPO 式裁剪限制代理目标继续奖励过大的概率比率变化，并不保证每个 token 的新概率被硬锁在区间内。若采用最大化目标，KL 惩罚应被减去；若写成最小化损失，则对应加上。",
          "PPO-style clipping limits the surrogate objective’s incentive for large ratio changes. It does not hard-constrain every new token probability to an interval. A KL penalty is subtracted from a maximization objective, or added to the corresponding minimized loss.",
          "PPO型のclipは、大きな確率比の変化を代理目的がさらに奨励することを抑えます。全tokenの確率を区間内に固定する制約ではありません。最大化目的ではKL罰則を引き、最小化損失では加えます。"
        ]
      ]
    },
    {
      "title": [
        "把目标函数的平均方式写完整",
        "Write the objective and its averaging explicitly",
        "目的関数と平均の取り方を明示する"
      ],
      "paragraphs": [
        [
          "记cᵢₜ=(q,oᵢ,<ₜ)，ρᵢₜ=πθ(oᵢₜ|cᵢₜ)/πold(oᵢₜ|cᵢₜ)，Lᵢ为有效回答长度。常见结果奖励版本最大化 J = E[(1/G)Σᵢ(1/Lᵢ)Σₜ{min(ρᵢₜÂᵢ, clip(ρᵢₜ,1−ε,1+ε)Âᵢ) − βkᵢₜ}]。括号明确KL在各有效token上计算，再按回答、按组平均；最小化实现使用−J。",
          "Let cᵢₜ=(q,oᵢ,<ₜ), ρᵢₜ=πθ(oᵢₜ|cᵢₜ)/πold(oᵢₜ|cᵢₜ), and Lᵢ be valid response length. A common outcome formulation maximizes J = E[(1/G)Σᵢ(1/Lᵢ)Σₜ{min(ρᵢₜÂᵢ, clip(ρᵢₜ,1−ε,1+ε)Âᵢ) − βkᵢₜ}]. KL is evaluated on valid tokens and averaged within responses and groups. A minimizing implementation uses −J.",
          "cᵢₜ=(q,oᵢ,<ₜ)、ρᵢₜ=πθ(oᵢₜ|cᵢₜ)/πold(oᵢₜ|cᵢₜ)、有効回答長をLᵢとします。一般的な結果報酬版はJ = E[(1/G)Σᵢ(1/Lᵢ)Σₜ{min(ρᵢₜÂᵢ, clip(ρᵢₜ,1−ε,1+ε)Âᵢ) − βkᵢₜ}]を最大化します。有効tokenでKLを計算し回答内・グループ内で平均します。最小化なら−Jです。"
        ],
        [
          "prompt、padding和工具环境输出不应被意外当成生成token训练。1/Lᵢ让每个回答先取均值：同优势时，短回答每token的直接系数更大；改成全batch有效token归一化或固定长度归一化，会改变样本权重。优势相同并不意味着各token梯度相同，它们的概率、上下文与参数导数都不同。",
          "Do not accidentally train prompt tokens, padding, or tool-environment outputs as policy-generated tokens. The 1/Lᵢ factor averages each response first, giving a larger direct per-token coefficient to shorter responses at equal advantage. Batch-token or fixed-length normalization changes weighting. Shared advantage does not imply identical token gradients: probabilities, contexts, and parameter derivatives differ.",
          "prompt、padding、環境が返したツール出力を生成tokenとして誤って学習しないようにします。1/Lᵢで回答ごとに平均すると、同じ優位性でも短い回答の各token係数が大きくなります。batch全tokenや固定長での正規化は重みを変えます。優位性が同じでも確率・文脈・導関数が違うので勾配は同じではありません。"
        ]
      ]
    },
    {
      "title": [
        "裁剪是单侧激励，不是概率围栏",
        "Clipping limits incentives in one direction",
        "clipは片側の誘因を抑える"
      ],
      "paragraphs": [
        [
          "取ε=0.2。正优势A=1时，ρ=1.5给出min(1.5,1.2)=1.2，继续抬高这个比率不再增加该项收益；但ρ=0.5时仍是0.5，不能把下降损失裁掉。负优势A=−1时，ρ=0.5给出min(−0.5,−0.8)=−0.8，进一步压低不再改善；ρ=1.5则仍是−1.5，错误增大概率仍受惩罚。",
          "Take ε=0.2. With A=1 and ρ=1.5, min(1.5,1.2)=1.2, removing further incentive from this term to increase the ratio. At ρ=0.5 the value remains 0.5; harmful decreases are not clipped away. With A=−1 and ρ=0.5, min(−0.5,−0.8)=−0.8, while ρ=1.5 remains −1.5. Increasing a negatively advantaged token is still penalized.",
          "ε=0.2とします。A=1、ρ=1.5ならmin(1.5,1.2)=1.2で、これ以上増やす誘因を止めます。ρ=0.5は0.5のままで、悪い方向の変化は消しません。A=−1、ρ=0.5ならmin(−0.5,−0.8)=−0.8、ρ=1.5なら−1.5です。負の優位性を持つtokenの確率上昇は罰せられます。"
        ],
        [
          "共享参数、其他样本和KL仍会改变概率，因此不能声称所有ρ被硬性限制在[0.8,1.2]。ρ是同一已采样前缀处的token条件概率比，不是自动修正任意陈旧序列分布的完整轨迹重要性权重。采样器与训练器不同步时，要记录真实行为logprob、更新次数与样本年龄。",
          "Shared parameters, other samples, and KL can still change probabilities; ratios are not hard-constrained to [0.8,1.2]. Each ρ compares token probabilities at the same sampled prefix. It is not a full-trajectory importance weight that automatically corrects arbitrary stale data. With asynchronous sampling, record actual behavior log probabilities, update counts, and sample age.",
          "共有パラメータ、他標本、KLでも確率は変わるため、全ρが[0.8,1.2]内に固定されるわけではありません。ρは同じ採取済み前置文脈でのtoken確率比で、任意の古い系列分布を補正する完全軌跡重みではありません。非同期生成では実際のbehavior logprob、更新回数、標本の古さを記録します。"
        ]
      ]
    },
    {
      "title": [
        "KL 约束的方向与采样分布都要明确",
        "KL needs a direction and a sampling distribution",
        "KLの向きと標本分布を明確にする"
      ],
      "paragraphs": [
        [
          "固定上下文，令x=πref(a)/πθ(a)。常用k3=x−log x−1，非负且两概率相同时为0。若a确实从πθ抽样并有适当支持集，E[x]=1，于是E[k3]=E[log(πθ/πref)]=KL(πθ||πref)。这是估计值的推导；当样本来自πold，或对估计式怎样反传时，不能不查采样修正就沿用“无偏”的说法。",
          "At a fixed context, set x=πref(a)/πθ(a). The common k3=x−log x−1 is nonnegative and vanishes at equality. If a is sampled from πθ with appropriate support, E[x]=1, so E[k3]=E[log(πθ/πref)]=KL(πθ||πref). This derives an expectation identity. Sampling from πold, or differentiating an estimator, requires separate reasoning rather than an automatic claim of unbiasedness.",
          "文脈を固定しx=πref(a)/πθ(a)とすると、k3=x−log x−1は非負で等しい確率なら0です。適切な支持の下でaをπθから引けばE[x]=1なので、E[k3]=E[log(πθ/πref)]=KL(πθ||πref)です。これは期待値の等式です。πoldからの標本や推定式の微分にもそのまま不偏性があるとは言えません。"
        ],
        [
          "old控制相对本批采样策略的更新，ref约束相对参考的漂移，二者不能互换。固定SFT参考是常见配置，但原始迭代GRPO算法也在外层iteration刷新参考；应记录实际冻结与刷新周期。β太小或太大分别可能约束不足或压过奖励信号，其效果必须从奖励、KL、熵和独立任务表现共同判断。",
          "The old policy controls updates relative to this batch’s sampler; the reference anchors drift against another baseline. They are not interchangeable. A fixed SFT reference is common, but the original iterative GRPO algorithm also refreshes it at outer iterations. Record the actual schedule. Assess β using reward, KL, entropy, and independent task performance rather than treating stronger regularization as universally better.",
          "oldはバッチの生成方策に対する更新、refは参照からの逸脱を制御し、交換できません。固定SFT参照は一般的ですが、元の反復GRPOは外側iterationで参照を更新します。実際の周期を記録します。βは報酬・KL・エントロピー・独立課題の性能を合わせて判断し、強い制約を常に良いとしません。"
        ]
      ]
    },
    {
      "title": [
        "一次更新需要冻结哪些量",
        "What stays fixed during one update",
        "1回の更新で固定する量"
      ],
      "paragraphs": [
        [
          "先抽问题批次，冻结本批πold，按明确temperature/top-p和长度上限每题生成G答；保存回答、有效mask、行为logprob和奖励理由。随后固定奖励及优势，计算当前logprob、参考logprob、比率和目标，做若干minibatch更新；到约定边界再刷新采样策略。重复使用同批数据时分母仍应对应真正生成它的策略。",
          "Sample prompts, freeze πold for the batch, and generate G responses per prompt with explicit temperature, top-p, and length limits. Save responses, valid masks, behavior log probabilities, and reward evidence. Hold rewards and advantages fixed while recomputing current and reference log probabilities and optimizing minibatches. Refresh the sampler at defined boundaries. Reusing a batch requires its denominator to retain the policy that actually generated it.",
          "問題を選び、このバッチのπoldを固定して、temperature・top-p・長さ上限を明示し各G回答を生成します。回答、mask、behavior logprob、報酬根拠を保存します。報酬と優位性を固定して現在・参照logprobと目的を計算し更新します。所定の境界で生成方策を更新し、バッチ再利用中の分母は実際の生成方策に保ちます。"
        ],
        [
          "G决定同题比较的采样预算，优化epoch决定同一数据被训练几次，二者不是同一个参数。G增加不保证奖励有差异；反复优化旧样本也不等于得到新的探索。对数概率、mask与权重可以分开缓存，算法中的三个策略角色不必对应三套始终驻留显存的模型副本。",
          "G sets the within-prompt sampling budget; optimization epochs set how often that data is trained on. They are different controls. Larger G does not guarantee reward diversity, and repeatedly optimizing old samples does not create new exploration. Cached log probabilities, masks, and weights can represent policy roles without three model copies always residing on the accelerator.",
          "Gは同じ問題の生成予算、最適化epochは同じデータを学習する回数で、別の制御です。Gを増やしても報酬差は保証されず、旧標本の反復学習は新しい探索ではありません。logprobや重みを分けて保存でき、3つの役割が常駐する3モデルを意味するわけではありません。"
        ]
      ]
    },
    {
      "title": [
        "什么情况下，组内比较没有信号",
        "When group comparison is uninformative",
        "グループ比較から信号が得られない場合"
      ],
      "paragraphs": [
        [
          "如果四条回答奖励相同，减去均值之后全为零。这组数据没有结果奖励的相对优势信号；若还启用 KL 或其他目标，它们仍可能产生梯度。全对与全错也需要分开看：前者可能太容易，后者可能超出当前采样能力。",
          "Equal rewards produce zero centered advantages. The group has no relative outcome-reward signal, although KL or other enabled objectives may still produce gradients. All-correct and all-wrong groups also mean different things: a task may be easy, or success may be beyond current sampling capability.",
          "報酬がすべて同じなら中心化後の優位性はゼロです。結果報酬の相対信号はなくても、KLなど別の目的から勾配は生じ得ます。全正解は簡単すぎる可能性、全不正解は現在の生成能力では成功に届かない可能性があります。"
        ],
        [
          "训练是在修改参数，让未来的单次生成更有希望成功。Best-of-K 则是在推理时多生成、再挑答案。评测时应固定每题采样次数，同时报告单次准确率和预算，才能看清收益来自模型还是额外尝试。",
          "Training changes parameters so future single attempts can improve. Best-of-K instead samples extra answers at inference and selects one. Fix sampling counts in evaluation and report single-attempt accuracy alongside budget to distinguish model improvement from extra attempts.",
          "訓練はパラメータを変えて将来の1回の生成を改善します。Best-of-Kは推論時に追加生成して選ぶ方法です。評価では試行数を固定し、単発の正解率と予算を併記して区別します。"
        ]
      ]
    },
    {
      "title": [
        "读懂训练曲线，要同时看奖励和样本构成",
        "Read reward curves together with sample composition",
        "報酬曲線と標本構成を一緒に見る"
      ],
      "paragraphs": [
        [
          "二值奖励组若成功比例为p，忽略稳定项时σ=√(p(1−p))，成功答案A+=√((1−p)/p)，失败答案A−=−√(p/(1−p))。例如一成功三失败时，A+=√3、A−=−1/√3。它说明标准化会改变不同难度组的尺度；p=0或1必须单独处理，不能将分母为0写成有效信号。",
          "For binary rewards with success fraction p, ignoring stabilization gives σ=√(p(1−p)), A+=√((1−p)/p), and A−=−√(p/(1−p)). One success among four yields A+=√3 and A−=−1/√3. Standardization therefore changes scales across groups of differing difficulty. p=0 or 1 requires explicit handling rather than treating division by zero as a signal.",
          "二値報酬の成功割合pなら、安定化項を無視してσ=√(p(1−p))、A+=√((1−p)/p)、A−=−√(p/(1−p))です。4回答中1成功なら√3と−1/√3になります。標準化は難易度の異なるグループ間の尺度を変えます。p=0、1ではゼロ除算を有効信号とせず、別処理が必要です。"
        ],
        [
          "因此至少一起看：原始奖励、全对/全错/混合组比例、有效样本与token数、回答长度/截断、优势尺度、ratio/clip fraction、KL、熵和独立评测。过滤全同奖励组会改变训练问题分布，要单独报过滤率。这里讲结果奖励版本；原始GRPO也讨论过程奖励，不能把“每答所有token同优势”扩成所有GRPO的定义。不同变体的归一化和裁剪选择另行比较，不从名称推断实现。",
          "Track raw reward; all-correct, all-wrong, and mixed groups; valid samples and tokens; length and truncation; advantage scale; ratios and clip fraction; KL; entropy; and independent evaluation. Filtering equal-reward groups changes the training distribution, so report its rate. This article uses outcome rewards; original GRPO also discusses process supervision. Shared response-level advantage is not a definition of every GRPO variant. Compare implementation choices explicitly.",
          "原報酬、全正解・全不正解・混合グループ率、有効標本とtoken、長さと切詰め、優位性尺度、ratioとclip率、KL、エントロピー、独立評価を確認します。同報酬グループの除外は問題分布を変えるので率も報告します。ここでは結果報酬版を説明しましたが、元のGRPOには過程報酬もあります。回答全tokenの同じ優位性を全変種の定義にしないことが大切です。"
        ]
      ]
    }
  ],
  "figures": [
    {
      "id": "loop",
      "title": [
        "GRPO 训练循环",
        "The GRPO training loop",
        "GRPOの学習ループ"
      ],
      "caption": [
        "一次组内比较产生训练信号；更新后的策略用于后续采样。参考策略只提供约束，不参与选出“最佳答案”。",
        "Group comparison produces a learning signal; the updated policy samples future batches. The reference is an anchor, not an answer selector.",
        "グループ比較から学習信号を得て、更新後の方策で次を生成します。参照方策は回答選択器ではありません。"
      ],
      "rows": [
        {
          "nodes": [
            [
              "同一问题",
              "One prompt",
              "同じ問題"
            ],
            [
              "采样 G 条",
              "Sample G responses",
              "G個の回答を生成"
            ],
            [
              "验证奖励",
              "Verify rewards",
              "報酬を検証"
            ]
          ]
        },
        {
          "nodes": [
            [
              "组内优势",
              "Group advantages",
              "グループ内の優位性"
            ],
            [
              "裁剪目标 + KL",
              "Clipped objective + KL",
              "clip目的＋KL"
            ],
            [
              "更新策略",
              "Update policy",
              "方策を更新"
            ]
          ]
        }
      ],
      "bridges": [
        [
          0,
          2,
          1,
          0
        ]
      ],
      "loop": [
        1,
        2,
        0,
        1
      ]
    },
    {
      "id": "example",
      "title": [
        "同题四答：相对奖励",
        "Four answers: relative reward",
        "同じ問題の4回答：相対報酬"
      ],
      "caption": [
        "教学计算，采用总体标准差；数值不是训练实验结果。",
        "Worked example using population standard deviation; these are not training results.",
        "母標準偏差を用いた計算例です。訓練実験の結果ではありません。"
      ],
      "rows": [
        {
          "connect": false,
          "nodes": [
            [
              "14",
              "14",
              "14"
            ],
            [
              "21",
              "21",
              "21"
            ],
            [
              "18",
              "18",
              "18"
            ],
            [
              "18",
              "18",
              "18"
            ]
          ],
          "sub": [
            [
              "r = 0 · A = −1",
              "r = 0 · A = −1",
              "r = 0 · A = −1"
            ],
            [
              "r = 0 · A = −1",
              "r = 0 · A = −1",
              "r = 0 · A = −1"
            ],
            [
              "r = 1 · A = +1",
              "r = 1 · A = +1",
              "r = 1 · A = +1"
            ],
            [
              "r = 1 · A = +1",
              "r = 1 · A = +1",
              "r = 1 · A = +1"
            ]
          ]
        }
      ]
    }
  ],
  "references": [
    [
      "DeepSeekMath · GRPO",
      "https://arxiv.org/abs/2402.03300"
    ],
    [
      "Proximal Policy Optimization Algorithms",
      "https://arxiv.org/abs/1707.06347"
    ],
    [
      "Understanding R1-Zero-Like Training",
      "https://arxiv.org/abs/2503.20783"
    ],
    [
      "Schulman et al. · Generalized Advantage Estimation",
      "https://arxiv.org/abs/1506.02438"
    ],
    [
      "Schulman · Approximating KL Divergence",
      "https://joschu.net/blog/kl-approx.html"
    ]
  ],
  "related": [
    "grpo-design-choices",
    "on-policy-distillation",
    "foundation-model-data"
  ]
}
