SFT update

RSR updates only the Actor, using verified public trajectories with private guidance removed.

Frozen

During the first 4 updates of each RPR stage, Actor parameters are fixed. The Actor runs current tasks to calibrate the Critic.

PPO update

After calibration, the Actor performs PPO updates using advantages while the Critic continues to fit return targets.

Retain weights

Critic weights remain unchanged during RSR and pass directly to the next RPR stage.

Value fitting

First, 4 Critic-only updates adapt to the policy and reward using current Actor trajectories. A separate one-epoch SWR value warm-up precedes the first entry into RPR-1.

Value fitting

Return-target fitting continues during joint training. Each RPR stage has one calibration period of 4 updates, at its start.

RSR · SFT

RPR · 4-step calibration

RPR · Joint training

Actor

Critic

Green: update parameters White: retain parameters