Actor

Policy θ

Successes

Verified

Rewrite & verify

Fresh execution

SFT

Actor only

Rollouts

Actor ↔ terminal

Verification

FDR / SPR → R

GAE / TB-GAE

From R and V

Actor update

PPO with A

Critic

Retained weights

Values V

At each prefix

Critic update

Regress to Ĝ
A · RSR: successful experience → new demonstrations
B · RPR: execution outcomes → parameter updates
Actor after supervised tuning
Reward R
Generated prefixes in each trajectory
V
A
Ĝ
Retain Actor for the next RSR round
Retain Critic for the next RPR phase
Each RPR phase: 4 Critic-only updates, then joint training