Learning & methods · November 2026学习与方法 · 2026 年 11 月

Learn to act in imagination, or imagine before acting?在想象中学会行动,还是行动前再想一遍?

DreamerV3 and V-JEPA 2-AC both predict the future. One uses predictions to train a policy; the other searches with them before each action.DreamerV3 与 V-JEPA 2-AC 都预测未来,却把预测放在不同位置:前者用它训练策略,后者用它在每次行动前搜索。

1 · What happens after predicting the future?1 · 预测未来,接下来做什么?

一个机械臂面对桌上的杯子。它可以先在内部试想几种移动,判断哪一种更接近目标;也可以在训练时反复试想,把经验压进一个直接输出动作的网络。两种做法都有“想象”,但想象发生在哪里,决定了系统需要什么数据、怎样指定目标,以及每次行动要等多久。

DreamerV3 把世界模型作为训练场。真实交互产生观测、动作与奖励,模型学习它们的关系,再生成潜状态轨迹供 actor 与 critic 学习。actor 是选动作的策略,critic 是估计未来回报的网络。执行时,系统更新当前潜状态,随后从 actor 取动作,不为每个动作重新搜索许多未来。

V-JEPA 2-AC 则先训练好表征和动作条件预测器;使用时,给当前观测与目标照片,在线比较候选动作造成的未来,再执行选中的动作。它把更多决策工作留在部署阶段。杯子是帮助理解的贯穿例子,不是两篇论文共同做过的对照实验:DreamerV3 的证据来自多类控制基准,V-JEPA 2-AC 的证据包含真实机械臂。

An arm faces a cup on a table. It could imagine several movements and choose the one that gets closest to its goal. Or it could repeatedly imagine movements during training, storing what it learns in a network that directly chooses actions. Both approaches use imagination. Where that imagination enters the system determines the data it needs, how goals are specified, and how long each decision takes.

DreamerV3 uses its world model as a training environment. Real interaction supplies observations, actions and rewards. The model learns their relationships and generates latent trajectories for an actor and a critic: the actor chooses actions, while the critic estimates future returns. During execution, the system updates its current latent state and samples an action from the actor, without searching through many futures for each decision.

V-JEPA 2-AC first learns representations and an action-conditioned predictor. At deployment, it takes a current observation and a goal image, compares the futures implied by candidate actions, and executes the selected action. More of the decision work happens at deployment. The cup is a running teaching example, not a shared experiment: DreamerV3 is evaluated across control benchmarks, whereas V-JEPA 2-AC includes real-arm experiments.

2 · Dreamer: turn imagined experience into a policy2 · Dreamer:把未来经验写进策略

Dreamer 的内部状态由两部分组成:循环记忆 ht 保存历史,随机潜变量 zt 表达当前信息。遇到真实观测 xt 时,编码器形成后验;进入想象后,没有新照片,模型只能从历史预测先验。这一切换让它能“闭眼”向前走。

ht=fϕ(ht−1,zt−1,at−1),zt~qϕ(zt∣ht,xt),z^t~pϕ(zt∣ht).

at−1 是上一步动作,ϕ 是世界模型参数;q 看到了当前真实观测,p 没有。训练让先验追上后验,同时要求潜状态能重建观测、预测奖励和回合是否继续。因此,隐状态不是任意的压缩码,它必须保留支持这些预测的信息。

Dreamer has a recurrent memory ht and a stochastic latent variable zt. When a real observation xt arrives, the encoder produces a posterior. During imagination, no new image arrives, so the model predicts a prior from its history. This switch lets it advance with its eyes closed.

ht=fϕ(ht−1,zt−1,at−1),zt~qϕ(zt∣ht,xt),z^t~pϕ(zt∣ht).

Here at−1 is the previous action and ϕ denotes world-model parameters. The posterior q sees the current observation; the prior p does not. Training brings the prior toward the posterior while requiring the state to reconstruct observations, predict rewards, and predict continuation. The latent code must retain information useful for these predictions.

Static diagram静态图Dreamer: interaction and imagined training
Real replay trains the world model and supplies starting states for imagination. Imagination improves the actor; new interaction keeps updating the model. Arrows show data, model and behavior dependencies, not lookahead search before every executed action.真实回放既训练世界模型,也给想象提供起点。想象改善 actor;新的真实交互继续校正模型。箭头表示训练数据、模型或行为的依赖,不表示每次执行动作都先搜索未来。

训练从真实回放片段编码出的状态出发,让 actor 在模型中连续行动,论文使用 16 步想象轨迹。模型给出预测奖励,critic 用自举估计补上有限视野以外的回报,actor 再提高较好动作的概率。这里不是只用固定数据集的 offline RL:系统持续与环境交互,更新回放与世界模型。critic 也有真实回放轨迹上的辅助价值损失,不能说它完全不接触真实数据。

用杯子例子理解:假设世界模型预计某种接近方式能得到更高回报,多次训练后,actor 在相似状态下更倾向于那种动作。真正执行时不必再比较所有接近方式;过去的想象已经改变了策略参数。

Training starts from states encoded from real replay sequences. The actor acts inside the model for 16 imagined steps in the paper. Predicted rewards and bootstrapped critic estimates provide returns, including an estimate beyond the finite horizon. The actor then increases the probability of better actions. This is not offline RL on a fixed dataset: interaction continues to update replay and the world model. The critic also has an auxiliary value loss on real replay trajectories, so it is inaccurate to say it never uses real data.

In the cup example, suppose the world model predicts higher return for a particular approach. Repeated training makes the actor prefer it in similar states. At execution, it need not compare every approach again: earlier imagination has already changed its parameters.

3 · How can one algorithm handle different reward scales?3 · 同一算法怎样面对不同奖励尺度?

DreamerV3 的贡献还在于让这个循环跨任务稳定工作。若一个任务奖励为 1,另一个为 1000,直接做平方误差回归时,大数值目标容易主导更新。V3 使用保留正负号的对称对数压缩,并对奖励、回报采用分桶分布预测,减少数值量级对优化的干扰。

symlog(y)=sign(y)ln(1+|y|).

例如教学数值 y=−100,0,100 分别变成约 −4.615,0,4.615。大的正负数都被压缩,小数值附近仍接近原值。对奖励和回报的 twohot 标签,则把一个数分配给相邻两个桶:若相邻桶位置是 10 与 20,目标 12 的权重是 0.8 与 0.2。用这些软标签训练交叉熵,梯度不会像原始平方误差那样直接随目标绝对量级放大。

actor 还用回报第 95 与第 5 百分位的差来估计尺度,只缩小大回报,不把微小稀疏信号强行放大。表示学习中的 KL 项设有 free-bits 下限,避免一味追求容易预测而丢失观测信息。这些设计共同支撑固定超参数的表现,并非某一个归一化公式单独“解决通用强化学习”。

论文在 150 多项任务上复用的是算法配置;不同任务仍分别训练模型。它不等于同一个预训练模型零样本掌握所有环境。

DreamerV3 also makes this loop stable across tasks. If rewards are around 1 in one task and 1,000 in another, large targets can dominate updates under direct squared-error regression. V3 uses a sign-preserving symmetric logarithm, and predicts distributions over bins for rewards and returns, reducing sensitivity to numerical scale.

symlog(y)=sign(y)ln(1+|y|).

For the teaching values y=−100,0,100, the transformed values are approximately −4.615,0,4.615. Large magnitudes shrink on both sides of zero, while small values remain close to their original scale. A twohot target assigns a number to its two adjacent bins. With bins at 10 and 20, a target of 12 receives weights 0.8 and 0.2. Cross-entropy training on these soft targets does not directly amplify gradients with the absolute target magnitude in the way raw squared error does.

The actor estimates a return scale from the difference between the 95th and 5th percentiles. It scales down large returns without amplifying tiny sparse signals. Free-bits floors in the representation-learning KL terms prevent pressure for predictability from simply removing observation information. Together, these choices support the fixed-hyperparameter results; no single normalization formula solves general reinforcement learning.

What is reused across more than 150 tasks is the algorithm configuration. Models are trained separately for different tasks. This is not one pretrained model mastering every environment zero-shot.

4 · V-JEPA: judge imagined futures against a goal image4 · V-JEPA:用目标照片来评价每一个未来

V-JEPA 2 的第一阶段从海量视频学习表征:遮住一部分视频,让预测器补出这些位置的特征,目标由滑动平均编码器提供,而不是要求补出每个像素。这个阶段没有机器人动作。第二阶段冻结编码器,用不足 62 小时的 DROID 视频及末端执行器状态,训练动作条件预测器。这里的“无标注”指没有奖励、任务类别和成功标签;每帧仍有机器人状态,动作由相邻状态差分得到。

后训练同时做两件事:在真实历史上预测下一状态,以及把自己的预测回喂后再预测。前者学习单步关系,后者针对连续使用模型时的误差累积。它不需要先学习“拿起杯子值多少分”,因为使用者提供的是杯子处于目标位置时的照片。

V-JEPA 2 first learns representations from large-scale video. It masks parts of a clip and predicts their features, using a moving-average encoder to supply targets rather than reconstructing every pixel. This stage has no robot actions. A second stage freezes the encoder and trains an action-conditioned predictor with under 62 hours of DROID video and end-effector states. “Unlabeled” means no reward, task-category or success labels: state measurements still accompany each frame, and actions are derived from state differences.

Post-training combines next-state prediction from real histories with prediction that feeds the model's own outputs back in. The first learns one-step relationships; the second addresses error accumulation during repeated prediction. The system need not first learn how many reward points lifting a cup deserves, because the user supplies an image of the desired state.

Static diagram静态图V-JEPA: learn prediction, then plan online
The upper row trains the model; the lower row is deployment-time MPC. Deployment searches actions rather than updating model weights. A new observation enters the next search after the first action is executed.上排是模型训练,下排是部署时的 MPC。部署搜索动作而非更新模型权重;执行第一步之后,新观测重新进入搜索。
a1:H*=\argmina1:H‖Pϕ(a1:H;st,zt)−zg‖1,zt=E(xt),zg=E(xg).

E 是冻结编码器,Pϕ 是动作条件预测器,st 是末端状态,xg 是目标图像,H 是规划步数。L1 距离把对应特征元素的绝对差加起来。优化变量是候选动作序列,不是模型参数。论文用交叉熵方法 CEM:采样动作,保留距离较小的候选,用它们更新采样分布,再重复。这个 CEM 是搜索方法,与监督学习里的交叉熵损失不是同一个使用场景。

为看清目标函数,设一个仅有两维的教学特征目标为 (1,1)。候选 A 预测 (0.8,1.2),距离为 0.4;候选 B 预测 (0.2,0.9),距离为 0.9,因此选 A。真实模型的特征高维且没有“杯子坐标”这样的直接含义。选出序列后只执行第一步,再拍摄新画面、重新规划;这个闭环允许现实纠正想象。

a1:H*=\argmina1:H‖Pϕ(a1:H;st,zt)−zg‖1,zt=E(xt),zg=E(xg).

E is the frozen encoder, Pϕ the action-conditioned predictor, st the end-effector state, xg the goal image, and H the planning horizon. The L1 distance sums absolute differences between corresponding feature elements. Candidate actions are optimized, not model parameters. The paper uses the cross-entropy method, CEM: sample actions, retain candidates with lower distance, update the sampling distribution from those candidates, and repeat. CEM is a search method; its role here differs from cross-entropy as a supervised-learning loss.

A two-dimensional teaching example makes the objective concrete. Let the target feature be (1,1). Candidate A predicts (0.8,1.2), at distance 0.4; B predicts (0.2,0.9), at distance 0.9, so A is preferred. Actual features are high-dimensional and are not directly interpretable cup coordinates. After selecting a sequence, the controller executes only its first action, observes again, and replans. This closed loop lets reality correct imagination.

5 · Compare the decision mechanisms, not a single leaderboard5 · 比较的核心是决策方式,不是一张总榜

问题DreamerV3V-JEPA 2-AC
想象主要改变什么训练时的策略参数当前候选动作的排序
目标从哪来环境奖励与回报目标图像的特征
执行时怎样选动作actor 前向计算MPC 搜索,执行第一步
模型错误的主要入口影响策略学到的行为影响本次搜索的评分

V-JEPA 2 论文的真机实验提供了具体代价:在单张 RTX 4090 上,每次动作规划约 16 秒;同文适配的 Cosmos 基线约 4 分钟。两者使用的候选数量分别为 800 与 80,均做 10 轮细化,表 3 的规划时域为 1。这是特定实现和配置的比较,不是世界模型家族的普遍速度定律,更不是实时控制。

同文两间实验室的杯子抓取放置平均成功率为 80%,但抓盒子只有 25%;每个实验室、每种任务只做 10 次尝试。抓取放置还使用人工提供的三张子目标图,相机位置也经过人工挑选。结果说明表征预测可以支撑这些闭环任务,没有说明单张最终目标能解决任意长任务。

读两篇论文时,最有用的选择问题是:任务是否有可用奖励,是否希望快速直接执行策略,是否经常需要更换目标,以及能为每次动作支付多少搜索时间。关于明确地图上的多机器人分配和避碰,可接着读机器人感知与规划;那里的离散路径约束,与这里从观测学习动态、预测候选后果,处理的是不同层次的问题。

QuestionDreamerV3V-JEPA 2-AC
What does imagination primarily change?Policy parameters during trainingThe ranking of current candidate actions
Where does the objective come from?Environment rewards and returnsFeatures of a goal image
How is an action chosen at execution?An actor forward passMPC search; execute the first action
Where do model errors enter?The behavior the policy learnsThe scores in the current search

The V-JEPA 2 robot experiments give a concrete cost: around 16 seconds to plan an action on one RTX 4090, versus around four minutes for the paper's adapted Cosmos baseline. They use 800 and 80 candidates respectively, both with ten refinement rounds and a horizon of one in Table 3. This comparison concerns particular implementations and configurations. It is neither a universal speed law for model families nor real-time control.

Across two labs, cup pick-and-place averages 80% success, whereas box grasping averages 25%. Each lab runs only ten attempts per task. Pick-and-place also uses three manually supplied goal images, and camera positions were selected manually. These results establish that feature prediction can support these closed-loop tasks; they do not establish arbitrary long tasks from one final goal image.

The useful design questions are whether rewards are available, whether fast direct policy execution is needed, how frequently the goal changes, and how much search time each action can afford. For assignment and collision avoidance on explicit maps, continue with robot perception and planning. Its discrete route constraints address a different layer from learning dynamics from observations and predicting action consequences here.

Continue reading继续阅读

  1. DreamerV3 · arXiv v2 (2024)
  2. V-JEPA 2 · arXiv v1 (2025)
  3. V-JEPA 2 · official implementation
  4. Original study notes原始学习笔记 · dreamerv3
  5. Original study notes原始学习笔记 · v jepa 2

Related Writings相关 Writings