Learning & methods · November 2026学习与方法 · 2026 年 11 月
When does a generated video become a useful world model?生成的视频,什么时候才算可用的世界模型?
Cosmos connects video compression, conditioning and physical evaluation. Visual realism, geometric consistency and correct action consequences impose different requirements.从 Cosmos 的视频压缩、条件接口和物理评测出发,分清画面逼真、几何一致与行动后果正确这三种要求。
1 · One ball, three different questions1 · 同一颗小球,三个不同的问题
一颗小球沿斜坡滚下。生成视频可能色彩自然、阴影柔和,但球的速度突然减慢,甚至穿过挡板。对内容创作而言,这可能只是一处瑕疵;对用视频预测机器人操作后果的系统而言,它会改变动作选择。世界模型的“好”,必须连着用途来定义。
Cosmos 论文提出一套可适配的世界基础模型平台。它把历史画面和条件输入映射为后续画面,条件可以是文字、相机轨迹,也可以在后训练后成为机器人动作。本文讨论 2025 年报告的 Cosmos-Predict1 路线,而不是把后来所有同名产品的能力放进一个模型里。
需要分别问三件事:画面是否像真实视频;不同视角能否对应同一个场景;给定动作与初始状态后,后续运动是否正确。第一项好不保证第二项,前两项好也不保证第三项。Cosmos 有意义之处,不仅是给出能生成视频的模型,还把压缩、数据处理、条件适配和评测做成可分别检查的链条。
A ball rolls down a ramp. A generated video may have natural colors and soft shadows, yet the ball abruptly slows or passes through a barrier. For content creation, that might be one visual flaw. For a system predicting the consequences of robot actions, it can change which action is selected. A good world model must be defined in relation to its use.
The Cosmos paper presents an adaptable world foundation model platform. It maps past frames and conditions to future frames. Conditions may be text or camera trajectories, or, after post-training, robot actions. This article covers the Cosmos-Predict1 approach in the 2025 report rather than combining every later Cosmos product into one model.
Three questions are distinct: does the result look like real video; can different views belong to one scene; and, given an action and initial state, does the subsequent motion match? Success on the first does not guarantee the second, and both together do not guarantee the third. Cosmos matters not only for generating video, but for exposing compression, data preparation, conditioning and evaluation as parts that can be examined separately.
2 · Decide what information must survive2 · 先决定哪些信息值得保留
训练视频模型之前,要先排除“剪辑造成的运动”。厨房切到街道,是镜头切换,不是厨房瞬间变成街道。Cosmos 的数据管线先按镜头切片,再做运动、质量、覆盖文字和视频类型过滤,补充描述、去重并分片。它面向物理 AI 配置数据分布,因此去除动画和游戏画面等选择,与 Genie 从游戏录像学习交互的选择不同。
下一道瓶颈是 tokenizer。它把像素视频编码为较小的时空网格;模型在这个网格里生成,最后解码回画面。连续表示用于扩散模型,离散码字用于自回归模型。二者不是同一套 token 换个名字:前者预测连续潜变量,后者预测有限词表中的类别。
Before training a video model, editing must be separated from motion. A cut from a kitchen to a street is not a kitchen physically turning into a street. Cosmos splits videos into shots, filters motion, visual quality, overlaid text and video types, and then captions, deduplicates and shards the data. Its Physical AI distribution excludes content such as animation and game footage, unlike Genie's choice to learn interaction from gameplay recordings.
The next bottleneck is the tokenizer. It encodes pixel video into a smaller space-time grid, where the model generates before decoding back to images. Continuous representations serve diffusion models; discrete codes serve autoregressive models. These are not identical tokens with different names: one models continuous latent values, the other predicts categories from a finite vocabulary.
Static diagram静态图
报告中的主要连续视频配置沿时间、高度和宽度分别压缩 8、8、8 倍;主要离散配置为 8、16、16。以一个理想化的 时空块说明,两者分别剩 与 个网格位置。这里忽略真实实现对首帧的单独处理,只比较网格压缩;并不表示字节数或运行时间恰好减少四倍,因为表示维度和生成算法也不同。
更强压缩能减少模型要处理的位置,但可能抹去手指接触、细杆或小物体。后面的世界模型无法可靠恢复每一项已经丢失的信息。报告还用扩散解码器把离散表示转换为连续表示以改善画质;清晰度恢复与真实运动恢复仍是两个问题。
The report's main continuous video configuration compresses time, height and width by 8, 8 and 8; the main discrete configuration uses 8, 16 and 16. For an idealized space-time block, these leave and grid positions. This example ignores the implementation's special first-frame treatment and compares only grid compression. Bytes or runtime need not fall by exactly a factor of four, since representation dimensions and generation algorithms also differ.
Stronger compression reduces the positions a model processes, but may remove finger contacts, thin rods or small objects. The world model cannot reliably recover every detail already lost. The report also uses a diffusion decoder to turn discrete representations into continuous ones for better visual quality. Recovering sharpness and recovering correct motion remain separate goals.
3 · A camera instruction is different from a camera ray3 · 一句话运镜,和一条相机射线,信息量不同
预训练让模型见过许多运动,但不会自动赋予它某台机器人的动作坐标系。后训练把具体控制量接进去:相机条件模型接受位姿,机器人模型接受末端位移与夹爪变化,多视角驾驶模型还接收机位和轨迹。条件接口明确了用户能改变什么,而不仅是生成之后检查画面是否合意。
以相机为例,“向右绕一点”可以对应不同距离、角度和速度。精确位姿则可以转成逐像素的射线。Cosmos 的相机条件示例采用 Plücker 表示,将射线方向和位置相关的矩拼接进潜表示。
是世界坐标中的相机中心, 是同一坐标系中的单位射线方向,叉积得到 。例如方向都为 时,相机中心 对应零矩;中心平移到 后,矩变为 。两条射线朝向相同但位于不同位置,六维表示保留了这个差别。
机器人动作条件同样重要:训练数据不只给“抓杯子”的一句描述,还给逐步末端位移、姿态和夹爪控制量。模型要学习这些量与画面变化的关系。接上条件只提供可表达的接口,并不形成几何或物理正确性的硬约束;是否真的遵从,需要在指定动作下检验预测。
Pretraining exposes a model to many motions, but does not automatically teach it the action coordinate system of a particular robot. Post-training adds concrete controls: camera-conditioned models receive poses, manipulation models receive end-effector and gripper changes, and driving models receive view identities and trajectories. The conditioning interface specifies what the user can change rather than merely inspecting whether an output looks desirable.
For a camera, “orbit a little to the right” can mean different distances, angles and speeds. Precise poses can instead become per-pixel rays. Cosmos's camera-conditioning example uses a Plücker representation, concatenating ray direction and a position-related moment with latent representations.
is the camera center in world coordinates, a unit ray direction in the same coordinates, and the cross product gives . For direction , a center at gives zero moment; moving the center to gives . The rays point the same way but occupy different positions, and the six-dimensional representation preserves that distinction.
Robot-action conditioning matters similarly. Instead of only “grasp the cup,” training supplies stepwise end-effector displacement, orientation and gripper controls. The model must learn how those quantities relate to image changes. Adding an interface does not impose hard geometric or physical correctness. Compliance must be tested under specified actions.
4 · What changes when more frames are observed?4 · 多看几帧,究竟改变了什么?
一张小球照片几乎不能确定速度。球可能向左、向右,也可能静止;这些状态可以共享同一瞬间的位置。补上连续历史帧,才能从位置变化推断速度,进而预测后续。报告用 800 段合成刚体视频检验预测,场景包括自由落体、斜面、堆叠、多米诺等。评估使用总长 33 帧的序列,像素指标 PSNR、SSIM 排除条件帧:给 1 帧时评估其余 32 帧,给 9 帧时评估其余 24 帧。
A single image of a ball barely determines velocity. It could be moving left, moving right, or stationary while occupying the same position at that instant. Consecutive observations reveal position changes and help predict what follows. The report tests prediction with 800 synthetic rigid-body videos, including free fall, ramps, stacks and dominoes. Evaluation uses sequences of 33 frames in total. The pixel metrics PSNR and SSIM exclude conditioning frames: one observed frame leaves 32 evaluated frames, while nine leave 24.
Static diagram静态图
| Video2World 配置 | 历史条件 | PSNR (dB) ↑ | 对象掩码 IoU ↑ |
|---|---|---|---|
| 7B | 文字 + 1 帧 | 17.34 | 0.332 |
| 7B | 文字 + 9 帧 | 21.06 | 0.592 |
| 14B | 文字 + 9 帧 | 20.21 | 0.598 |
这是原文表 20 的节选。PSNR 衡量像素预测与参考画面的接近程度;IoU 是预测对象区域与参考区域的交集除以并集,范围为 0 到 1。7B 从 1 帧到 9 帧时,两项都明显改善,但更多上下文与更短的待预测跨度同时发生变化,不能把它当作同跨度下的纯上下文消融。保持 9 帧后换成 14B,IoU 略高,PSNR 反而降低。因此,更多参数没有带来所有物理预测指标上的一致提升。
几何测试则问另一个问题:静态房间的不同画面能否拟合出相容的相机运动和场景结构。它可以抓到墙面漂移,却不直接测试球是否遵守重力。图中把这些检查分开,是因为一个逼真的房间漫游和一个正确的碰撞预测,不能由同一个视觉分数替代。
| Video2World configuration | History conditions | PSNR (dB) ↑ | Object-mask IoU ↑ |
|---|---|---|---|
| 7B | Text + 1 frame | 17.34 | 0.332 |
| 7B | Text + 9 frames | 21.06 | 0.592 |
| 14B | Text + 9 frames | 20.21 | 0.598 |
This is an excerpt from Table 20. PSNR measures pixel-level agreement with reference images. IoU divides the intersection of predicted and reference object regions by their union, on a scale from zero to one. For 7B, moving from one to nine frames improves both. However, more context and a shorter prediction span change together, so this is not a context-only ablation at a fixed prediction horizon. With nine frames fixed, 14B has slightly higher IoU but lower PSNR. More parameters therefore do not improve every physical-prediction metric consistently.
Geometric testing asks something else: can different images of a static room be explained by compatible camera motion and scene structure? It can reveal drifting walls without directly testing whether a ball obeys gravity. The diagram separates these checks because a plausible room tour and a correct collision prediction cannot be replaced by one visual score.
5 · Closed-loop control requires another validation step5 · 视频预测到闭环控制,还隔着一次验证
Cosmos 报告展示了相机、机器人和驾驶的后训练示例;动作条件下一帧预测也有定量结果。但其中关于策略训练、策略评估与规划的一组下游用途,作者在第 2.1 节明确作为前景讨论,没有把它们全部验证。不能把“可接入这些流程”写成“已经证明能替代真机试验”。
V-JEPA 2 论文后来把适配的 Cosmos 用于机器人在线规划:候选动作先生成预测,再据此选择。这个实验增加了真实闭环证据,也暴露了计算代价和操作成功率问题。它补上的是一个具体使用场景,不会反过来使 Cosmos 中所有生成测试都成为控制实验。
如果要为自己的机器人使用视频世界模型,最重要的顺序是:先确认目标设备的控制量能进入模型,再检查任务相关的预测误差,最后验证用这些预测选出的动作是否在真实闭环中有效。小球示例中,纹理不准可能无关紧要,挡板穿透却会改变决策。评测应该跟随这种区别。
另一个方向是把空间本身交付为可漫游的资产。Atlas 的空间上下文讨论位姿、深度与显式 3D 输出;想象与规划讨论怎样把预测接到动作。它们与视频生成共享工具,却分别需要空间证据与控制证据。
The Cosmos report demonstrates post-training for cameras, robots and driving, including quantitative next-frame prediction under action conditions. However, the policy training, policy evaluation and planning uses discussed in Section 2.1 are proposed applications rather than a complete set of demonstrated downstream results. “Can be connected to these workflows” should not become “proven to replace real-machine testing.”
The V-JEPA 2 paper subsequently uses an adapted Cosmos model for online robot planning: candidate actions lead to predictions, which inform action selection. That experiment adds real closed-loop evidence and exposes computational costs and manipulation failures. It validates one particular use setting; it does not turn every Cosmos generation test into a control experiment.
For a robot deployment, first verify that its controls can be expressed through the model, then test prediction errors that matter for the task, and finally test whether actions chosen from those predictions succeed in a real closed loop. In the ball example, a texture error may be irrelevant, while passing through a barrier changes the decision. Evaluation should follow that distinction.
Another direction delivers the space itself as an asset that can be explored. Atlas's spatial context concerns poses, depth and explicit 3D output; imagination and planning concerns using predictions to choose actions. They share tools with video generation but require spatial and control evidence respectively.
Continue reading继续阅读
- Cosmos · arXiv v3 (2025)
- Cosmos Predict1 · official code
- V-JEPA 2 · downstream control comparison
- Original study notes原始学习笔记 · cosmos