Learning & methods · November 2026学习与方法 · 2026 年 11 月

How does Genie learn controls without recorded actions?没有按键记录,Genie 怎么学会让世界听指令?

Compress changes between frames into a small codebook, then let the user select a code. Discovering controls from video is different from observing real actions.把相邻画面的变化压进一个小码表,再让使用者选择码字:从视频里发现动作,与知道真实动作,是两回事。

1 · Video shows outcomes, not button presses1 · 视频有结果,没有按键

看一段平台跳跃游戏录像,我们能看到角色从地面跳到平台,却未必知道玩家何时按了跳跃、按了多长时间,也没有游戏引擎内部的坐标。普通视频预测可以学习“接下来常出现什么画面”,但要让使用者随时改变走向,还缺一个可以干预的接口。

2024 年的 Genie 论文把这个接口当作需要学习的变量。它不先找人给所有视频补按键标签,而是问:如果把历史画面交给一个解码器,再给它一个很小的额外编码,这个编码必须携带什么,才能解释下一帧为什么这样变化?模型由此学习离散的潜动作。

“潜”表示编号的含义不是预先写好的。编号 3 可能在一类场景里对应向右,但论文并没有保证 3 就是通用的“右键”。小码表让人可以尝试每个编号,摸清它在模型里的作用。这里讲的是初代论文的机制,不把后续产品展示的分辨率、时长或实时性反推为初代实现细节。

A platform-game recording shows a character jumping onto a ledge. It may not reveal when the player pressed jump, how long the key was held, or the coordinates inside the game engine. Ordinary video prediction can learn which frame tends to come next. Letting a user change the course of events requires an intervention interface as well.

The 2024 Genie paper makes that interface a variable to be learned. Instead of asking people to add button labels to every video, it asks what a small extra code must contain when a decoder already has the past frames and must reconstruct the next one. This creates a discrete latent action.

“Latent” means the code's meaning is not specified in advance. Code 3 might correspond to moving right in a class of scenes, but the paper does not establish a universal right-button identity for code 3. A small codebook lets people try each code and discover its effect. This article concerns the first paper's mechanism; later product demonstrations do not establish the first model's resolution, duration or real-time capabilities.

2 · A bottleneck forces the model to explain change2 · 一道信息瓶颈,迫使模型解释变化

潜动作模型 LAM 有一个编码器与一个解码器。编码器可以同时看历史和下一帧,推断这次变化;解码器只看历史和推断出的动作,尝试重建下一帧。两边看到的信息不同,是这个设计能产生学习信号的关键。

ut=g(x1:t+1),at=Q(ut)∈{0,…,7},x^t+1=d(x1:t,eat).

这是对论文数据流的简写:x 是画面,g 是编码器,ut 是连续编码,Q 把它量化为 8 个码字之一,eat 是对应嵌入,d 是解码器。重建失败会推动编码器把对未来有用的变化放进码字,向量量化则限制通道容量。

The latent action model, LAM, has an encoder and a decoder. The encoder sees both the history and the next frame, allowing it to infer the change. The decoder sees only the history and inferred action and attempts to reconstruct that next frame. The information asymmetry creates the learning signal.

ut=g(x1:t+1),at=Q(ut)∈{0,…,7},x^t+1=d(x1:t,eat).

This abbreviates the paper's data flow. Frames are x; g is the encoder; ut is a continuous code; Q quantizes it to one of eight codes; eat is that code's embedding; and d is the decoder. Reconstruction errors encourage the encoder to place useful information about the change in the code, while quantization constrains the channel's capacity.

Static diagram静态图Training: discover a small action codebook
The encoder sees the next frame; the decoder does not. A small codebook must supply useful change information. Past frames provide scene content directly, and reconstruction error trains the LAM.编码器看到了下一帧,解码器没有;小码表必须补上有用的变化信息。历史画面直接提供场景内容,重建误差监督 LAM。

假设同一段历史之后,录像可能出现向左、向右或起跳。只凭历史不能确定哪一种;动作编码提供额外信息。8 种离散取值每步最多携带 log28=3 bit 的选择信息,远不足以塞下一整张未来画面。解码器必须复用历史中的场景与角色,把码字主要用于解释变化。这个 3 bit 是码表容量的算术上限,不是论文测得的有效动作信息量。

瓶颈不自动识别因果。镜头滚动、角色运动、其他物体运动都可能进入“变化”。因此,潜动作有可用控制效果,不等于模型恢复了游戏引擎的真实按键,也不保证把可控因素与背景自主变化彻底分开。

After the same history, a recording might show movement left, movement right or a jump. History alone cannot distinguish them; the action code supplies extra information. Eight discrete choices carry at most log28=3 bits of selection information per step, far too little to store an entire future frame. The decoder must reuse the scene and character in the history and use the code mainly to explain change. Three bits is the arithmetic capacity of the codebook, not a measured amount of effective action information.

A bottleneck does not automatically identify causality. Camera scrolling, character motion and other object movement may all enter the change variable. A useful control effect therefore does not mean the original engine buttons have been recovered, or that controllable factors are perfectly separated from autonomous background changes.

3 · Two codebooks and a dynamics model3 · 两种码表,一种预测器

Genie 还需要视频 tokenizer,把高维画面压缩为离散视觉 token。它与潜动作码表的职责不同:视觉 token 表示场景内容,潜动作表示跨帧变化。论文先训练视频 tokenizer,再共同训练 LAM 与动力学模型;LAM 直接读像素,动力学模型读视觉 token。

动力学模型把历史视觉 token 与动作嵌入结合,预测下一帧的 token。动作输入阻断来自动力学损失的梯度,LAM 自己仍通过重建任务学习。这样,“如何发现动作”和“如何依动作生成未来”相连,但不是让动力学损失直接随意改写动作发现器。

生成过程还要区分两条轴:沿时间,是一帧接一帧;在一帧内部,采用 MaskGIT 式迭代并行补全,而不是像写文字那样每次只生成一个视觉 token。被遮盖位置逐步获得预测,直到下一帧可以解码。将它笼统说成“GPT 一样逐 token 画视频”,会漏掉真正影响效率的部分。

三个组件都使用分解的时空注意力:先在单帧内交换空间信息,再沿同一空间位置交换历史信息。时间因果性限制模型使用未来;训练阶段 LAM 为第 t 次转移读到 xt+1 是有意的监督构造,并不表示部署时也能偷看未来。

Genie also needs a video tokenizer to compress high-dimensional frames into discrete visual tokens. Its codebook serves a different purpose from the action codebook: visual tokens describe scene content; latent actions describe changes between frames. The video tokenizer is trained first, followed by joint training of the LAM and dynamics model. The LAM reads pixels directly; the dynamics model reads visual tokens.

The dynamics model combines previous visual tokens with action embeddings to predict the next frame's tokens. Gradients from its loss are stopped at the action input, while the LAM continues learning through its own reconstruction objective. Action discovery and action-conditioned generation are connected, but the dynamics loss does not directly rewrite the action-discovery module.

Generation has two distinct axes. Across time, frames are generated one after another. Within a frame, MaskGIT-style iterative parallel completion predicts masked positions; it does not produce just one visual token at a time like a text model. Successive iterations fill the frame until it can be decoded. Describing the whole process simply as GPT-like next-token video generation misses an important efficiency choice.

All three components use factorized spatial and temporal attention: exchange information within a frame, then across historical frames at a spatial position. Temporal causality limits access to the future. The LAM's access to xt+1 while labeling transition t is an intentional training construction, not future information available during deployment.

4 · At use time, the player replaces the action encoder4 · 使用时,玩家替换掉动作编码器

提示图先被编码为视觉 token。使用者选择一个动作编号,码表把编号变成嵌入,动力学模型预测下一帧 token,解码器还原为画面。接下来,再选动作。除动作码表外,训练用的 LAM 编码器和解码器都不参与这条路径。

A prompt image becomes visual tokens. The user chooses an action index; the codebook turns it into an embedding; the dynamics model predicts the next frame's tokens; and the decoder reconstructs the image. Then another action can be chosen. Apart from the action codebook, neither the LAM encoder nor its decoder participates in this path.

Static diagram静态图Use: a person chooses; the model continues
User-selected action indices replace the LAM, retaining its action codebook. Generation advances frame by frame in time, with MaskGIT iteratively completing tokens within each frame.LAM 被人给出的动作编号替代,只保留动作码表。时间上逐帧生成,帧内用 MaskGIT 迭代补全视觉 token。

这解决了一个看似矛盾的问题:训练时需要看完变化才能推断动作,为什么使用时不需要未来?因为推断动作是为视频补出训练标签的手段;使用时动作已经由人给出。模型无需再从尚未发生的结果中反推动作。

想象一张手绘的平台场景。先试编号 0,再试编号 1,观察哪种选择让角色上升。弄清接口之后,可以重复该编号尝试越过障碍。这个过程是在学得的生成环境里互动,不是在恢复一个隐藏运行的真实游戏引擎;生成结果可能偏离原游戏的碰撞与运动规则。

初代模型训练使用 16 帧片段,论文也将有限记忆、幻觉和约每秒 1 帧的推理速度列为限制。因此,可逐帧选择动作的“交互性”,与流畅实时游戏的“交互体验”,应分开判断。

This resolves an apparent contradiction: if training needs to see the completed change to infer an action, why does use not require the future? Inferring actions is a way to provide training labels for videos. At use time, the person already supplies the action, so it need not be inferred from an outcome that has not happened.

Imagine a hand-drawn platform scene. Try code 0, then code 1, and observe which makes the character rise. After learning the interface, repeat that code to attempt an obstacle. This is interaction inside a learned generative environment, not recovery of a hidden game engine. Generated collisions and movements may depart from the original game's rules.

The first model trains on 16-frame clips, and the paper identifies limited memory, hallucination and inference at about one frame per second as limitations. Frame-level action selection establishes one sense of interactivity; a responsive real-time game experience is a separate requirement.

5 · Does changing a code actually change the future?5 · 换一个码字,真的改变了未来吗?

论文用两次预测比较动作的作用:第一次使用从真实视频推断的动作序列,第二次换成随机动作序列。若第一种预测更接近真实目标帧,说明动作通道确实携带有用的信息。论文统一报告 t=4 时目标帧的 PSNR 之差,而不是只比较紧接着的一帧;PSNR 衡量像素误差,越高表示越接近。

教学上,若真实推断动作得到 28 dB,随机动作得到 25 dB,差值为 3 dB。这个差值不是“动作正确率 3%”,也不能证明动作 2 就是人类理解的跳跃。它检验动作是否影响有用的预测;语义一致、长期稳定与真实控制能力,还需要别的证据。

更接近控制的一项证据来自 CoinRun:LAM 为专家视频推断潜动作,策略学习模仿这些动作,再用少量带真实动作标签的样本学习潜动作到真实动作的映射。论文报告,在该设置中仅需 200 个专家标注样本即可达到 oracle 行为克隆的相当分数。这个实验使用的是动作表示来训练策略,不是把完整生成环境当作强化学习模拟器训练出的迁移结果。

Genie 的价值在于把无动作录像变成可学的控制接口。若问题转为“已经有动作记录,怎样决定执行哪一步”,可接着读想象与规划;若需要生成指定相机运动下的画面,则要看Cosmos 的条件接口。两者都不需要把一个无语义潜动作编号误当成精确的物理指令。

The paper tests the action channel with two predictions. One uses an action sequence inferred from a real video; the other substitutes a random action sequence. If the first prediction is closer to the true target frame, the channel carries useful information. The paper consistently reports the difference in PSNR at t=4, rather than evaluating only the immediately next frame. PSNR measures pixel error on a logarithmic scale; higher means closer.

In a teaching example, 28 dB with inferred actions and 25 dB with random actions give a 3 dB difference. This is not 3% action accuracy, and it does not prove that action 2 means a human-interpretable jump. It tests useful influence on predictions; semantic consistency, long-term stability and real control require further evidence.

The CoinRun experiment moves closer to control. A LAM labels expert videos with latent actions; a policy learns to imitate those actions; then a small set of examples with real action labels provides a latent-to-real action mapping. In this setting, the paper reports a score comparable to oracle behavior cloning with as few as 200 expert labels. This tests an action representation for policy learning. It is not transfer from reinforcement learning inside the full generated environment.

Genie's contribution is a learnable control interface extracted from recordings without actions. If actions are already recorded and the problem is deciding which to execute, continue with imagination and planning. For images conditioned on specified camera motion, see Cosmos's conditioning interfaces. Neither task requires treating an unnamed latent code as a precise physical command.

Continue reading继续阅读

  1. Genie · arXiv v1 (2024)
  2. Genie · author project page
  3. Original study notes原始学习笔记 · genie

Related Writings相关 Writings