Learning & methods · November 2026学习与方法 · 2026 年 11 月

How Models Learn, Imagine, and Act从世界模型到 Agent:五条阅读路线

Study notes reorganized around specific questions. Nine new essays join the existing illustrated articles to connect models, learning, feedback, and decisions.围绕具体问题重组学习材料。九篇新文章,与已有图解一起,串起模型、学习、反馈和决策。

World models & spatial generation世界模型与空间生成

Start with the action the model must support, then examine its prediction space and how its outputs are checked.先问模型要帮助谁行动,再看它在什么空间预测,以及生成结果如何被核验。

4 new essays新整理的文章 · 4
  1. Learn to act in imagination, or imagine before acting?在想象中学会行动,还是行动前再想一遍?

    DreamerV3 and V-JEPA 2-AC both predict the future. One uses predictions to train a policy; the other searches with them before each action.DreamerV3 与 V-JEPA 2-AC 都预测未来,却把预测放在不同位置:前者用它训练策略,后者用它在每次行动前搜索。

  2. How does Genie learn controls without recorded actions?没有按键记录,Genie 怎么学会让世界听指令?

    Compress changes between frames into a small codebook, then let the user select a code. Discovering controls from video is different from observing real actions.把相邻画面的变化压进一个小码表,再让使用者选择码字:从视频里发现动作,与知道真实动作,是两回事。

  3. When does a generated video become a useful world model?生成的视频,什么时候才算可用的世界模型?

    Cosmos connects video compression, conditioning and physical evaluation. Visual realism, geometric consistency and correct action consequences impose different requirements.从 Cosmos 的视频压缩、条件接口和物理评测出发,分清画面逼真、几何一致与行动后果正确这三种要求。

  4. Are unseen places reconstructed or imagined?看不见的地方,是重建还是想象?

    Atlas organizes images and camera poses into a spatial context. More views constrain generation, but a navigable 3D world is not a fully measured reality.Atlas 把图像与相机位姿组织成空间上下文。更多视角能约束生成,但可漫游的 3D 世界不等于完整测量过的现实。

Reinforcement learning & policy optimization强化学习:从基础到策略优化

Actions change the future and rewards drive updates; training curves provide different views of that process.选择改变未来,奖励驱动更新,而训练曲线只是观察学习过程的不同窗口。

1 new essays新整理的文章 · 1
  1. Does a Higher Reward Mean a Better Model?奖励涨了,模型真的变强了吗?

    Read reward, KL, entropy, policy loss, and validation scores as different measurements—and learn what evidence is needed to connect them.把奖励、KL、熵、策略损失与验证分数放回各自的测量对象,理解一条曲线能回答什么,以及还缺什么证据。

  1. From Selection to Planning: Bandits and MDPs从选择到规划:Bandit 与 MDP

    Distinguish a one-off choice from an action that changes what happens next.先区分一次选择与会改变未来的行动。

  2. What an Agent Sees—and What Actually HappenedAgent 看到的,和环境里发生的

    From the Markov property to hidden states, belief updates, and MCMC.从 Markov 性质到隐藏状态、信念更新和 MCMC。

  3. Inside a GRPO Update一次 GRPO 更新,究竟学到了什么?

    Follow sampling, relative rewards, and a single policy update.沿一次采样、奖励比较和策略更新读懂 GRPO。

  4. What Changes Beyond GRPO?GRPO 的改进,改在了哪里?

    Separate changes to normalization, clipping, and sample handling.比较归一化、裁剪和样本处理各自改变的对象。

Post-training, preferences & distillation后训练、偏好优化与蒸馏

Begin with pairwise preferences, compare the fitting objective with the resulting policy, then follow learning from the student’s own answers.从成对偏好出发,比较拟合目标与实际策略,再看学生怎样从自己的回答中学习。

1 new essays新整理的文章 · 1
  1. When Better Preference Fitting Makes a Worse Policy偏好拟合得更好,为什么策略反而变差?

    A three-action example reveals how a restricted policy family and pair frequencies can misdirect DPO—and why AuxDPO separates policy movement from preference fitting.从一个三动作算例理解 DPO 的模型错配:策略能表达什么、哪些比较出现得更多,以及 AuxDPO 怎样把两种自由度分开。

  1. From Cross-Entropy to Preference Learning从交叉熵到偏好学习

    Connect binary classification, pairwise choices, and reward differences.把二分类、成对选择和奖励差连成一条线。

  2. Learning from the Model’s Own Answers模型如何从自己的回答中学习

    Understand student trajectories, teacher supervision, and KL direction.理解学生轨迹、教师监督,以及 KL 方向的含义。

  3. What Data Does Self-Distillation Need?自蒸馏时代,数据生产什么

    Separate tasks, feedback, and the distribution a student can learn from.把数据拆成任务、反馈和可学习的分布。

Agent engineering & trajectory dataAgent 工程与轨迹数据

Follow task creation, action representation, feedback, and compute allocation to understand how an agent gets work done.沿任务来源、动作表达、反馈和算力分配,理解一个 Agent 怎样完成工作。

3 new essays新整理的文章 · 3
  1. Where Do Agent Tasks Come From?Agent 的任务从哪里来?

    OS-Genesis starts with exploration; AgentTrek starts with tutorials. Both must turn a plausible goal into an execution trace that can be checked.从 OS-Genesis 的探索后命题,到 AgentTrek 的教程引导重放:把有意义的目标变成可验证的执行轨迹。

  2. Why Let an Agent Act Through Code?为什么让 Agent 写代码?

    CodeAct puts control flow into a program and returns execution feedback to the model. Fewer interactions do not remove the need to verify the result.CodeAct 把控制流交给程序,把环境反馈交还模型。它能压缩交互,但运行成功与任务正确仍是两件事。

  3. When Should an Agent Think More or Try More?什么时候多想,什么时候多试?

    Separate parameter learning, inference-time search and reasoning-mode selection. Spend additional computation where it can change the outcome.把参数训练、推理搜索与思考模式选择分开:同一份预算,应该花在能够改变结果的地方。

  1. Building Training Trajectories for Coding Agents为 Coding Agent 构建训练轨迹

    From task specifications and execution feedback to trajectory selection.从任务规格、执行反馈到轨迹筛选。

  2. Repairing Specs with Feedback用评审反馈修复 Spec

    See how feedback becomes the input to the next revision.看反馈如何成为下一次修改的输入。

  3. From Designs to Artifacts: Training Rubric Judges从设计到产物:训练规则评审模型

    Separate judging rules, model decisions, and evidence that can be checked.区分评审规则、模型判断和可核验的依据。

Reasoning state, spatial workspaces & decisions推理状态、空间工作区与决策

These questions already have independent essays: begin with preserved state, then spatial representations, and finally decisions.这组问题已有独立图解:先读状态如何被保存,再看空间如何被表示,最后理解答案怎样变成决策。

  1. Collect the Clues, Then Read Again先整理线索,再重读长文

    Use a reasoning trace as state for rereading, while tracing information and cost.把推理轨迹作为后续重读的状态,检查信息与成本的来源。

  2. How Models Represent Where Things Are模型如何表示物体的位置

    From spatial representations that can be read out to an internal workspace that can be intervened on.从可读出的空间表征走向可干预的内部工作区。

  3. Turning Language Models into Decision Models让语言模型做选择

    Separate candidate scoring, probability calibration, and abstention.把候选项评分、概率校准和拒绝决策分开。

Return to the collection回到原始资料

The original 47-document collection remains available. These paths organize its first five themes, covering 27 source documents. Existing essays are reused where they already answer a question; reference material such as post-training tool overviews stays in the collection.原来的 47 篇资料仍保留。这里整理的是前五个主题的 27 份材料;已有文章覆盖的问题沿用原稿,后训练基础中的工具概览等内容继续留作补充阅读。

Knowledge collection since September →9 月以来的知识整理 →