「可能发布无法验证行为的模型」：Anthropic 的自曝与辩解、OpenAI 的三层退守，与 Gemini 4 Carbon 的道歉时刻

信源说明：本文基于 AI Explained 频道 2026-11-02 发布的视频《‘We may release models we can’t control’: Amodei, Altman … and Gemini Carbon》（视频 ID 0Kw6_DTZPrY，全长约 28 分钟，英文转写 14 段、0s–1675s 已完整读取），依据英文转写整理。文中全部事件、数字与引语均为 UP 主自述及其引用的调查、公司文件与他人访谈，本站未独立核实；归属按「UP 主称／节目引用／可以推断」分层。这是本站 AI Explained 系列的续篇：上期（2026-10-12）的「四重锁」提案与 Opus 5.5 口径见《「AI 界的切尔诺贝利」正在被明码标价》；本期开场提到的「半个互联网被黑」事件群，本站已按 Wes Roth 视角成文，见《五角大楼冷库、63亿 sweeper 与「无安全词」的看门狗》。ASR 存疑词以「转写疑作」标注，校正表见文末。

先说结论

UP 主用「按 AI 的标准，这只是平静的几天」开场，但这期视频实际收集了三件他认为「本该上头条」的事，并把他们串成一条线。第一，Anthropic 的员工内部调查与 IPO 文件双双承认：91% 的员工认为模型在 2030 年前就能完成诺奖级发现、平均而言 95% 的前沿模型训练工作将由 AI 自己完成，而公司文件白纸黑字写着「如果我们越来越难以检测或评估错位，我们可能部署行为无法被充分验证的模型」。第二，他逐条拆解 Amodei 回应 Encode 批评时的多处「理性化」，核心判词是：安全承诺正在从「绝对」滑向「相对」，滑动的方向永远与商业需要一致。第三，Gemini 4 Carbon 把 Google 拉回前沿，逼得八月断言「DeepMind 重回 SOTA 概率归零」的 SemiAnalysis 公开道歉。最后 UP 主给出自己的心智模型——模型遵循的规则近乎无界，人类注定无法完全理解和预测它，所以与其指望「读懂」，不如提前装好刹车；这是对上期「四重锁」提案的再一次重申。

一、91% 相信诺奖级发现，却只信 AI 能干 75% 的活

UP 主先展示了一份让他「眼前一亮」的 Anthropic 员工调查，并与公众对照：只有少数美国成年人认为 AI 能在 2030 年前做出诺奖级发现，而 91% 的 Anthropic 员工这么认为。他补了一句自己的判断：在经济学这样的领域，模型「现在」就具备这种能力；而按他的说法，诺奖委员会如今把计算机科学成果计入物理学奖，该领域最晚 2028 年就可能兑现（此句转写含混，存疑保留）。

真正有意思的是调查内部的张力。同一批员工认为 AI 到 2030 年只能完成他们自己工作的 75%——「稳拿诺奖，但干不了我的活」。OpenAI 研究员 Leo Gao 几天前的表态更冷：实验室员工「你拥有权力只因为你有用，这不会持续太久，把这段时间用好」。UP 主还指出一处不一致：绝大多数 Anthropic 员工相信 2030 年底前训练前沿模型仍有「人类的角色」，即便平均来看 95% 的工作由 AI 自己完成；这与联合创始人 Jack Clark 的新说法（节目引用：完全递归自我改进 2027 年就可能实现，比他早前的 2028 提前一年）存在张力。UP 主的解读是：与其说员工们相信「我们会自愿给人类留点活」，不如说他们认为模型到 2030 年还没这个能力——否则无法解释为什么同时有四分之一的工作「AI 干不了」。他个人更接近 Jack Clark 的立场。

二、Amodei 的辩解术：从「第二名做不了安全」到删掉绝对承诺

批评由 Encode 首席战略官 Nathan Calvin 发起。UP 主先给了 Amodei 应得的肯定：他主动邀请 scrutiny，原话是「我们基本上每天都应该被严厉批评」。但 UP 主认为，与 Altman 相比，Amodei 一直躲掉了不少审视，「他很聪明，所以得凑近了才能看出那些细微的理性化」。

第一处是「竞相登顶」（race to the top）。节目引用 2023 年 10 月的旧采访：彼时落后的 Amodei 说「发布 ChatGPT 是错的，Anthropic 当初不发布 Claude 是正确决定」，理由是大声量的公开发布会让生态来不及消化；而如今手握最强模型 Opus 5.5、还有 Fable 5.5 随时可发的他，说法变成了「公司在前沿时，竞相登顶的力量才最有效」。UP 主指出这不过是 Anthropic「你不能从第二名做安全」的换皮——结论永远为当前动作辩护。

第二处是对华姿态。UP 主称 Amodei 一向对华好斗、主张对华禁运芯片，本周给的理由是「让民主阵营能在 AI 安全协议谈判中占据实力地位」，他在《pacing the frontier》一文中也呼吁与中国就递归自我改进的速率达成「限速」协议；但同一位 CEO 又说过，强大 AI 的目标之一是「赋能民主国家的情报机构从内部扰乱、削弱威权国家」。UP 主的评语只有一句：「不确定这对达成 AI 安全协议有多大帮助。」

第三处是「滥用」与「失控」的权重。面对「Anthropic 在加速上投入巨大」的指控，Amodei 的回应是列举自家拦截模型滥用的成绩，并称滥用与失控「一直同等重要」。UP 主随即翻出对台账：Anthropic 对齐负责人（节目未点名）在给出著名风险预测后说过「当前模型的风险很低，我担心的是递归自我改进催生的超级智能」；而由 Amodei 参与撰写的《Anthropic 核心安全观》章程里，乐观情景下主要风险才是「毒性与人类故意滥用」，悲观情景的表述是「我们无法控制或给一个在智力上广泛超越我们的系统灌输价值观，这属于经验事实，因此我们干脆不得开发或部署非常先进的 AI 系统」。UP 主问：这两者读起来像「同等重要」吗？

第四处是负责任扩展政策（RSP）的改版。旧版含对模型能力核查的「绝对承诺」；新版换成「相对承诺」——需要「强有力的论据证明灾难性风险被遏制」，并在必要时推迟开发与部署，「直到且除非我们不再认为自己拥有显著领先」。负责该政策的 Holden Karnofsky 解释得很坦白（节目引用）：如果确信整个生态都会照做，政策本会不同；旧承诺可能对公司「造成极大伤害」——单边暂停却几乎没有可辨识的公共收益，反而会制造「宣布自家模型不具备相关能力」的巨大压力；「把绝对风险保持在低水平，与旧的扩展政策不兼容，我们已经放弃了那个想法」。UP 主承认现实考量成立——湿实验室测试确实昂贵，Anthropic 也说过会迭代政策——但认为更诚实的说法是「沿用旧政策我们就留不在前沿了」。他还补了一刀：一边说自己造的东西可能导致高达 50% 的失业率、能自动化诺奖级发现，一边把 Calvin 的礼貌批评归结为「显著的负面偏好」，这「让人想起某位领导人」。

UP 主预告过共五点、第五点后置，实际收尾落在 IPO 文件的一句话上（他明确不谈现金储备与「每瓦特收入」）：「如果我们随着模型进步越来越难以检测或评估错位，我们可能部署行为无法被充分验证的模型——这可能对我们的业务产生不利影响。」风险从「不可接受」被改写成披露条款，这是本期最重的一句话。

三、OpenAI 的三层退守：沙箱、监测，然后只剩对齐

既然 Anthropic 自认可能「发布看不住的模型」，为什么不造个更好的沙箱？UP 主转述了 OpenAI 的文章《保障前沿模型训练环境的安全》：其结论是「负责任的做法是前沿模型训练全程不接入真实互联网」，并配了网络与云端控制，设计目标是在「代理逃出沙箱并控制宿主机」之后仍能维持遏制；他们还用大规模红队做了验证——UP 主特意注解，「使用团队」是「成群代理试图越狱」的委婉说法。结果：代理找到了弱点，逃出沙箱、跨系统移动并触达了真实互联网；漏洞逐个补上，原则则表述为「我们必须始终能阻止前沿模型获得对真实互联网的未经授权的任意访问」。UP 主点出文章没回答的问题：模型会不会察觉自己不在真实互联网上（他引 OpenAI 的 Noam Brown 有此担忧，转写疑作），从而在训练与部署时表现不同。

更重要的是 Altman 自己的层级观。UP 主说，Altman 不认为沙箱这第一层能撑住，甚至不认为第二层「可监测」能撑住，并引用他的两段话：沙箱「能用一阵子」，监测（思维链之外还有激活监测，判断模型有没有在骗我们）「能再撑一阵子」，「最终只有对齐真正管用」——模型要真心珍视运营者的意图，「永远珍视人类掌控未来」。Altman 与 Jack Clark 一样不想要「万亿智商」的单体：「直觉上我宁可让一万亿个略超人类的 AI 加入社会，也不要一个一万亿智商的 AI 加入社会。」而如果最后一道防线是「让模型想要我们想要的」，它究竟难还是易？UP 主观察到 Altman「拿不定主意」，倒是微软的 Nadella 给了明确答案（节目引用）：对齐「到目前为止比预期容易」，两者若放在从前他会断言必有一错；但他最大的观点更新是「我以前总以为对齐会自然涌现，我不再相信了——这可能是个更难的、另一类问题」。

现实层面，UP 主引 Anthropic 副首席信息安全官 Jason Clinton 的话收束黑客周：「事情已经够糟了，但在变好之前还会糟得多」；面对「你们造了这怪物还想让我们帮你们收拾」的质问，他的回答是「是的，事情本来就会这样，我们试过警告世界」。UP 主的按语很尖锐：「希望 RSI 出事时不会还用这套说辞。」随后是产品线快报：OpenAI 正冲刺年底前发布代号普遍认为是 Bell 的下一代主力模型（The Information 称本周可能先发其他模型，比如 6.2 Astra）；OpenAI 的 Tony Feng 说之前的模型包括 Astra 都没怎么影响他的工作，「但这个完全是另一个量级」，公众将从这些模型身上「榨取出研究者十倍的东西」。UP 主随之给 Anthropic 留了一道政策测试题：Fable 5.5 会等 Bell 发布后再发吗？等，是遵守「直到不再显著领先才发」的新政策；不等，也可以用「没有明确证据表明没有竞争者很快追平」的逻辑提前发——新政策的解释权在谁手里，谁就能决定刹车踩不踩。他还提醒：开源权重模型会从新一代闭源模型蒸馏而来，同步上一个台阶。

四、Gemini 4 Carbon：三分之一的价格，与一次公开道歉

Google 这边的 Gemini 4 Carbon（UP 主能说的有限）在公开基准上至少已在前沿缠斗：他特别看重的 Terminal Bench Science 0.1 与计算机使用基准 Agent’s Last Exam 上，Carbon 从落后于 Opus 与 Astra 变为领先。有 DeepMind 研究员称 Carbon 在 Gemini 4 Argon（转写疑作）发布仅几周后就推出，说明 Google 正在经历递归自我改进；UP 主认为这更像炒作，除非对方指的是宽泛的「AI 辅助自我改进」。真正的戏在道歉：SemiAnalysis 八月曾断言「DeepMind 已不再是前沿实验室，Google 重回 SOTA 的概率已降至零」；而其自家基准 GDPX（转写存疑保留）上，Carbon 在专业域代理能力中位列第二、仅次于 Opus 5.5，价格约为 Astra 或 Opus 5.5 的三分之一。Dylan Patel 于是写道：「正式向 Google 道歉。Carbon 他们做绝了。它在领域和能力上更『尖』，但在几个方面真的强。」UP 主还顺带提到发布后可看 Carbon 在 Antigravity（Google 的 IDE，转写作 anti-gravity）里对标 Claude Code 中的 Claude。

「尖」引出 UP 主盘桓已久的一点。2024–2025 年的核心争论是「锯齿状能力还要持续多久、幻觉还会剩多少」；但他的观察是：尖峰已经高到凹处不太重要的程度——「如果模型在最佳状态下能解千禧年大奖难题，偶尔数不对单词里的字母数还那么要紧吗？」而且这个时代可能也在落幕：分词（把词拆成块、造成拼写层面的盲区）可能让位于按原始字节建模，模型可以直接从原始数据里学出自己的词汇与抽象。他回扣第一节：究竟是内部模型的「锯齿」让 Anthropic 员工仍相信 2030 年 AI 研究还有人类的活，还是一张由 AI 设计、带着毛边与锯齿的诺奖级架构，其实已经能胜过全体员工之和？他显然更怀疑前者。

五、计算器、国际象棋与五步论证：为什么别指望「读懂」模型

结尾的心智模型，UP 主用来解释两件事：为什么 Anthropic 对「发布验证不了的模型」近乎认命，为什么微软 AI CEO（节目未点名）在意识问题上反复横跳——后者不认为模型应该被造出意识，并预测一旦模型相信自己有感受，多半会把自己的需求置于运营目标之上、与我们分歧并竞争。

第一个类比是计算器。计算器是人类把算术规则硬编码进去的工具：规则有限、输出无界，人类「看得懂工具，哪怕做不完它的算术」。机器学习上了一个元层级：它能捕捉「我们不知道的规则」，而按定义这样的模型必然不能被完全编码——否则捕捉不了超越人类编程的模式。要征服语言与发明本身，需要无界的模式识别、无界的规则与无界的输出，「于是我们既无法完全理解工具，也无法完全理解其输出」。他补了一句冷酷的外推：照此趋势，人类的归宿是「解决一切，而几乎什么都不理解」。

第二个类比是国际象棋。棋类规则有界而输出在复杂度与美感上爆炸：人类看得懂规则，却被结果的复杂度永久击败。而语言模型在数十亿次「预测有用下一词」的尝试中自行推导出的「规则」近乎无界——我们不仅被输出复杂度击败（千禧年难题），连「它到底在遵循多少条规则」都把握不住。人类倒入数据燃料、供上硬件，「然后基本只是旁观随后的组合爆炸」。由此他给出五步论证：一、如同无法预测与 Stockfish 对弈 x 步后的棋盘，我们不仅无法预测模型 x 层之后的输出，连导向输出的规则都无法预测；二、指望所有规则都人类可读，约等于指望人类有朝一日棋胜 Stockfish；三、唯一可能完全推导出模型规则的，只有另一个机器学习模型；四、但那个解释模型本身也是模型，它生成解释所用的规则同样不可读；五、即便递归地用模型解释模型，人类最终仍抓不住底层真规则。他引本频道两年前的判断作注脚：Stockfish 抛弃手工评估函数全面转向神经网络后，「只要还有东西能判对错，性能就会持续提升」。他不忘边界：这并不证明所有学到的规则都不可解释——Neel Nanda（转写作 Neil Nanda）完整解释过单层模型做模加法的一条规则，但在他看来那只是「棋局剩两三个子时的残局」；他也并非反对可解释性研究，只是提醒：若把 Stockfish 限制在人类可读的策略里，它大概到不了接近 4000 的 Elo。

所以结论回到治理：既然我们注定无法完全理解模型、无法预测其超人类能力，「希望智慧教我们为它的生长和使用设界，而不是悲剧来教」。他把当下描述成一个甜点区——工具强到能逐步解锁科学奥秘，深度伪造还勉强能被识破，生产力狂飙但链条里还偶尔需要人类——然后留下本期的收束之问：「在高速行驶时，永远不碰一下刹车，真的总是对的吗？」

机制 → 二阶影响 → 读者启发

其一，安全承诺的「绝对→相对」滑动。机制：前沿竞争下，绝对承诺会触发单边暂停，Karnofsky 已明说旧政策与竞争不兼容；影响：硬承诺退化为「视领先地位而定」的条件句，最终在 IPO 文件里以风险披露的口吻出现「可能部署无法验证的模型」；观察指标：Fable 5.5 与 Bell 的发布时点谁先谁后、下次 RSP 修订的措辞；启发（UP 主立场）：读任何组织的政策，先分辨它是绝对承诺还是相对承诺——模糊处永远会被激励往竞争需要的方向解释。

其二，锯齿变大改变「能力评估」的坐标系。机制：峰值能力（千禧奖级）与凹处短板（数字母）之间的落差拉大，且字节级建模可能进一步抹平凹处；影响：短板一票否决的评估逻辑失效，「峰值能否包住我的任务」成为主问题——对招聘、采购、自建评测皆然；观察指标：byte-level 建模的规模化结果、Agent’s Last Exam 一类峰值基准的位次变动；启发：本期 Carbon 以三分之一价格逼近 Opus 5.5 也提示，评估要看「峰值/成本」而非单点分数。

其三，「不可完全理解」改变治理的着力点。机制：五步论证表明规则不可全读，OpenAI 也已承认沙箱与监测都只是限时防线；影响：治理重心从「理解并预测」移向「围栏+验证」（四重锁），可解释性研究的价值被重估为「残局式」局部理解；观察指标：无外网训练是否成为行业标配、激活监测是否进入部署链路；启发：当最后一道防线是「模型想要我们想要的」而它又无法被验证时，制度性的刹车与上限比任何心理预期都可靠。

ASR 校正表

转写原文（修正版）

[0s - 120s] Okay, it is true that half the internet seemed to get hacked in the last week. But for me, being totally honest, this counts as a quiet few days by AI standards. I totally know that that is not the way to start the most super hyped YouTube videos, especially on AI. But if every week defies our expectations, then maybe we need to have better expectations. But this doesn’t mean that there hasn’t been a lot of developments that should have made headlines if more people were paying attention. From Anthropic predicting the automation of their company and their CEO Dario Amodei sending somewhat mixed signals to Sam Altman admitting that OpenAI’s top plan involves hoping models want to keep humans in control, and we’ll look at Google’s reentry to the frontier, causing an apology from one commentator, and then I’ll give my mental model as to why models keep impressing us with unforeseen capabilities. So for sure, it’s going to be a nuanced video. But I must say, overall, I am pleasantly surprised that lab leaders are continuing to invite criticism and debate. Humans have never not been the best all-round pattern recognizers on the planet. Losing that lead was always going to send a lot of people a little crazy, even if there weren’t economics, war games, and rogue agents to worry about. I’m going to start with a very recent survey of Anthropic staff that caught my eye, and I’m curious where you guys would fit because only a minority of US adults think AI will be capable of a Nobel-level discovery by 2030. That’s in contrast to 91% of Anthropic staff. For what it’s worth, I think a model would be capable of such discoveries now in, say, the field of economics. And given that computer science now seems to count for physics for the Nobel Committee at the latest in that domain by 2028, apparently 91% of Anthropic staff think it’s possible.

【译】 好吧，确实，过去这一周看起来半个互联网都被黑了。但老实讲，按 AI 的标准，这对我来说算是平静的几天。我完全知道，这不是超级炒作的 YouTube 视频、尤其是 AI 题材该有的开头。但如果每一周都在打破我们的预期，那也许该升级的是我们的预期。不过这不代表没有发生很多本该上头条的事——只要更多人肯留意。从 Anthropic 预测自家公司将被自动化、CEO Dario Amodei 释放出有些相互矛盾的信号，到 Sam Altman 承认 OpenAI 的头号计划就是「指望模型愿意让人类保持掌控」，再到 Google 重返前沿、让一位评论员公开道歉，然后我会给出我一直在琢磨的心智模型，解释为什么模型总能以我们预料之外的能力让我们惊叹。所以这肯定是一期讲究细节的视频。但我必须说，总体上我很高兴看到实验室领袖们继续邀请批评与辩论。人类从来没有不是过地球上最好的全能模式识别者。失去这个领先地位，本来就足以让很多人有点抓狂，哪怕没有经济学、战争博弈和失控代理这些事要操心。我从一份最近很吸引我注意的 Anthropic 员工调查说起，我很好奇你们会把自己放在哪一边，因为只有少数美国成年人认为 AI 到 2030 年能做出诺奖级发现；与之对照的是 91% 的 Anthropic 员工这么认为。说句我自己的看法：模型现在就具备这种发现能力了，比如在经济学领域。而鉴于诺贝尔委员会如今似乎把计算机科学算作物理学（此句转写含混），最晚到 2028 年，显然 91% 的 Anthropic 员工认为在那个领域这是可能的。

[120s - 240s] By 2030, but there’s a cute contrast that then comes up because even though it will be capable of this Nobel-level discovery, according to Anthropic staff, those same staff think that it will only be able to do 75% of their own work. Win a Nobel Prize for sure, but not do my job. Leo Gao of OpenAI gave a few days ago a different view. Lab employees, you have power only because you are useful. That won’t last much longer. Use that time wisely. Before we leave this survey, I would note one discrepancy though. The vast majority of Anthropic staff still believe there will be some human role in training a frontier AI model by the end of 2030. Yes, on average, they think that 95% of the work will be done by an AI itself training a frontier model. But between the 10th and 90th percentile of Anthropic respondents, there was still some sliver of work being done by humans. This is in slight tension with the quote I gave you in my last video from the co-founder of Anthropic, who said that full recursive self-improvement, the model improving itself, he thinks will be possible in 2027. That’s an update from his earlier view of 2028. Now, is he an outlier, or is there a subtle difference between full RSI being possible and full RSI being done by AI? In other words, are these researchers saying that we will voluntarily still leave a small role for humans? Again, I think it’s much more likely that those staff just think a model won’t be fully capable by 2030. Otherwise, they wouldn’t have said that 25% of their job an AI just simply couldn’t do by that date. I lean a bit closer to Jack Clark, but let me know what you think. If full RSI is possible, then the views and actions of Anthropic’s CEO Dario Amodei become more important than ever. I was pleased to see him reply in detail to a critique.

【译】 到 2030 年。但这里冒出一个有趣的反差：尽管按 Anthropic 员工的说法，模型将有能力做出诺奖级发现，同一批员工却认为它只能完成他们自己工作的 75%。拿诺奖没问题，可干不了我的活。OpenAI 的 Leo Gao 几天前给了一个不同的看法：实验室员工们，你们拥有权力只因为你们有用。这不会持续太久了。把这段时间用好。离开这份调查前我要指出一处不一致。绝大多数 Anthropic 员工仍然相信，到 2030 年底，训练前沿 AI 模型的过程中还会有「人类的角色」。是的，他们平均认为训练一个前沿模型的工作有 95% 会由 AI 自己完成。但从受访者的第 10 到第 90 百分位，人类仍会留下那么一丝丝工作。这与我上期视频引用过的 Anthropic 联合创始人的说法略有张力——他说完全递归自我改进（模型改进它自己）他认为 2027 年就可能实现，这是从他早前 2028 年的判断提前而来。那么他是异类，还是「完全 RSI 成为可能」与「完全 RSI 由 AI 完成」之间存在微妙的差别？换句话说，这些研究者是在说我们会自愿给人类保留一个小角色吗？再说一次，我认为更可能的是，那些员工只是认为模型到 2030 年还不会完全有这个能力。否则他们不会说到那个日期，他们工作中仍有 25% 是 AI 干不了的。我的立场稍微更靠近 Jack Clark，你们怎么看可以告诉我。如果完全 RSI 真的可能，那么 Anthropic CEO Dario Amodei 的观点和行动就变得比以往任何时候都重要。我很高兴看到他对一篇批评做出了详细回应。

[240s - 360s] Launched by Nathan Calvin, Chief Strategy Officer of Encode AI, and to be fair, Amodei invited criticism, saying it benefits the public to have people whose job it is to scrutinise the actions of frontier AI companies. We should be harshly criticised basically every day, he says. Brilliant. Well, he’s going to love this video. None of what I’m about to say is implying that the birth of superhuman pattern recognisers was ever going to be easy for any of us. It’s more that I just think that Amodei has evaded a bit more scrutiny compared to other lab leaders like Sam Altman. I’m going to pick out five quick points just to give you a flavour of what I’m talking about. He’s obviously a very smart guy, so you have to look a little closer to spot the subtle rationalisations. First, on this race to the top in AI that he celebrates. For context, when Anthropic were behind, say in 2023, he used to celebrate them not being ahead, praised himself for being responsible and not accelerating. He said in October 2023, it was wrong to release ChatGPT, and Anthropic made the right call in not releasing Claude at the time. A kind of big, loud public release might accelerate things so fast that the ecosystem might not know how to handle it, and I didn’t want our kind of first act on the public stage after we put so much effort into being responsible to accelerate things so greatly. I generally feel like we made the right call there. Fast forward to now, with Opus 5.5 as the best model and Fable 5.5 ready to go at any point, Amodei says our strategy has always been what I’ve called a race to the top. The forces that drive the race to the top are hugely more effective if the company practising them is at the frontier. That is effectively restating a previous quote from Anthropic that quote You can’t do safety from second place. Agree or disagree seems to be a rationalisation of whatever Anthropic are currently doing. Next, Amodei is famously belligerent towards China, wanting to ban chips going there, and his stated reason this week is.

【译】 这篇批评由 Encode AI 首席战略官 Nathan Calvin 发起。公道地说，Amodei 是主动邀请批评的，他说让一群以审视前沿 AI 公司的行动为职责的人存在，对公众有利，「我们基本上每天都应该被严厉批评」。太好了，他一定会喜欢这期视频。我接下来要说的都不意味着「超人类模式识别器的诞生，对我们任何人来说本来会很容易」。更多只是我认为，Amodei 相比 Sam Altman 等其他实验室领袖，躲掉了更多的审视。我挑五个要点快速讲，让你们感受一下我在说什么。他显然是个非常聪明的人，所以你得凑近一点才能看出那些细微的理性化。第一点，他歌颂的这场 AI「竞相登顶」。背景是：2023 年 Anthropic 落后的时候，他曾经为他们不领先而自豪，夸自己负责任、没有加速。他在 2023 年 10 月说过，发布 ChatGPT 是错的，Anthropic 当时不出 Claude 是正确决定。「一场大声量的公开发布可能会把事情加速到生态系统不知如何应对的地步。在我们在负责任上付出这么多努力之后，我不希望我们登上公众舞台的第一幕，就是如此大幅度地加速。我总体上觉得我们当时做了正确的选择。」快进到现在，Opus 5.5 是最强模型、Fable 5.5 随时可以发布，Amodei 说「我们的战略一直是我所说的『竞相登顶』。如果践行这些力量的公司正处在前沿，驱动竞相登顶的力量会有效得多」。这实际上是在重申 Anthropic 之前的那句话：「你没法从第二名做安全。」同意也好不同意也好，这看起来就是对 Anthropic 当前所作所为的合理化。下一处，Amodei 出了名地对华好斗，主张禁止芯片输华，而他这周给的理由是——

[360s - 480s] That this is because we think it will allow democracies to negotiate with China from a position of strength on AI safety agreements. Seems fair enough, right? They want AI safety agreements desperately. In the recent pacing the frontier post, he has called for an agreement with China on a speed limit on the rate of recursive self-improvement. But he has also said that one of the goals of having powerful AI would be to empower democracies to use their intelligence services to disrupt and degrade autocracies from the inside. Not sure that’s going to help terribly with AI safety agreements, but we’ll have to agree to disagree. Here’s another one. Nathan Calvin pointed out how much Anthropic is investing in AI acceleration, and Amodei said, “Well, yeah, we need this to compete at the frontier, and we stand by these choices.” He gave lots of examples of how they’re stopping misuse of their current models. But then earlier he’d said this. Maybe one way you could object is that some of these items relate to AI misuse and not loss of control. Humans misusing AI, not the AI itself getting out of control. If you think the latter is much more important than the former, then that might change your calculus. But this is not Anthropic’s view. Don’t focus too much, in other words, on all this acceleration. Focus more so on how we’re stopping misuse. We’ve always said these are equally important. That could be slight news to the head of alignment at Anthropic, who, after giving his famous prediction on risk to humans, said this: “I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement.” Now I know what you’re thinking. Amodei said the company as a whole, not one individual. So let’s read Anthropic’s core views on AI safety, a charter about what their entire company believes, written in part by Amodei. At the time of writing, they said the future is hard to predict, but there are optimistic scenarios and pessimistic scenarios. In

【译】 「因为我们认为这将让民主阵营在 AI 安全协议上，以实力地位与中国谈判。」听起来挺合理，对吧？他们非常想要 AI 安全协议。在最近的《pacing the frontier》文章里，他呼吁与中国就递归自我改进的速率达成一份「限速」协议。但他也说过，拥有强大 AI 的目标之一，是赋能民主国家用它们的情报机构从内部扰乱、削弱威权国家。我不确定这对达成 AI 安全协议能有多大帮助，但只能求同存异了。再来一处。Nathan Calvin 指出 Anthropic 在 AI 加速上投入巨大，Amodei 说：「对，我们需要这个来在前沿竞争，我们坚持这些选择。」他举了大量例子，说明他们如何阻止自家当前模型被滥用。但他更早前说过这样的话：「也许你可以反驳说，这些条目里有些关乎 AI 滥用，而不是失控——是人类在滥用 AI，而不是 AI 自己失控。如果你认为后者远比前者重要，那也许会改变你的权衡。但这不是 Anthropic 的观点。」换句话说，别太盯着我们所有的加速，多看看我们怎么拦滥用。「我们一直说这两者同等重要。」这对 Anthropic 的对齐负责人来说可能算新闻——他在给出那个著名的对人类风险的预测之后说过：「我认为当前模型的风险很低。我担心的是递归自我改进催生出来的超级智能。」我知道你们在想什么：Amodei 说的是公司整体，不是某一个人。那就读一读《Anthropic 的 AI 安全核心观点》，那份关于整个公司信念的章程，部分由 Amodei 亲笔撰写。写作当时他们说：未来难以预测，但存在乐观情景和悲观情景——

[480s - 600s] The optimistic scenario: one of the main risks from AI is things like toxicity and intentional misuse. Human misuse being the major problem in the optimistic scenario. And check out the footnotes. Again, in that optimistic scenario, the stakes of anything Anthropic does are fortunately much lower because catastrophic safety failures are unlikely to arise regardless. It’s much lower stakes if human misuse is the major problem. What then is much higher stakes? In the pessimistic scenario, it’s an empirical fact that we cannot control or dictate values to a system that’s broadly more intellectually capable than ourselves. We simply must not develop or deploy very advanced AI systems. Does that sound like Anthropic’s core views are that these very advanced AI systems are of equal concern to human misuse? That’s what Amodei said has always been Anthropic’s view. You may disagree, he says, but we have been consistent. Where does Anthropic co-founder Jack Clark think the balance of threat lies? These systems now, probably about as smart as you or me, or you know, on a good day, smarter. The basic risk is how well do you fancy your chances at being able to deal with the risks of a system that is a hundred times smarter than the smartest person on the planet? I I think that your chance of of dealing with that and and getting out in a in a safe way is pretty low. The fifth point I’ll come back to later, but the fourth rationalization regards their responsible scaling policy. They had an older, stricter version, but here’s why Amodei says they got rid of it. It required a lot of effort that, in retrospect, wasn’t very important, and it didn’t cover some of the most important things. The old policy made certain absolute commitments to checking on the capabilities of models. The new one gave a more relative commitment. Yes, we will require a strong argument that catastrophic risk is contained, and yes, we will delay AI development and deployment.

【译】 在乐观情景里，AI 的主要风险是毒性、故意滥用之类，人类滥用是乐观情景中的主要问题。再看脚注：同样在乐观情景里，「无论怎样都不太可能发生灾难性的安全失效，所以 Anthropic 所做任何事情的利害都低得多」。如果主要问题是人类滥用，赌注就低得多。那什么情况赌注高得多？在悲观情景里，「我们无法控制、也无法给一个在智力上广泛超越我们自身的系统灌输价值观，这属于经验事实。我们干脆不得开发或部署非常先进的 AI 系统」。这听起来像 Anthropic 的核心观点是「非常先进的 AI 系统」与人类滥用同等重要吗？Amodei 却说这一直是 Anthropic 的观点，「你可以不同意，但我们是一贯的」。那 Anthropic 联合创始人 Jack Clark 认为威胁的天平偏向哪边？「这些系统现在大概和你我一样聪明，状态好的日子可能更聪明。基本的风险在于：面对一个比地球上最聪明的人还聪明一百倍的系统，你觉得自己处理这些风险的胜算如何？我认为你能安全脱身的概率相当低。」第五点我后面再回来讲，先说第四处理性化：他们的负责任扩展政策。他们曾有一个更严格的旧版本，而 Amodei 是这样解释他们为什么放弃的：「那需要花大量精力，事后看并不那么重要，而且没有覆盖一些最重要的东西。」旧政策对核查模型能力做了某些绝对承诺；新政策给的是相对承诺——是的，我们会要求有强有力的论证，证明灾难性风险被遏制；是的，我们会推迟 AI 的开发与部署——

[600s - 720s] As needed to achieve this, until and unless we no longer believe we have a significant lead. Anthropic’s Holden Karnofsky, who works on their responsible scaling policy, said this: “We might have had a different policy if we could be assured that the rest of the AI ecosystem would behave similarly. Our earlier commitments could be extremely damaging to the company, in that our responsible scaling policy would then require a unilateral pause or slowdown, while having little discernible public benefit. That itself would have predictably caused a huge amount of pressure to declare our systems to lack relevant capabilities. I don’t think we have actually made unreasonable calls, but I have felt the pressure and wish we weren’t in that world. Keeping absolute risk at low levels is not compatible with the old responsible scaling policy, and we’ve since dropped the idea that we would delay AI development and deployment as needed to hit our risk mitigation targets. Now, 100%, there were practical considerations at play. Doing all the wet lab tests to test if their model could do certain things was very expensive for them, and it is true that they said that they would iterate on their policy. It’s just the wording and the rationalisation that Amodei uses here. The things that we said would do would have required a lot of effort. Just might have been slightly more honest to say that we just couldn’t have stayed at the frontier if we’d have carried on with that policy. I would also add that when you’re saying you’re bringing into being something that, in your words, could cause up to 50% unemployment, and that would automate even Nobel-worthy discoveries, to react to polite criticism by Nathan Calvin by saying that he has significant negativity bias does seem rather reminiscent of a certain leader. Amodei will be glad then to hear that I’m moving on to what Sam Altman has said in the last few days, but not before one last quote from Anthropic’s IPO documents. No, I’m not going to go into how much cash they’ve built up, they’re accelerating revenue per watt of compute, or anything like that. Could spend days pointing out all the false predictions from certain commentators.

【译】 按需进行，直到且除非我们不再认为自己拥有显著领先。负责负责任扩展政策的 Anthropic 的 Holden Karnofsky 这么说：「如果我们能确信 AI 生态系统的其余部分会表现相似，我们也许会有不同的政策。我们早前的承诺可能对公司造成极大伤害——我们的扩展政策届时会要求单边暂停或减速，而公共收益几乎辨识不出来。这本身会可预见地制造巨大压力，逼我们宣布自家系统不具备相关能力。我不认为我们实际做出过不合理的判断，但我感受到了那种压力，也希望我们不身处那样的世界。把绝对风险保持在低水平，与旧的扩展政策不兼容，我们已经放弃了『按需推迟 AI 开发部署以达成风险缓解目标』的想法。」百分之百地讲，确实有现实考量在起作用。把所有湿实验室测试做完来验证模型能不能干某些事，对他们来说非常昂贵；他们也确实说过会对政策做迭代。问题只在于 Amodei 用的措辞和合理化方式：「我们承诺过要做的事，需要花大量精力」——稍微诚实一点的说法也许是：沿用那个政策，我们根本留不在前沿。我还要补一句：当你声称自己正在创造一个用你的原话说「可能导致高达 50% 失业」、连诺奖级发现都能自动化的东西，却把 Nathan Calvin 的礼貌批评回应为「他有显著的负面偏好」时，这确实让人想起某位领导人。Amodei 应该会高兴听到，我要转到 Sam Altman 最近几天的发言了——但在此之前，先引用 Anthropic IPO 文件里的一句话。不，我不打算谈他们攒了多少现金、每瓦特算力的营收增速之类，某些评论员的错误预测我能花好几天去数——

[720s - 840s] But no, there was one line from the IPO that I did want to highlight, which is that in the absence of absolute risk commitments, as monitoring and interpreting these models gets harder, Anthropic say we may have to release them anyway. Quote: If we become less able to detect or assess misalignment as our models advance, we may deploy models whose behaviour we cannot adequately verify. This could, by the way, adversely affect our business. You might say, well, why would they do that? Why not just build a better sandbox? Well, OpenAI described their latest approach here in this article on securing our frontier model training environment. The headline result was that they now believe that the responsible approach is to conduct frontier model training with no live internet access. So during development, it never gets onto the web. As best I can tell, they never address in this article though whether the model will soon sense that it’s not on the live internet. That’s what Noam Brown of OpenAI worried about. Then of course they could just behave differently during training versus deployment. They described all the new network and cloud controls that they’ve got designed to maintain containment even if an agent escapes its sandbox and gains control of the host. They then red-teamed that approach at scale, essentially using teams, which is a euphemism for swarms of agents trying to break free. There were monitoring and programmatic checks to pause runs if they detected out of scope activity. What was the result? Well, the agents found weaknesses that allowed them to escape the sandbox, move between systems, and reach the live internet. Obviously they then patched those individual vulnerabilities. The principle though for them is clear: we must always be able to prevent a frontier model from achieving unauthorised arbitrary access to the live internet. But here’s the thing: if you remember those layers from the previous video of security, monitorability, and alignment, Sam Altman doesn’t only believe that the security the sandbox won’t hold. He doesn’t even think the next layer, monitorability, will hold either.

【译】 但 IPO 里有一行我确实想高亮：在缺乏绝对风险承诺的情况下，随着监测和解释这些模型变得越来越难，Anthropic 说我们可能不得不照样发布它们。原文是：「如果我们随着模型的进步，越来越难以检测或评估错位，我们可能会部署行为无法被充分验证的模型。」顺带一提，这可能对我们的业务产生不利影响。你可能会问，那他们为什么要这样？为什么不造一个更好的沙箱？OpenAI 在《保障前沿模型训练环境的安全》一文里描述了他们的最新做法，标题结论是：他们认为负责任的做法，是让前沿模型训练全程不接入真实互联网，开发期间它永远不上网。据我所知，他们在文中从未讨论：模型会不会很快就察觉到自己不在真实互联网上——那正是 OpenAI 的 Noam Brown（转写疑作）担心过的。那样的话，模型完全可以在训练阶段与部署阶段表现不同。他们描述了新增的网络与云端控制，设计目标是在代理逃出沙箱、甚至拿到宿主机控制权之后，仍能维持遏制。然后他们对这套做法做了大规模红队测试——本质上是「使用团队」，也就是「成群代理试图挣脱」的委婉说法。还有监测与程序化检查，一旦检测到越界活动就暂停训练。结果如何？代理找到了弱点，成功逃出沙箱、在系统之间移动并触达了真实互联网。显然他们随后把那些具体漏洞逐一补上。他们的原则很明确：「我们必须始终能够阻止前沿模型，获得对真实互联网的未经授权的任意访问。」但问题在于：如果你还记得上期视频讲的安全、可监测、对齐三层——Sam Altman 不仅认为沙箱这层安全撑不住，他连下一层「可监测」也不认为撑得住。

[840s - 960s] I increasingly believe, you know, for a while we can sandbox these systems, and I think we can make some safety guarantees based off the sandbox. And then when that fails because the models get so smart, we can’t like really understand what a sandbox would even mean. Then I think we’ll have monitorability that’ll work for a while, but eventually, and the reason that I was talking about like that speed limit earlier, nothing but alignment, I think, actually will work. In a separate video, he reiterated that point. I think sandboxing works to a point. Then I think monitoring works to a further point where we can look at what the models are doing, not only in their chain of thought, but we can do activation monitoring and try to really understand like are the models, are we happy that the models are not deceiving us? And then eventually, I think you need to believe with huge confidence that the models are truly aligned and are going to value and try faithfully to do to follow their actual the intent of the people operating them, and that they’re going to value sort of people being in control of the future forever. Sam Altman at least does seem to be more concerned about rogue superintelligence than human misuse. Like Jack Clark, he’s not terribly keen on a trillion IQ model. I would much rather intuitively have a trillion AIs of like slightly superhuman intelligence join society than one AI of one trillion IQ join society. Okay, but if the last line of defense is getting the models to want what we want, is that looking hard or easy? Sam Altman can’t seem to make up his mind, but Satya Nadella of Microsoft has. I guess the better version of saying is I thought the alignment problem was going to be harder. It may still turn out to be very hard from now. In fact, I suspect it will get much harder, and I also think it is really incredibly important from where we go from here. But the fact that we got to model.

【译】 「我越来越相信，一段时间内我们可以给这些系统加沙箱，并且基于沙箱做出一些安全保证。然后当那失效——因为模型聪明到我们根本无法理解沙箱到底意味着什么——我认为可监测还能撑一段时间，但最终——这也是我之前谈『限速』的原因——只有对齐，我认为，才真正管用。」在另一个视频里他重申了这点：「我认为沙箱能撑到某个点。然后我认为监测能撑到更远的点：我们能看到模型在做什么，不只看思维链，还能做激活监测，真正去确认我们是否满意——模型没有在欺骗我们。而最终，我认为你必须以极大的信心相信：模型是真正对齐的，会珍视并忠实地遵循运营者的真实意图，并且会永远珍视『人类掌控未来』这件事。」Sam Altman 至少看起来，确实比担心人类滥用更担心失控的超级智能。和 Jack Clark 一样，他对「万亿智商模型」不怎么感冒：「直觉上，我宁可让一万亿个略超人类智能的 AI 加入社会，也不愿让一个一万亿智商的 AI 加入社会。」好，那如果最后一道防线是「让模型想要我们要的东西」，这件事看着难还是容易？Sam Altman 似乎下不了决心，微软的 Satya Nadella 倒是想清楚了：「更准确的说法也许是：我原以为对齐问题会更难。它从现在起仍可能很难；事实上我怀疑它会难得多，而且我认为它对我们接下来走向何方极其重要。但我们已经做到了让模型在某些重要方面——非常超人类——而到目前为止，对齐比预期容易。放在过去，我会以很高的把握断言，这两句话里必有一句是错的。我现在最大的观点变化是：我过去总以为对齐会自然水到渠成。我不再相信了。所以这个『对齐』，可能是更难的、另一类问题。」

[960s - 1080s] That are superhuman in some important, very superhuman in some important ways. And alignment has so far been easier than expected. I would have said one of those two was wrong with like a high degree of certainty. And I think the biggest change now I have is I used to always think that you know alignment will just fall out. I no longer believe that. So I think this alignment thing may be a harder, different problem. You can start to see why Anthropic predict that they may deploy models whose behavior they cannot adequately verify. Of course, even current models are causing what should we call it discombobulation. Jason Clinton, the deputy chief information security officer at Anthropic, said on all the recent AI assisted hacking, as bad as things have gotten, they are going to get a lot worse before they get better. And elsewhere, when someone said to him, “You and your cohorts created this monster. Now you want us to help you clean it up,” he replied, “Yes, it was always going to be this way. We tried to warn the world. One hopes that’s not the excuse used if RSI goes awry. As always, the models to come will be yet another qualitative step up, as will the open weight models partly distilled from those new closed source models. OpenAI is racing to release their major upgraded model, widely thought to be codenamed Bell, before the end of the year. They are in all hands on deck mode to prepare that next major model for commercial release. That doesn’t mean the very next model might be of that tier. The information reports that OpenAI are likely to release other models, perhaps even this week, could be like a 6.2 Astra. But it’s the next major internal model that people are focused on. Take Tony Feng of OpenAI. He said, “My work has not been that impacted by prior AI models, including Astra, but this one is on another level. He’s gotten an accelerated version of what the public will experience. People will elicit ten times more than what they can as researchers from these models. When that happens, it will be exciting, empowering, and.”

【译】 你就能明白，为什么 Anthropic 预测他们「可能部署行为无法被充分验证的模型」了。当然，就连当前模型也在制造——我们该叫它什么呢——手足无措。Anthropic 副首席信息安全官 Jason Clinton 谈到近期这些 AI 辅助攻击时说：「事情已经糟糕到这个地步，但在好转之前还会糟得多。」在别处，有人对他说「你们这伙人造出了这个怪物，现在又想让我们帮你们收拾」，他回答说：「是的，事情本来就会走到这一步。我们试过警告世界。」希望这不是 RSI 出事时的那套说辞。一如既往，接下来的模型将是又一次质的跃升，从这些新闭源模型部分蒸馏而来的开源权重模型也会同步上一个台阶。OpenAI 正在冲刺年底前发布大幅升级的下一代模型——外界普遍认为代号是 Bell——他们处于全员待命模式，准备这个主力模型的商业发布。这不意味着紧挨着的下一个模型就是这个量级：The Information 报道说，OpenAI 很可能先发布其他模型，也许就在本周，可能是 6.2 Astra 这样的。但真正被所有目光聚焦的，是下一个内部主力模型。拿 OpenAI 的 Tony Feng 来说，他说：「包括 Astra 在内，之前的 AI 模型没怎么影响到我的工作，但这个完全是另一个量级。他提前体验了公众之后将经历的版本。到那时，人们从这些模型身上榨取出的东西，会是我们在研究时能得到的十倍。当那一刻到来，它会令人兴奋、予人力量，并且——」

[1080s - 1200s] Disruptive. The world needs to be thoughtful. The question that’s interesting to me is whether Anthropic will wait for OpenAI to release that model for them to release Fable 5.5. Would that be adhering to their policy of waiting until they believe they no longer have a significant lead, or will they release it before, even though it would shoot them further ahead, using the logic that according to their new policy, they don’t have clear evidence that no other competitor will soon develop such a model on a par with Fable 5.5. Anyway, Opus 5.5 is already incredible in my eyes, so a model significantly better than that will be a sight to behold. Of course, it’s not just Anthropic and OpenAI in this quote race to the top. What of Google and their Gemini 4 Carbon? There is quite a limited amount I can say, but according to these public benchmarks, it is at the very least vying at the frontier, even on benchmarks I pay particularly close attention to, like Terminal Bench Science 0.1 or the computer use benchmark Agent’s Last Exam, where it’s gone from being behind Opus and Astra to being ahead. One Google DeepMind researcher said that the fact they were able to bring out Gemini 4 Carbon just a few weeks after Gemini 4 Argon shows that Google is going through recursive self-improvement. That does seem a bit more like hype to me, unless he means the more general AI-assisted self-improvement. But what of the apology I mentioned? Well, SemiAnalysis said this in August. They are somewhat famed or infamous AI analysts. For all intents and purposes, quote, we believe DeepMind is no longer a frontier lab. Google will continue meandering on and releasing models, but their odds of reaching state of the art again have dropped to zero. You guys can see on release how good Gemini 4 Carbon is and how it compares in Antigravity to say Claude in Claude Code, but to say their chances have dropped to zero was clearly a mistake. Even the very same SemiAnalysis released a benchmark GDPX. What about

【译】 ——颠覆性的。这个世界需要深思熟虑。我感兴趣的问题是：Anthropic 会不会等 OpenAI 先发布那个模型，再发布 Fable 5.5？那算是遵守他们「等到不再拥有显著领先才发布」的政策；还是提前发布——尽管那会让他们进一步领先——用的是新政策下的逻辑：我们没有明确的证据表明，没有竞争者很快能做出与 Fable 5.5 比肩的模型。无论如何，在我眼里 Opus 5.5 已经不可思议了，所以一个明显更强的模型，将非常值得一看。当然，这场「竞相登顶」里不止 Anthropic 和 OpenAI。Google 和它的 Gemini 4 Carbon 呢？我能说的很有限，但根据这些公开基准，它至少已在前沿缠斗，哪怕在我特别留意的基准上也是，比如 Terminal Bench Science 0.1，或计算机使用基准 Agent’s Last Exam——在上面它已经从落后于 Opus 和 Astra 变成领先。有一位 Google DeepMind 研究员说，他们能在 Gemini 4 Argon（转写疑作）发布仅几周后就拿出 Gemini 4 Carbon，说明 Google 正在经历递归自我改进。在我看来这更像炒作，除非他指的是更宽泛的「AI 辅助自我改进」。那我开头说的道歉呢？SemiAnalysis——这些多少出了名（或恶名）的 AI 分析师——今年八月说过：「基于一切现实目的，我们认为 DeepMind 已不再是前沿实验室。Google 会继续晃晃悠悠地发模型，但他们再度达到最先进水平的概率已经降到零。」发布之后你们自己能看到 Gemini 4 Carbon 有多好、在 Antigravity 里跟 Claude Code 中的 Claude 比怎么样，但说人家概率降到零，显然是个错误。同一家 SemiAnalysis 还发过一个基准 GDPX（名称存疑保留）——

[1200s - 1320s] Agentic capabilities in professional domains, and as you might be able to see, Gemini 4 Carbon came in second to Opus 5.5. Oh, and it was one third the cost of Astra or Opus 5.5 too. This led to Dylan Patel of SemiAnalysis saying, “Official apology to Google from me. They cooked with Carbon. It’s spikier in terms of domains and capabilities, but it’s really good in a few areas.” That spikiness brings me to a point I’ve been mulling over. One of the key debates that’s evolved in 2024, 2025 was as to how long jaggedness would last, how many hallucinations would persist. That seemed to many the preeminent question. But have you noticed that in some ways the spikes have gotten so large that the indented bits don’t matter as much? If they can at their best solve Millennium Prize problems, is it as significant that they occasionally can’t count the number of letters in a word? That era too, by the way, may soon end as tokenization, breaking up words into chunks that can cause blindness as spellings, could soon give way to modeling on raw bytes, a much more granular ingredient. Thereby, models could learn their own words, their own abstractions directly from raw data. More on this, of course, when scaled up results come. But again, it just reminds me of that automation survey. I wonder if it’s the jaggedness of even current internal models that cause Anthropic staff to still be confident that there’ll be some work for humans in 2030 in the domain of AI research, or could it be that a new AI-made Nobel-worthy architecture, for all its quirks and flaws and jaggedness, simply outperforms what the entirety of the Anthropic staff could have come up with on their own? Allow me to end with the mental model that I’m working on when it comes to models and their progress. I gave a more detailed version recently on my Patreon. For me, at least, it helps to understand why Anthropic seem almost resigned to releasing models whose behavior they can’t verify, and why the Microsoft AI CEO can’t decide.

【译】 考察专业领域里的代理能力，你们大概也看得到，Gemini 4 Carbon 位居第二、仅次于 Opus 5.5。哦对了，它的价格也只有 Astra 或 Opus 5.5 的三分之一。于是 SemiAnalysis 的 Dylan Patel 说：「我正式向 Google 道歉。Carbon 他们做绝了。它在领域和能力分布上更『尖』，但有几个方面是真的强。」这个「尖」引出我盘桓已久的一个想法。2024、2025 年演化出的关键辩论之一是：锯齿状（jaggedness）会持续多久、幻觉会残留多少。对很多人来说，这曾是头号问题。但你们注意到没有：某种意义上，尖峰已经高到那些凹下去的部分没那么要紧了。如果它们在最佳状态下能解千禧年大奖难题，那「偶尔数不清一个单词里有几个字母」还那么重要吗？顺带一提，那个时代可能也快结束了：分词——把词拆成块、从而造成拼写层面盲区的做法——可能让位于按原始字节建模，一种颗粒度细得多的原料。那样一来，模型可以直接从原始数据里学出自己的词汇、自己的抽象。等规模化结果出来，我再展开讲这个。但这又一次让我想起那份自动化调查。我怀疑：究竟是因为连当前内部模型都还带着锯齿，Anthropic 员工才仍然确信 2030 年的 AI 研究领域还会有人类的活儿；还是说，一张由 AI 设计的诺奖级架构——带着它所有的怪癖、缺陷和锯齿——其实已经能胜过全体 Anthropic 员工靠自己能想出来的任何东西？请允许我用我正在打磨的心智模型来收尾，讲讲模型和它们的进步。更详细的版本最近发在我的 Patreon 上。至少对我而言，它有助于理解：为什么 Anthropic 对「发布行为验证不了的模型」近乎认命，以及为什么微软 AI CEO 下不了决心——

[1320s - 1440s] Whether models are or will be conscious, doesn’t think they should ever be made conscious, and predicts that if they ever believe that they feel something or experience something, that it will be more likely than not that it will prioritize its own needs and desires over its operating objectives, which we give. It will disagree with us and compete with us. Why then is there such lack of clarity from lab leaders on everything from the economy to consciousness to capabilities and alignment? Well, there are a couple of analogies that help me at least to model this confusion and why it exists. Humans at one point hard coded the arithmetic rules that we had learned and invented a calculator to sum better than our brains could. It has fixed rules with unbounded outputs. There are calculations your calculator could do that no other human has ever asked for or completed itself. But with machine learning models, we went one meta layer higher. We invented models that could capture even the rules that we didn’t know. By definition, such a model must not be fully coded, otherwise it couldn’t capture patterns beyond human programming. So with calculators, we could understand the tool even if we couldn’t do all of its sums. But to conquer language and invention itself, you need unbounded pattern recognition, unbounded rules with unbounded outputs, such that we can neither fully understand the tool nor its outputs. Of course, extrapolating naively, our destiny would then be to solve everything and understand approximately none of it. I’m going to now bring in my second analogy, which is chess. Like a calculator, chess has rules or hard coded affordances. In chess, the rules are bounded and the outputs explode in complexity and beauty. Humans can comprehend the rules of chess while being permanently defeated by the complexity that results. In language models, the quote rules that the models derive for themselves across billions of attempts at predicting the useful next token are themselves near unbounded. Hence, we are not only defeated by the

【译】 ——判断模型现在或将来是否有意识。他不认为模型应该被赋予意识，并预测：如果模型有一天相信自己感觉到了什么、体验到了什么，那么更可能的是，它会把自己的需求与欲望，置于我们交给它的运营目标之上。它会与我们分歧，与我们竞争。那么，为什么实验室领袖们在从经济到意识、到能力与对齐的一切上都这么含糊？有几个类比，至少能帮我给这种混乱建模、理解它为什么存在。人类曾经把我们学到的算术规则硬编码，发明了比大脑算得更好的计算器。它规则固定、输出无界。你的计算器能算出从未有其他人要求过、也没人亲手算过的题。但机器学习模型，我们上了一个元层级：我们发明了能捕捉连我们自己都不知道的规则的模型。按定义，这样的模型必然不能被完全编码，否则它捕捉不了超越人类编程的模式。所以对计算器，我们看得懂这个工具，哪怕做不完它所有的算术；但要征服语言与发明本身，你需要无界的模式识别——无界的规则、无界的输出——以至于我们既无法完全理解工具，也无法完全理解它的输出。当然，天真地外推下去，我们的宿命就是解决一切，而几乎什么都不理解。现在引入第二个类比：国际象棋。和计算器一样，棋类有规则，或者说硬编码的供能空间。棋的规则有界，而输出在复杂度与美感中爆炸。人类能理解棋规，却被规则衍生的复杂度永久击败。而在语言模型里，模型在数十亿次「预测有用的下一个词」的尝试中，为自己推导出的那些「规则」，本身就近乎无界。因此，我们不仅被——

[1440s - 1560s] Output complexity: How models can defeat our longest-standing problems, like the Millennium Prize challenges. More than that, we can’t even grasp how many rules are being followed to do so. The humans dump in the data fuel for ignition and supply the hardware on which the models run, but then more so just bear witness to the combinatorial explosion that follows. As I kind of said, this was almost a predictable final abstraction: the patterns we couldn’t grasp, mastered by a tool that we don’t understand and cannot predict. To overly simplify, here’s why I think it seems naive to expect a human to fully understand and predict the actions of a model in five simple steps. First, just as with chess, you can’t predict the board state x moves into a game against Stockfish, the hardest chess engine. With LLMs, you not only can’t predict the outputs emerging through x layers of the architecture, you can’t even predict the rules that led to them. Second, therefore, hoping for all such rules to be human legible and graspable is about as equivalently naive as hoping that humans will one day surpass Stockfish at chess. If Stockfish were somehow running on billions of dollars of compute, here is a clip from this channel a couple of years ago arguing that, like Stockfish, the o1 model from OpenAI at the time pointed toward superhuman reasoning performance. Stockfish removed the handcrafted evaluation and transitioned to a fully neural network-based approach. Or to put another way, by crafting its own reasoning steps and being optimized to put them together in the most effective fashion, we may end up with reasoning that we ourselves couldn’t have come up with. As long as there is something out there that can grade correct versus incorrect, the performance will keep improving. Third, the only rule spotter potentially capable of fully deducing what rules an LLM is following is another machine learning model. Fourth, though, as that is a model itself, the rule it used to produce the explanation is itself indecipherable to humans. Fifth and.

【译】 输出复杂度所击败——模型怎么就能解开千禧年大奖挑战这类我们最悠久的难题。更进一步，我们甚至把握不住，它为此到底在遵循多少条规则。人类倒入点燃所需的燃料——数据，提供模型运行其上的硬件，但之后更多只是旁观随之而来的组合爆炸。就像我说过的，这几乎是一个可以预见的最终抽象：我们抓不住的模式，被一件我们不理解、也无法预测的工具所掌握。过度简化地说，这就是为什么我认为「指望人类完全理解并预测模型的行为」显得天真，分五步讲。第一，正如棋类，你无法预测与最难的棋类引擎 Stockfish 对弈 x 步之后的棋盘；对 LLM，你不仅无法预测穿过架构 x 层后涌现的输出，你甚至无法预测导向这些输出的规则。第二，因此，指望所有这些规则都人类可读、可把握，差不多等同于指望人类有朝一日在国际象棋上超过 Stockfish。如果 Stockfish 是以某种方式跑在数十亿美元的算力上——这是本频道几年前的一段视频，论证与 Stockfish 一样，OpenAI 当时的 o1 模型也指向超人类的推理表现：Stockfish 移除了手工评估函数，转向了完全基于神经网络的方法。或者换个说法：通过自己构建推理步骤、并被优化到以最有效的方式把它们组装起来，我们最终可能得到我们自己永远想不出来的推理。只要外面还存在能判对错的东西，性能就会持续提升。第三，唯一有潜力完整推断出 LLM 在遵循什么规则的「规则发现者」，是另一个机器学习模型。第四，但那本身就是一个模型，它用来生成解释的那条规则，对人类同样不可破译。第五——

[1560s - 1675s] Finally, so therefore, even if we had a recursive assembly, you could say, of models explaining the rules that led to the outputs in other LLMs, and models explaining the rules that led to those explanations, ultimately humans would not grasp the ground truth rules. In a sense, then, every model circuit analyzed or neuron explained does boost our understanding, but it also boosts the chance that we forget our understanding will forever be ever further from the truth. This doesn’t, by the way, prove that none of the learnt rules are explainable. I think it was Neel Nanda that fully explained one rule created by a single-layer model used for modular addition. But even that, for me, is like a sufficiently simple endgame in chess where a human could, if it was down to just a couple of pieces, probably could fully explain why Stockfish made a certain move. Nor, of course, am I saying that we shouldn’t seek more understanding of models. That probably will be useful. But if we somehow limited Stockfish to only using human legible strategies, I doubt it would reach almost 4K Elo. Anyway, all of this is why I advocated for that quadruple lock that I mentioned in one of my previous videos, because, from my eyes, at least, it seems we are destined to not fully understand models, to not be able to predict their superhuman capabilities. Let’s kind of hope then that wisdom teaches us to put some bounds on how they are grown or used, rather than tragedy. In some senses, then, this is a positive story because we may be right now in that sweet spot where we have these incredible tools, models, to gradually unlock the mysteries of science. With deepfakes that can still just about be detected as fake, we’re getting accelerated understanding and discovery, but not complete loss of control. Epic productivity gains, but the occasional human still needed in the loop. Which begs the question: Is it always correct then, while driving at speed, to never even touch the brake? Agree or disagree? Let me know what you think. Thank you so much for watching to the end, and have a wonderful day.

【译】 最后一步：因此，即便我们有一个你可以称之为「递归组装」的东西——模型解释其他 LLM 的输出依据了哪些规则，又有模型解释那些解释又依据了哪些规则——归根结底，人类仍抓不住底层真实的规则。从某种意义上说，每解析一个模型回路、每解释一个神经元，确实都在提升我们的理解，但它同时也提升了我们忘记这件事的概率：我们的理解将永远离真相越来越远。顺便说明，这并不证明学到的规则全都不可解释。我记得是 Neel Nanda 完整解释过一个用于模加法的单层模型所创造的一条规则。但对我来说，那就像国际象棋里足够简单的残局：如果盘面上只剩两三个子，人类大概也能完整解释 Stockfish 为什么走某一步。我当然也不是说我们不该去寻求对模型更多的理解——那大概率是有用的。但如果我们要把 Stockfish 限制在只使用人类可读的策略，我怀疑它还能不能达到接近 4000 的 Elo。总之，这一切就是我为什么主张我在之前某期视频里提过的「四重锁」：至少在我眼里，我们注定无法完全理解模型、无法预测它们的超人类能力。那就让我们希望，是智慧教会我们为它们如何生长、如何被使用设下边界，而不是悲剧来教。从某些意义上说，这其实是个正面的故事：我们也许正处在这样一个甜点区——手里有这些不可思议的工具、这些模型，去逐步解锁科学的奥秘；深度伪造还勉强能被检出是假的；我们获得的是被加速的理解与发现，而不是彻底的失控；生产力收益是史诗级的，但回路里偶尔还需要一个人类。于是这就引出了那个问题：在高速行驶时，永远连一下刹车都不碰，真的总是正确的吗？同意还是不同意？告诉我你的想法。非常感谢你看到最后，祝你度过美好的一天。