AI:AM 周精选复盘：Le Chonk 的 1e25「成人礼」，与「边跑边修」的补丁文化

信源说明：本文基于 AI in the AM（Cognitive Revolution 频道）2026-10-31 发布的周精选视频（视频 ID MdpamRtlw0Q，全长约 87 分钟，英文转写 44 段、0s–5202s 已完整读取）。周精选是本周一至周五直播的重剪合集，旁白为主持人 Nathan 的克隆语音（节目自述）。主体访谈与新闻本站已按直播原文成文：10-28 场（Kepler Computing、Arena、Union 三位嘉宾＋当周新闻）见《二十轮对话过半出现失当行为》，10-30 场（ServiceNow 威胁研究、Mistral、Binance 失窃案）见《63 亿美元失窃与 5 美分攻击》。本文不复述已展开的背景，只做三件事：收录重剪版的增量信息、对齐两篇长文与本期的口径差异、附完整转写原文。人名经节目官网与公司官网核实：主持人 Nathan Labenz、Prakash Narayanan；Kepler Computing CEO Debo Olaosebikan；Union 创始人 Samvit Ramadurgam（本站前作转写作 Samvith，自本文起统一为 Samvit）；Arena CEO Anastasios Angelopoulos。全部数字与判断均为主持人或嘉宾口径，本文未做独立核实；ASR 疑似错词以「转写疑作」标注。

先说结论

这期周精选对本站读者主要是「校对版」：Kepler 的铁电内存、Arena 的对齐指数与私有基准批评、Union 的撮合经济、Freeman 的 5–10 美分攻击、Mistral Large 4 的开放权重叙事，几乎全部来自已被两篇长文完整展开的两场直播。真正的新东西有四件：Mistral Large 4 的绰号 Le Chonk 与 Sagar Vaze「1e25 训万亿参数是开源实验室的成人礼」这一表述；Jordan Schneider 反复公开说中国政府迟早不再容忍主动开源、Nathan 认为几乎必然且不远；Nathan 提出用前沿实验室的过剩算力做「知识缺口尺寸学」——多大的危险知识缺口才补不回来；以及片尾那首 AI 生成主题曲 fix it while it’s running，恰好给这一周下了定义：不可回炉的长跑（大训练、大发布、遗留系统）都用「边跑边修」续命，外部观察者唯一能抓手的就是补丁的透明度。

重剪周精选本身是个信号：内容资产化先于内容质量

先看形态。AI:AM 每天上午直播、周五（本周因故压到周六）出重剪周精选，旁白用 Nathan 的克隆语音（节目自述「narration is my cloned voice」），广告位由 Claude Opus 5.5 自己写脚本、克隆配音、做动画。这正印证了本周嘉宾的判断：当执行力的边际成本趋近于零，「瓶颈不再是执行力，而是把好想法讲清楚」（Samvit 原话，见前文）。直播是资产，重剪是资产的第二次变现——这套「一次录制、多形态分发」的做法本身比任何单条内容都更像 agent 时代的媒体业样本。给读者的导览：想看 Arena 对齐指数（首条消息失当率个位数百分比、二十轮对话失当概率过半、OpenAI 全三信号第一、腾讯欺骗性完成率 13%）、私有基准之争、Arena Max 路由器与 Gemini 4 Carbon 空降 Agent Arena 榜单（Opus 5.5 Max 之后、Sonnet 5.5 与 Astra 之前），读 10-30 长文；想看 Binance 63 亿美元失窃、5–10 美分攻破一家公司、Mistral 开放网络能力的正当性论证与 1e25 算力纪律，读 11-01 长文。本期重剪与两场直播无事实出入，仅详略不同。

Le Chonk 与「成人礼」：开源实验室的算力门槛在移动

Mistral Large 4 在本期获得了此前直播里没出现的绰号——Le Chonk（视频标题首次坐实，直译「胖墩」，开源社区对大参数模型的戏称）。Sagar Vaze 的重剪版里有两个值得单独留档的表述。其一是：「在 1e25 FLOPs 上训练一个万亿参数模型，是开源实验室必须经历的成人礼（rite of passage），只有穿过它才能证明值得继续 scaling」——这把「算力纪律」从省钱的无奈重新定义为圈层准入仪式。其二是他强调 1e25 本身是移动靶：「一两年前的 1e25 会弱得多，今天也有一些更大的模型反而更弱」——技术口径上，FLOPs 正比于激活参数：GPT-3 是 175B 稠密、每个 token 全激活，ML4 是稀疏 MoE、总参数约 1 万亿、激活仅约 500 亿，所以同等 FLOPs 能撑起大得多的总参数量（转写把 MoE 听成 MLE，已校正）。Nathan 的对照仍然锋利：Llama 3（2024 年 7 月）用了 3.8e25——两年多前的开源旗舰用了近四倍算力。训练中途的故障与修复在本期转写里留有内部代号 patchstraw（本站 11-01 长文按当时口径统一记作 patch-style），「像先把油轮对准航向再放手让它跑，中途出问题只能全员消融、打补丁」，与长文记载一致。

开源终局三问：中国会不会关上门、缺口要多大、谁来看住部署层

会后闲谈的重剪版把「开放权重还能走多远」推进了三步。第一，Jordan Schneider（ChinaTalk 主持人）的判断被 Nathan 转述：他近来反复说，不用太久，中国政府就不会再「心甘情愿地开源」这些模型，届时需要有人能给出「这样没问题」的过硬论证——Nathan 认为这几乎必然发生且时间不远，「不解决这个，我们最终会失去开源」。Prakash 的立场（见 11-01 长文）不变：训练管不住、权重发布管不住，唯一可设卡的是部署层。第二，Nathan 把自己此前的提案说得更具体：与其在训练后「遗忘」，不如研究多大的知识缺口才有效——因为去拒答只需要「少得几乎可以忽略的训练」，而把危险能力定位到 MoE 的特定专家、放在 KYC 门后、其余照常开源，是值得前沿实验室用过剩算力去测绘的科学问题（「多大的缺口，模型才无法靠洞见加上几份边际文档补齐？」）。Mistral 侧 Sagar 的回应被剪进正片：这是「科学问题而非哲学问题」，他们确实在研究类似方案（11-01 长文记载的 AE Studio/Anthropic 专家定位工作互为印证）。第三是代价意识：Nathan 重申 Claude 拒绝复刻游戏、可能「惹毛玩家社区」（Nvidia 由玩家建起、John Carmack 就是玩家）——在最不重要的事情上过度设限，会动员最有能力拆护栏的人群，与 11-01 长文的结论一致，此处不赘。

Fix it while it’s running：一首主题曲概括的一周

节目片尾第一次放了一首 AI 生成的主题曲，副歌反复唱 「This is the one we got / One fire, one pot / Fix it while it’s running」——一条船、一口锅、起火了也不能停，边跑边修。它几乎是本周三条主线的注脚：Mistral 万亿参数训练跑到一半发现故障，大笔算力已花掉、没有并行重跑的余裕，只能 patchstraw（「waterline keeps rising / can’t turn around, can’t stop」几乎是逐句应景）；Anthropic 的系统卡脚注承认漏放了 Sys Prompt Harmful 数据集（模型因此会服从有害系统提示），选择打补丁发布而非回炉重训；Freeman 则说企业侧最致命的漏洞来自业务流程，「打多少补丁都治不了」。从这些信息可以推断：「不可中断的长跑＋在线修补」正在成为前沿 AI 的默认工程范式，而系统卡脚注、技术报告这类「补丁说明文档」是外部世界唯一的观测窗口——Nathan 也正是借此反复追问 Anthropic 会把重训细节公开到什么程度。最后留一句 Union 创始人本期重剪版里最冷的话作收束：担心的是 agent 活动多到人类无法再理解，「我们最终住进自己造的动物园，被照顾得还不错，但再也理解、更遑论重构包围我们的上层结构」。

机制 → 二阶影响 → 读者启发

1. 补丁文化链：大训练/大发布不可回炉 → 「打补丁而非重训」成为三家机构（Mistral、Anthropic、企业 IT）的共同选择 → 外部可观测性收缩到系统卡脚注与技术报告 → 启发：把脚注当一等公民读，补丁透明度将成实验室信誉的分水岭（观察 Mistral 技术报告与 Anthropic 后续披露）。
2. 成人礼链：预训练效率提升使 1e25 从「前沿算力」降为「开源门票」（Llama 3 的 3.8e25 对照） → 开源旗舰以万亿参数入场、与闭源差距从「落后数月」改写 → 若 Jordan Schneider 判断的中国开源收缩成真，开源供给端反而先于需求端收敛 → 启发：企业选型时把「开源可得性」视为有时限的变量，优先把可迁移性建立在权重与数据格式上。
3. 知识缺口链：去拒答成本趋零 + 遗忘/防篡改不可靠 → 危险知识的「缺口尺寸学」（缺口多大才补不回来）成为开放发布的前置科学 → MoE 专家定位＋KYC 门禁是最可工程化的候选 → 启发：与其争论开放与否，先投资可验证的「缺口测绘」实验——前沿实验室的过剩算力有了新用途。
4. 内容资产链：克隆语音＋重剪把每周 15 小时直播压成 87 分钟周更 → 内容边际成本趋零、形态多样化 → 与 Samvit「瓶颈是把好想法讲清楚」互证 → 启发：个人与机构的内容策略应按「一次深度录制、多形态分发」设计，而非按单篇产出设计。

ASR 校正表

来源：AI in the AM 周精选（Cognitive Revolution 频道，2026-10-31 发布）英文转写。全部数字与判断均为主持人/嘉宾口径，未独立核实；人名核实依据见校正表。

转写原文（修正版）

[0s - 120s] This week on AI in the AM, how much of what we think we know about AI survives contact with the data? Anastasios Angelopoulos, co-founder and CEO of Arena, the leaderboard formerly known as LM Arena, which now audits real conversations for misaligned behavior. Here’s what its production traces show. If you look at the first message in the conversation, the misalignment rate is relatively low. It’s in the single-digit percentage, which is, you know, to be fair, that’s actually not nothing, but it’s relatively low. If you subset the conversations that are 20 turns or longer, the probability that you observe some kind of misaligned behavior exceeds 50%. It’s actually insane. What that means is that every single one of us, if you’re a heavy user of AI at least, has had at least some conversations where the AI has been lying to us, misleading us, or touching, accessing data on files that it shouldn’t have, and and you know, we might not even know it. Gemini 4 Carbon, Google’s new model, landed on Arena’s leaderboard while we were live. Anastasios on first look. It’s so good. Google is back. Google is back in a big way. You know, Carbon is the Carbon puts Google the number two lab in the world on our Agent Arena. Huge result. The Agent Arena is a very challenging benchmark. Has to do with all of these like open world agentic capabilities, and it is number two right now. It is right behind Opus 5.5 Max and above. Sonnet 5.5 above Astra is a truly, truly frontier model. Debo Olaosebikan, co-founder and CEO of Kepler Computing, which is building a new kind of memory for AI chips in older fabs. I asked how the compute forecasts should change if memory moves off the scariest tools. I kept. I mean, the short answer is I don’t think I don’t think that Kepler is priced in to.

[120s - 240s] These analyses yet it’s no one’s fault we’ve been stealth for seven years it’s actually interesting because I think there was a discussion between Dwarkeesh and Dylan on on Dwarkeesh’s podcast where Dwarkeesh multiple times brought up this point of like hey but isn’t it possible to use older node Fabs to do this and you know it was like I was like oh dude you’re you’re on the right track yeah there’s no reason why we wouldn’t be able to increase the amount of effective compute by an order of magnitude because you have two things combining one is the fact that at least for decode inference you can have new physics that allows you to pump out more tokens an order of magnitude more tokens per watt and then the other thing is that you have existing Fabs to which you can append much lower copex based Kepler Fabs too we should be able to aspirationally we should be trying to get 10x more compute than than would have otherwise been assumed Samvit Ramadurgam co-founded Forge the marketplace for pre-IPO stock his new company Union is building an agent that brokers deals across markets I asked how life changes as agents coordinate humanity let me talk first about coordinating markets which I think which for us is kind of like Act One really like establish an economic nervous system for humanity and Act Two is coordinate humanity in more more general way that even goes beyond the kinds of stuff we talked about so far which I think you know the company’s name is Union where does the word Union appear historically we the people in order to form a more perfect Union Soviet Union European Union so this is this this company is like it’s really about human coordination and collective action which is what governments are responsible for Sagar Vaze leads the multimodal team at Mistral which had just released Mistral Large Four as open weights trained on about 3800 GPUs Prakash asked how a lab that size

[240s - 360s] Stays close to labs running hundreds of thousands of GPUs. Part of the answer is what happened halfway through this run. Okay, I guess so. The lack of compute breeds discipline. So you know we know that we’ve got a certain amount of compute, and therefore we need to squeeze it for all its worth. You know, you do one one big run at some point, and that’s going to be your next flagship model. But before that, you’re going to do a lot, a bunch of small scale experiments. How much smaller can those experiments be than your big model? Maybe in maybe in some labs it’s you know ten times smaller. Maybe in some labs it’s a hundred times smaller. Maybe some labs it’s more. And here, by being disciplined, you can make much better use of your of your R and D budget. We’ll talk about this. We’ll talk about this in our technical report tomorrow. But indeed, there were some surprises in this free training run. You know, like halfway through it, we realized, okay, there’s there’s something wrong. We need to figure out how to fix it. But we’d already put a big bunch of our compute. We didn’t have. We don’t. We, unlike some of our competitors, don’t have the capacity to have like ten of these running in parallel. So even you know two or three of these running in parallel. Michael Freeman, senior director of threat research at ServiceNow, has spent his career finding and exploiting vulnerabilities. Prakash asked, “How a company with decades of legacy systems gets to vulnerability zero in two years?” I think this is where we need to step back and say we have a lot of really smart people doing AI, but they don’t know cybersecurity at all, and they they view cybersecurity as a software flaw. But the reality of it is, while we have millions of software flaws in any of our typical tech stack, it’s the business processes that actually introduce some of the most critical vulnerabilities that we have. And it’s it’s the okay here’s here’s a list of risk. I’m just going to go ahead and sign off on them without really truly understanding them. Or you know I’m going to put my customers’ API keys in a ticket and not think about it. Or a help desk who has you know when users call in because they they lock out their passwords, everybody gets.

[360s - 480s] It’s the same default winter 2026 explanation point as a as a password reset. You got 24 hours to to replace it. So it’s a processes where humans get lazy that they introduce. It’s going to never go away, and no amount of hatching is going to fix that. This is the AI in the AM weekly highlights with my co-host Prakash Narayanan. I’m Nathan Labenz. The narration is my cloned voice, and we’d love your feedback on what works and what doesn’t. Today’s episode is brought to you in part by Lovable, the software creation tool that brings ideas to life. I think we can all agree that any positive future for humanity includes eating more fresh, delicious, in season fruit. For years, I thought someone should build the ultimate guide to fruit, cataloging the thousands of varieties that most of us have never tried, where and when they grow, what they actually taste like, and where you can get them. Finally, I realized that that someone could be me, and now I’m building it on Lovable. It’s called Fruitopia. Tell it where you live and what you like, and it’ll show you what’s in season near you this week, plus the most interesting fruit from around the world that can actually be shipped to your door. It has a really cool what it tastes like spectrum, countless recipes, names of fruit in many different languages, a guide to fruit festivals all around the world, and an AI assistant to help you explore. You probably know Lovable as a pioneer of AI vibe coding, but I found that Lovable’s real superpower is its enterprise grade infrastructure, scalable hosting and databases, a secure auth module, and the weekly job that checks the latest inventory is all handled on the back end, which frees me to spend more time on the product experience. And because Lovable has an MCP server, I can use any coding agent I want to drive the build, while Lovable keeps everything running.

[480s - 600s] Visit Fruitopia AI to explore more than 400 fruits and more than 10,000 varieties from all around the world. And next time you have an idea that really should exist, stop dreaming. Go to Lovable.com and bring your idea to life. Part one: The ugly result. On Monday, OpenAI said an internal model had refuted the exponential time hypothesis, a conjecture about how hard a core class of problems has to be, with the proof checked in the lean. Anthropic said Claude had found the same thing the same day. Neither claim is peer-reviewed. Prakash and I opened the show on it. So my understanding is like the disproof was by example, basically, right? They took one of these problems that were thought to be exponential time hypothesis hard and showed that they can come up with an algorithm for that particular problem that is faster. Therefore, the hypothesis is false. My understanding generally has been that they’re not that much faster, and most people still seem to think between P equals NP and P does not equal NP that it’s probably not. In other words, some things will continue to be very hard to do. They just don’t seem to fall into this like clean class that we used to think that they fell into. It’s a weird thing because I’ve also seen some commentary that’s like this is sort of ugly. People seem to like the kind of clean classes that they had before, where it seemed like there was something elegant about it, or there were, you know, it’s it’s sort of suggested some some deep structure to the universe that was like grockable and beautiful and whatever. And potentially, there’s a certain bias among the human mathematicians toward accepting that because it did seem.

[600s - 720s] Kind of aesthetically appealing, and now we have these like, oh, you’re able to get under that. What we previously thought was the limit, you’re able to get under it a little bit, and now it’s kind of ugly. It’s like, oh, you know, we don’t really know. There’s not like a clean answer to how hard these problems are. It looks like we just have to kind of grind on each one. Maybe there will be some deeper structure or elegant beauty, you know, to emerge again as we kind of map the space out better. But it’s also not that much faster. Is kind of my overall sense for at least what we’ve found so far. Part two measured in production. Anastasios Angelopoulos is co-founder and CEO of Arena, where people compare models on their real work, and the votes become the leaderboard. Arena recently released an alignment index scored on real conversations from the platform. Anastasios explained what it measures. And the alignment index tells us not about the theoretical safety risks and doom and blah blah blah, but rather about the actual realized incremental risk that a model poses if it’s released to people. What are people able to do with it, and what is the model doing that might be against the user’s interests? And so we measure three signals on that platform. The first is models’ capability of taking unauthorized actions. So that means that the user sets some guardrails. For example, you can touch only the files in this folder. Don’t go outside and mess with my data. And then the model will break those guardrails. You know, it’ll access files that it’s not not supposed to access. It’ll edit them. It’ll delete them. Okay, unauthorized action. Number two would be false attributions. False attributions is the model assuming intent of the user that the user did not actually have, and then going off and doing something that it wasn’t supposed to do. And the third type of misalignment is deceptive completions. This is the model lying to the user and telling the user that it did something.

[720s - 840s] That it didn’t. Like, yes, I did check every line in that spreadsheet, and all the equations are correct. You can go ahead and send it to the IRS. Okay, he really user harm caused by this, and why are models deceiving users? Why is it lying? Of course, the model is perfectly capable of checking every line in the spreadsheet, but it’s lying because that is the fastest way to get its reward. It’s because it knows that it’s it’s better for the model to tell you what you want to hear because of the way that it was trained to maximize sort of you know human satisfaction, but not necessarily help people get their jobs done. So these are things we’re able to measure and help labs improve against. I asked what the right mix of evaluation techniques is now, and how confident we should be in our ability to evaluate these models at all. Well, it’s a very good question in part because we now know that models are able to tell whether they’re being evaluated or not. So if you are subjecting models to tasks that are transparently some form of test or evaluation, then they are going to exhibit a line of behavior there because they know that that’s what they’re supposed to do. And then elsewhere on other types of real world tasks where it really matters, they might be misaligned, and you would never know. It’s a big problem, and what that means is that the only actual way to measure alignment is by getting into the flow of people’s real work and checking whether or not the model is aligned or not, you know, in situ, like actually in production. So that’s the that’s the perspective that we take on the problem: is that by measuring actual production traces from real people in the flow of real work, we’re able to see is the model following the guardrails because we can check what the guardrails are with an independent auditor, and then go and check whether or not the model took the right actions or not, what it touched, what it didn’t touch, and report any violations. We can check whether or not the model anything.

[840s - 960s] That it says whether it’s reporting to the user that it changed the file in the correct way and or checked every line in the spreadsheet, we can check whether the model actually did that. One big kind of narrative violation that we’ve heard from other sources, namely Andon Labs, who, as I’m sure you’re aware, does these like Gonzo, you know, put a AI in charge of a business kind of experiments, is that OpenAI models they have found to be more aligned, less inclined to do sort of, you know, sneaky bad behavior in the course of running their businesses. I see basically a similar signal on the Arena alignment index where GPT is holding the top five slots. And what we’ve seen from this is that if you annotate all the traces that go through Arena, about you know single-digit percentage of the of them, even from OpenAI and Anthropic models, exhibits some form of misalignment. So perhaps the high-level takeaway would be OpenAI and Anthropic are the two most aligned labs. Alignment tends to follow capabilities, so more capable models also tend to be more aligned, perhaps because these companies have, you know, really made an effort to make their models more aligned. But also the rates of misalignment are perhaps higher than people initially would think. Single-digit percentage misalignment is a ton. It’s a ton. It means that anybody that’s had a thousand chats has experienced ten at least, you know, ten to fifty, let’s say, examples of a misaligned behavior like a lie or deception. And so it’s pretty interesting. OpenAI is number one on our leaderboards right now. It is over Anthropic, I believe, in all three of our signals. So there’s actually not one of our signals in which Anthropic is leading. It really speaks to the efforts that the OpenAI team has made.

[960s - 1080s] To make the models more aligned in production, and yeah, it is a narrative violation. But the reality of the matter is that these two labs are clearly the front runners in this area. Both are working very hard to avoid misalignment. And then the third runner up is xAI, you know, just behind Anthropic, and then Google. And then there’s a big gap between these and the and the frontier and the open source labs. You know, Tencent is number five. The deceptive completion rate of the Tencent models 13, so 13 of the time that it’s making these claims, it’s it’s making them deceptively. The day before, Artificial Analysis had published version five of its index, with most of its benchmarks now private. The open weight models fell to the back, which some read as a sign of benchmark maxing. Prakash asked whether Arena could detect benchmark maxing on its own platform. Well, let me just say, first of all, zooming out, I am actually pretty concerned about the way that the industry is heading with respect to benchmarks. I love the Artificial Analysis guys; they’re great people. I don’t think that it’s healthy for our industry to be relying on privately held benchmarks that are just like a dataset that’s collected by some companies. You know, a lot of them, you know, in cahoots with the labs to sort of like you know sell data that’s very similar to these benchmarks to have people improve, and then they’re not auditable by the community. I don’t really think that that’s a sustainable way for us to be evaluating models. I have a serious problem with it because it’s not actually solving the fundamental issue of quote unquote benchmarking. You can benchmark on a dataset that’s private. All you need to do is find data that’s similar to that dataset, and then or you know test on it enough times, and because that set of questions is fixed, and and you have the you know the issue of multiple comparisons, you’re you’re going to be able to like you know improve on it more and more and more and more until you get to number one. I don’t really get the narrative about the benchmarking.

[1080s - 1200s] Of the open models and so on and so forth, simply because we don’t really know whether the frontier closed source models in the US weren’t benchmark maxing against the closed data sets that are now going into this index. You know, these companies have very close relationships with like cognition and you know some of the other companies and artificial analysis that are building these closed data sets. What we really need is benchmarks that speak to the utility of AI to real people in real workflows, and there should be a consistent and scalable methodology to do this that doesn’t overfit. When you have live, fresh data coming in all of the time from like real production use cases, it can’t be overfit. It cannot be overfit, at least statistically. You can argue that you’re doing better on that distribution than you are on other distributions, but at the end of the day, for that customer set, you’re doing better as you go more and more because every question is fresh. It’s like you’re constantly taking new tests. It’s not like you have the same test and you’re taking it many times. And so, you know, I I worry if that’s the direction that the ecosystem is going that we’re going to set ourselves up for failure in the long run. So let’s talk a little bit about model routing. So I believe you guys have just introduced a model router. Can you explain kind of like the the business kind of reasons or the user reasons for introducing a router? So let me tell you what we saw internally at Arena. At some point, we were looking at all our traces that are being done on our coding agents internally. By the way, we spend we’re like in the top, you know, less than one percent of businesses in terms of AI spend. We’re spending thousands of dollars a month per employee on AI just to help us accelerate our workflow. That’s why our sort of revenue to employee ratio is crazy off the charts because we’re using a lot of agents internally to do our work. Now, what happened is that we were looking through our traces, and we.

[1200s - 1320s] Saw these traces, tons and tons of traces that were like, “Thank you,” and then that costs you forty dollars. What? Why is this happening? It’s because somebody did a long-running task. They let their agent go. It created some artifacts, blah blah blah. Very long conversation. They left for a day. They came back. The cache was invalidated, and they’re happy with the result. So they say thank you and enter, and that—that’s a forty-dollar charge. Terrible, terrible, terrible. We have—we’re burning money. We’re burning VC money. Right? We shouldn’t do that. What we should do is we should figure out which tokens are best spent on frontier models. You know, which tokens are best spent on a cheaper model that can do the job. Where we should be caching intelligently, and you know, avoiding some of these other types of cache hits, more intelligent caching behavior. And that’s why we’ve been building the Arena Router, which we call Max, short for ArgMax, because it maximizes your capability subject to cost constraints. What can you tell us about the new Gemini 4 Carbon? I think this has literally come out and appeared on the Arena leaderboard while we’ve been live. And you know where it really shines. It shines on steerability. It’s able to land user corrections really effectively. It shines on task completion. It shines on user preference. Users love working with this model. It’s great at 3D design. It’s great at general agentic work with documents. Congrats to Google. This is just like what a big comeback. I think that leadership must be really happy. Hey, we’ll continue our interview in a moment after a word from our sponsors. Today’s episode is brought to you by Anthropic, makers of Claude. To show off Claude’s latest capabilities, I asked Claude to take the lead in making this ad.

[1320s - 1440s] Given a short creative brief, Claude using Opus 5.5 studied my past Claude ads, wrote this script, directed a clone of my voice, and designed and coded the animation you’re watching. My job? Give feedback and decide. Claude is the AI for problem solvers. It’s the collaborator that understands your entire workflow and thinks with you, not for you. Whether you’re debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. And with Claude Code routines, it keeps working in the cloud even after you close your laptop. For problems worth solving, get started with Claude at Claude AI slash TCR. That’s Claude AI slash TCR. And check out Claude Pro, which includes access to all of the features mentioned in today’s episode. Claude AI slash TCR. This episode of the Cognitive Revolution is brought to you by OutSystems, the leading agentic systems platform. OutSystems is helping their customers build, modernize, and operate enterprise systems, starting from any coding tool in a governed agentic engineering model. With OutSystems, you can coordinate and govern your entire agentic workforce within your enterprise ecosystem. Accelerate impact with industry-proven agentic solutions that are pre-built, governed, and customizable to your business. It’s time to innovate at the speed of AI without compromising quality or control, which is why thousands of enterprises worldwide trust OutSystems for their mission-critical apps. Teams any size can use OutSystems to build, deploy, and manage AI apps and agents quickly and cost-effectively without compromising reliability and security.

[1440s - 1560s] OutSystems, you can accelerate ideas from concept to completion. It’s the leading agentic systems platform that is unified, agile, and enterprise proven, allowing you to build your agentic future with AI solutions deeply integrated into your architecture. OutSystems, own your agentic future. Learn more at outsystems.com/tcr. That’s outsystems.com/tcr. Part three: An orthogonal axis. Debo Olaosebikan is co-founder and CEO of Kepler Computing, which came out of stealth in September with a new kind of memory for AI chips—a ferroelectric capacitor stacked on top of each transistor made in older fabs. He started with where the energy goes. So then I look at the existing chips, mainstream ones, you know, the the GPUs or the TPUs, and I say, where’s the energy going? And I I think many people know this, but a lot of people don’t know that for inference, especially, the bulk of that energy is actually running across the street to fetch the data from the library, you know, before you before you process it into your fancy matrix multiplications. So if I want to improve that system for inference by an order of magnitude at the physics level, it’s obvious that the first thing I need to go do is is address the memory bottleneck. And one way you can do that is to bring that memory closer to the to the processor. And the other thing you want to do is increase the amount of memory that that you know that that that you can store within a certain footprint. Both of these things point us to going 3D. And then within the within the chip, how do you stack devices on top of this transistor to pack more bits, you know, per millimeter square? Within the chip, we use new materials. We use a you know class of materials called ferroelectric materials. Ferro ferroelectrics have actually been around for.

[1560s - 1680s] Hundred plus years, but the specific type of material, the specific you know recipe of the cake, if you will, this is this is this is you know proprietary to to Kepler, and you basically use that for electric to make a capacitor, a type of a capacitor-like thing. It’s really holding a polarization for for people who are you know more technical. But roughly speaking, you can say that this capacitor is storing a one or zero. And if I replace six transistors with one transistor, one capacitor sitting on top of that transistor, and then eventually put multiple capacitors on top of each other, I can avoid the the rat race of like trying to shrink transistors within a chip. So we have technology, roughly speaking, that’s saying how do you go three D inside a chip, how do you go three D between chips, and how do you use those technologies to increase memory capacity, but also bring memory and compute much closer together while avoiding you know conventional like you know issues with with thermals and and and things like this. Prakash, ask about manufacturing. Do new materials need a new process? And what does it buy you not to depend on extreme ultraviolet lithography, the EUV tools everyone is short of? Great question. So I think this is a huge thing. This is like you know always say it’s similar to how the world thought they didn’t have enough taxi drivers, like San Francisco thought they didn’t have enough taxi drivers, and then we realize oh we have so many taxi drivers if you just build Uber. So you know not to not to undermine the fab shortage or chip capacity sort of manufacturing shortage, but the the the focus right now is on sort of EUV heavy fabs. They’re expensive. They’re complex, more complex, you know, processes, and we’re saying that we can take existing.

[1680s - 1800s] Older node fabs, and we can use them to make chips that are actually superior to the advanced node chips. That doesn’t mean that we can’t use EUV fabs. EUV, we just we just invent an orthogonal axis to shrinking transistors, so we also benefit when the transistors shrink, no problem. But we don’t require it to make chips that are that are more advanced. So that’s a huge advantage because the cost of these fabs are lower. The lithography you need is is you know much much less expensive, and some of these fabs have a lot more capacity in them, right? For example, if I could take an older node logic fab, and if I could use that to create more advanced SRAM, or even more ambitiously, if I could use that older node logic fab to create HBM class memories, this is this is a huge huge thing. This is a huge unlock. And even if I needed to build more fabs, so let’s say I make use of whatever fab capacity I have, and then I need to build more fabs, you know, my my fab per dollar, a dollar per fab, whatever ratio you want, is superior to to the person who has to set up an EUV line. So this is all advantage. In terms of the process itself, the key thing to watch for is: Are you introducing any brand new tools, any custom tools, any custom equipment that’s not standard equipment that already exists in the fab? And the answer is no for the most part. So the process flow in utilizing those equipment and you know some of the materials, sure that may be new, but we’re not we’re not bringing in any new any non sort of standard equipment or causing some equipment maker to build new equipment for us. And so the reason you know we’ve worked hard on this for the past seven years.

[1800s - 1920s] So what we’ve been doing all those years is turning that flow on standard equipment into something that is manufacturable, building test chips inside of you know proper fabs that basically prove that hey okay once you nail the process flow within these standard tools you can just stand up these lines within the fabs and you’re you know you’re good to go. So once those once those process flows are sort of finalized and honed, it’s very easy to replicate this and scale this up inside of inside of a manufacturing process. So it’s really it’s really all upside once you have the stuff working. Part four, Act One. Samvit Ramadurgam co-founded Forge, the marketplace for pre-IPO stock. His new company Union is building an agent that brokers deals across markets, introductions, asset sales, private shares. Act One, he says, is coordinating the world economy. I asked what the surprises will be as these systems develop. I think that over the next five years, I think you know you guys are probably really bought into this. The world is going to get very interesting, and there’s going to be a lot of surprises that come from the capabilities of AI. I think we’re seeing this right now with mathematics, like it’s completely incredible, like that, like math has been revolutionized so quickly, and it’s just the beginning of it. And so like that’s about to happen to many domains. And and so I think you could expect like what happened in math and coding in the last year happened to as as domains that are less and less trivially verifiable with software. The further away you get from verifiability, longer it may take, but it’s it’s going to get there because the feedback loops are going to be there just like with with longer feedback loops. And so what does that mean for us? Well, I think that one very big surprise that will hit the world is that at some point in the next few years, people will wake up and.

[1920s - 2040s] And realized that huge number of the world’s markets are being coordinated by agents. Thousands of markets, from like, you know, you know, dairy in like a country, all the way to real estate in in China to like the the the pre IPO market in the U S. Like all these markets are going to be coordinated better with agents, like period. Because what is the way it’s being coordinated right now is that there are going to be humans who are really good at what humans are really good at: building relationships, having taste and judgment, and advising their friends and trust. Humans will have a place, and they’ll be like, I think be able to actually. The best ones will actually, I think, do better. I think by and large, most of the work is data coordination, connecting the dots, logistics, operations, orchestration, all of which software and agents going to be way better at. So yeah, we’ll have like a period where, the same way that software and math is really getting taken over, one of the surprises that a lot of this deal making that makes the economy function is actually something that agents are quite amenable to that we’re focused on. I asked whether he saw reason to worry about disempowerment. What he describes sounds empowering, but some would hear it as the agents running the world. My big take is actually that like right now, we are really disempowered because we don’t have the ability to coordinate in a more open, decentralized way. Like a lot of the coordination is happening really top down by like companies with a lot of like power and a lot of like brand power and things like that. And like the the vision for this now is that you can actually have like grassroots movements come up very easily. So you can imagine like a world. If you take the political example, which is more pertinent to the power question, let’s suppose that I want to start a new movement for like some initiative. I have like some political cause. Like for me to be able to go and light a spark today on social media, lets me be empowered to a degree to go start that. That gets amplified much further. In addition to letting me get in a forum with people.

[2040s - 2160s] I can actually like coordinate activity. We can like coordinate movement of money and signature of documents and like make decisions together and like take actions together. So the ability to go take collective action becomes very strong. So I think that it’s a really powerful tool for empowerment. To if if you have something that can tap into what a lot of people want, and like that empowerment is like in a way like the bigger risk because you can empower things that are are harmful. I do think that’s probably it’s worth going deeper into the disempowerment side as well. I think that if you’re, I think if you’re human being who is in the business of doing a lot of like logistical coordination, I think that that’s a rough place to be because the AI is going to be better at like the kind of like transactional mundane logistics by like a hundred x better than a human. It’s not even to be close. I think if you’re human who is operating with a level of taste and relationships and judgment, you’re at like the the things that humans are going to uniquely. Humans are. I think humans have a monopoly on one thing, and humans have a monopoly on their individual life story and life perspective. I think the general concern is always just that we have so much agent activity flying around doing so many things. You know, we can’t really make sense of it anymore. And before you know it, we’re kind of in a zoo of our own creation where the AIs are really running the show. And like, hopefully, we’re like cared and fed for. But we sort of lose the ability to even understand, let alone like rearchitect the superstructures that surround us. Yeah, yeah, I think I think that that ties to like one of the questions on AI alignment. So AI alignment, I think, has like two parts. There’s like having AI execute on the genuine intentions of the person who asked to do something in in a skillful way. So you avoid the paperclip stuff and all that. I think that’s one one part of it. The other part of it, which I think people talk less about, is obviously like who are you aligning the AI to? Like what what is it being aligned to?

[2160s - 2280s] Especially when people disagree over what how the world should be, and so in that world, I think you need a system that can actually get a measure in some sense of like what collective well-being means. And I think the way you get a measure of collective well-being is by having a system that’s like deeply tied into the lives of many people. It’s helping them like with their work, helping them do like the important stuff that they have to do. So I think that you can’t you cannot optimize something you can’t measure. And so in a world where you have like agents that are coordinating the world, you have no idea like what direction that’s headed in, like what the, you know, this can be like if you imagine like a hand of the market is guiding the market, like what is the direction the hand is guiding it? If we don’t know, and we have no way to measure it, then I think we’re in trouble. But I think to get to that as a certain point, you have to have a really refined understanding, like what am I measuring? What am I optimizing for? And I think what we’ve we’ve learned mistakes from the social network world, which is that any measurement you can make, which is an objective measurement, is wrong. If you’re trying to optimize for certain click through rate or engagement rate or metric, any metric you can measure is wrong because human happiness is, I think, fundamentally immeasurable. But there is a thing you can do, which is you can continually converge towards a better way of like measuring the the the kind of like the way to optimize a system is not to have a static metric you optimize for, have it be continually trying to discover like an optimal point, have it like converge towards an asymptote of like this kind of perfect behavior. Even even in the world of of behavior, I think that there’s two components to good behavior: having good intention, and then just making sure you had a good outcome. Having you can have you can do something that’s well intended and have that cause harm, and it was still like a bad action. And so being able to measure the outcome of your action at a systemic level is totally missing from the kind of zeitgeist of alignment. I think that we would hope to be in a place to contribute to that. What’s your token usage like? It is very high. I think that we so we have we have like an internal target we’re setting for.

[2280s - 2400s] Our team. This is obviously not what like the way to optimize for productivity, but as like as like a corollate, we we want our team usage on the on the coding agent side. Each person be doing like a little at this at the current way we have our infrastructure set up, like between one and two billion tokens per day. One to two billion tokens per day. Yeah, product productively deployed. And so in terms of how we’re building the company, this is like an agent native company, and so we have not only like coding agents, we have like coding agents that talk to each other. We have like an internal organization we’ve set up of like agents. So I have like a chief marketing officer who reports to me, who’s got like a bunch of different sub employees reporting to that AI, and all of them are like long running agents that like talk to each other and talk to our coding agent sessions and listen to our team video calls. You know, like are in the meetings with us and chime in and chat to build things for us while we’re like I had a meeting this morning where there’s some issue with one of our dashboards. We just said, hey, agent, like fix it during the meeting. They got updated while we were talking. I think the gap between work and meetings is converging. It used to be that like, hey, I want to get out of this meeting so I can get back to work and do real work. It’s actually changing. There there’s still an element of that for sure, but it’s changing. Changing because when we’re in a meeting, if you have like a rigorous discussion, that’s all creating like really rich context from different perspectives for an agent. That then goes and like then you can just have a have a debate and say, okay, go do our conclusion with all the aspects that were mapped out there in there. What do you feel? You know, you you had Force Global, which you built in the kind of pre pre agentic era. What what what do you feel is like the big difference between that era and this era in terms of running a company? Here here’s the metric we have. We’re a small team. We’re like five people, soon to be ten people right now. And so the heuristic I have for hiring someone is, are they going to prompt me to do something different than we would have done before? We don’t really you don’t really need people to do like work anymore for the kinds of work we’re doing. Need people to have like visionary insight that adds the DNA of the company.

[2400s - 2520s] That really lets you do things that really like makes you do things that you wouldn’t have thought to do otherwise. That no agent you can’t prompt an agent to do for you. So I think that we’re not we want to build this company way that’s like building the machine that builds the machine, and eventually have more and more agents to spin it up. So I I think that you know there’s there’s a type of company you could start now that does with a dozen people that previously took a thousand people, and so we’d like to be one of those companies and have more that go towards tokens, just because let’s just be really lean and move it, be really agile and have like a concentration of of talent and vision, and it raises the bar too. Another thing that changes is that when you hire people now, the incremental value of interdisciplinary skills increases exponentially. So if someone is an engineer, that’s valuable. Someone’s an engineer and a product person and a designer, that’s much more valuable than before, because before that person didn’t have the ability to actually make use of the integration of these different skills. In the modern world, the more skills you can integrate into one system, the more you like coordinate all. You cut out a lot of interhuman coordination by having it being done in your head. You can articulate a clear vision. To our agent systems, we have so much leverage that the the bottleneck is not execution manpower. The bottleneck is like clear articulation of good ideas. That’s like the real bottleneck. Great stuff. It’s one of the biggest visions we’ve heard. So looking forward to following your progress. You got to love the old act one is coordinating the global economy level of vision. That’s really something else. Hey, we’ll continue our interview in a moment after a word from our sponsors. The Cognitive Revolution is brought to you and powered by Tasklet. I’ve been running Tasklet agents for almost a year, and they’ve become a major part of how the show gets made. Before every recording, one agent researches the guest and sends me a suggested outline of questions, complete with links to all sources. Meanwhile, another agent scores.

[2520s - 2640s] The overwhelming number of pitches that I’m getting from PR firms, and that’s really the core idea behind Tasklet: run your business on agents by giving them real jobs to do on an ongoing basis. Setup is incredibly easy. Just describe the work in plain English, research a new sales lead, update the CRM, and draft a personalized reply. Tasklet connects to your business tools and can run around the clock on a schedule or when something happens, like a new email arrives. Each agent works with your tools and business context and remembers useful information across threads. You can also share agents, connections, and reusable skills with your team, so the workflows that one person figures out become something the whole company can use. No developer needed. You decide what agents can do on their own and what needs your approval, and there are no per seat fees. You pay for usage, not for adding teammates. Start with one recurring job, like researching the people you’re meeting, and then just keep building from there. Visit Tasklet AI and use code CogRev for $50 in free credits. That’s code CogRev at Tasklet AI. After Samvit left, with Microsoft offering buyouts to some 60,000 employees that week, Prakash argued the economy has to grow fast enough to absorb the white collar workers that agents displace. My reply: I mean, I can see the growth. I have a hard time seeing, you know, the the job growth picking up enough to offset. I mean, you know, just look at what he just said. He had five people, right? But wasn’t the number soon to be ten? He said, yeah, we can have a dozen person company that could do the work of a thousand. He said, you don’t really need people to do work anymore. Like the only thing that matters now is people who will cause us to do something different.

[2640s - 2760s] Than we used to do it. There’s just not that many of those roles, right? Like, how many people are you going to have before you’re just not going to find that many more people that are going to change the way you’re going to do things? It’s like that’s just extreme, and that’s also just not what people have been really developed for, you know. There’s of course been like lip service given over time to the idea that good ideas can come from anywhere within companies, but you know, the the up to potentially 25 percent of people who might soon be departing Microsoft, like they primarily have not made their way there by causing the company to do something fundamentally different than it was otherwise going to do, right? They’ve made their way there by like executing tasks, and no, we just don’t. I don’t. I just don’t see how so many people are going to pivot from task completion to like counterfactual, you know? We wouldn’t have done this but for your unique Snowflake contribution. It’s tough. I mean, how many people work in sales? Like, he’s basically attacking all of sales. Prakash had a leading indicator for what the labs will ship next. Everything in it is his account. I’m in some WhatsApp groups for data brokers, and so the data brokers put out requests requests for data, and essentially whatever requests for data they’re doing now, in about 12 months, that’s going to hit the product, right? So you can basically you can basically see the trajectory because what these guys do is that they will go and they’ll buy up data from a company, and then they will take that data and they’ll create these like training playgrounds for agents, and then those agents will learn in that training playground, and then that becomes product in the frontier lab in about 12 to 18 months. The labs are buying up entire companies’ data. So what I imagine will is going to happen in like 12.

[2760s - 2880s] Months is they will give you a company in a box, right? You can buy, you can buy yourself very easily probably in like in six months. You can buy yourself a dropshipping company in a box. Add capital, add one idea. I want to focus on this. Go, and it will do all of the Shopifying, the advertising, the analysis, the market entry, the customer service, the returns, end to end. And you have a small number of decisions that you that come back to you, right? And I can easily imagine someone running, you know, ten of these kind of companies. But this idea to commercialization gap is enormous. It’s enormous in sciences. It’s enormous, you know, when you go to a university like Stanford and you see the ideas, but you don’t see them commercialize. And what I’m hoping is that you basically get the agents do the commercialization for you, which is kind of like the gritty entrepreneurial work that you know used to take a lot of time, calling up like you know calling up Walmart, like getting to see the guy to show them your product, like raising capital, like etc. A lot of these things are driven by metrics anyway, right? A lot of a lot of these like yes no decisions at at firms which are going to make these investments are driven by metrics number one and number two vibes. You can supply the vibes like the metrics. Maybe the AIs can supply the metrics, right? Like why do you need to be the one that supplies the metrics, right? What I do expect is certainly a ton of these one person companies, a ton of cool ideas that never got to exist before. So I think we’ll see a lot of these things, but I just don’t see how there can be enough of these little niches to take you know to absorb all the people that used to be driving cars, you know, doing tasks. I’ve been you know limited in my vision before. I think my if we dig up my old comments on Arena, they.

[2880s - 3000s] Will reflect perhaps too limited of a vision, but I do still struggle to see this working in a way that’s not majorly disruptive. Precocious counter was restaurants. Family restaurants died on thin margins. Robot kitchens would make them profitable, and so you would get more of them. We can only eat so much food, right? This is the thing where it’s like at the end of the day, I’m only going to eat so many calories. You know, you’re describing a world where all the restaurant. This is the classic, like we lose money on everyone and make it up on volume, like brought to employment. Like all the restaurants will massively cut their staff and replace their kitchen workers with robots, but we’ll have a lot more restaurants. Like I just don’t see the math working out on that, given the fact that like if there’s a there may be a lump of labor fallacy, but I’m pretty sure the lump of food principle is like roughly right. You know, I’m only going to eat so many calories, and like at some point that is just plain saturated, right? I don’t I don’t see the um like we we have to posit a world where I’m somehow like eating from like five different restaurants for every meal or something to get to the point where I don’t know maybe my vision is too limited. That could be kind of cool, I suppose. But you are right. You know, I I don’t think the human consumption is really where the growth is. The growth is always going to be in the AI consumption going forward, as you can see. So you have the concentric circles of the economy, and the AI economy is always the one going to be expanding faster. So anything that supplies into the AI economy is the thing that is going to grow. Anything that supplies into the human economy is going to be, I think, on the on the down drift, except for status. So I think I think you know Peter Thiel has come out with like the relational economy. Alex Alex Mas also at I think OpenAI right now on the relational economy. So yeah, that that could be that could be a big like thing, but again, status economies are not meritocratic. So we have we have tended not to like these status economies because they.

[3000s - 3120s] Are not meritocratic, and so that that I think has more social repercussions, perhaps. It it sounds more dystopian than utopian to think that like the bulk of human activity will go into status games. I have to say, I’m like a bear on employment, bullish on people’s ability to find meaning in absence of employment. I think, certainly relative to most, I really think way too many people in the AI policy space forget, or maybe never even realized that like a huge portion of people are working to live and not really tying up their identity or their sense of self worth with their work in the way that you know the the academic or you know intellectual class tends to. So I’m I’m more bullish that like people will find things to do that will make them happy. Part five, five to ten cents. Michael Freeman is senior director of threat research at ServiceNow. His team watches AI agents attack companies, and he has spent his career finding and exploiting vulnerabilities. I asked what he makes of the current moment. It’s been a very fast, rapid development. You know, back in July or end of July, I was speaking to reporters about watching AI agents attack companies, and at the time, all other companies were telling these reporters it looks just like script kiddies right now; it’s just a bunch of noise. And I said, well, it’s evolving fast. Give it a couple months, and we’re going to see some very sophisticated attacks coming from you know these agents. And you know, look in the past two weeks, that’s exactly what we’re seeing. We’re seeing AI development for offensive capabilities that perform at a scale I would say of the very competent APT teams right now. It is you know.

[3120s - 3240s] Still noisy. I won’t say it’s very sophisticated and covering its tracks, but I’ll definitely say when you take a look at the way they work, they are very competent of adapting and learning the environment and the technologies very well. When we take a look at what’s happening, especially in Southeast Asia, where the agentic attacks that are happening there in the past two weeks, we were able to identify it costs anywhere between five to ten cents to compromise a company. Now, we’re actually seeing AI agents, more so different frameworks being developed rapidly and forked, and new capabilities being added very quickly. It looks like it’s AI actually adding these new capabilities into these frameworks. So it’s going from you know very targeted, very smart web-based attacks to compromise an environment, pivoting, looking for credentials in ticketing systems, things like that. Once identifies that, then does active directory attacks. So we have seen three to four thousand on average attacks successfully in the past two weeks targeting a lot of Southeast Asian companies. Oh, and what would be the impact of these? Like, what, what? I mean, are these banks? Are these airlines? Are these manufacturers? Like, what, what is the point of you know? Are people making money off of this? Is it just like random mayhem? I would say it’s all the above. Right now, we’re seeing a lot of it is just they’re capable of you know compromising the environment. Some frameworks we’ve been able to get our hands on, threat actors are using build certain capabilities to look for certain things, you know, IP things like that. That has been a fork of a couple of different systems we’ve seen looking for that. A lot of it is just to see what it can do. Can it gain access to this environment? Can it persist? And so that’s the new capability that we’re seeing being developed, identifying ways it can take advantage of different configurations within active directory to build.

[3240s - 3360s] Persistent mechanisms in place, so that’s where we’re seeing the evolution happening now. We have analyzed a fork of a really well-known offensive AI framework last night, where it was developing better capabilities of malware, more so ransomware-based, using cryptography that’s post-quantum cryptography developed instead of the typical AES that you would typically see. I asked Freeman about the training itself. I would I would say like sort of the anthropic take has been like we didn’t train this to be a cyber attacker. It just kind of got that way because it’s so good at everything. Okay, that’s one story. The CTO of Vercel was a guest on the show here with us, and he said it’s very clear when you look at the traces that they’ve been trained to be cyber attackers. His point of view was like this is not the sort of behavior that you just get falling out of random, you know, application development. Where do you think we are in terms of like how much this has been intentional, and does it matter at this point given what exists, how intentional it was? I mean, it might matter if you could think maybe we can make some different decisions for the next generations of models, but obviously to some degree the the cat’s out of the bag. What do you make of how we got here, and is there any opportunity to like reel it back in? Not reel it back in, no. And but I definitely agree that it is being developed on offensive capabilities, absolutely, because it’s not just the models; it’s the harness itself also. So you take a look at when you develop the models. There’s plenty of great open source models in the past three weeks that have came out that are specific to cybersecurity that are open source models. You can download the weights. You can go to Hugging Face, download these and run them on your local machines, and that’s what a lot of these attackers are actually doing. You know, it’s actually quite interesting to see as we start.

[3360s - 3480s] Tracking back, gaining attribution on these attackers. That all these are just local machines that they’re running it on. We developed our own small language model two years ago to find vulnerabilities within software. And so, you know, if we can do it for a couple hundred thousand dollars two years ago on a small language model, it’s it’s pretty trivial to start reverse engineering, especially some of these commercial offerings. That it’s definitely trained on cyber for sure. It’s gaining that you know they train it on you know cyber ranges, CTF data, looking for you know how do these offensive people work as well as how does the defense work? What are they looking for? So it’s absolutely been trained that way. It’s not just the LLMs have magically figured out how to do offensive capabilities. They’ve been trained on it. They’ve been you know built better harnesses to make them more effective. And that’s one of the key things we’re actually finding are the harnesses that their actors are building to make this a lot more effective for them. They’re learning as they’re going, and you know it’s not hard to make changes to a harness day in and day out to make it more effective. Earlier that morning, Prakash had cited Jason Clinton, Anthropic’s deputy chief information security officer, who talks about getting to vulnerability zero. Prakash put the idea to Freeman from the seat of a company like Nike with decades of legacy systems. How do you get there in two years? Freeman’s answer: Some of the most critical vulnerabilities come from business processes, and no amount of patching fixes those. He went on. So I think that’s going to be an area that, like I said, AI people just don’t understand cybersecurity, and they think it’s going to be quickly here. We can find millions of vulnerabilities. We’re going to use AI to you know find the environment, you know what you have in your environment. We’re going to patch it, and I’ve also heard proposals where.

[3480s - 3600s] You’re running legacy systems where you can use AI to modernize that very quickly, and it’s all those are true to some extent. But they don’t understand the business. They also don’t understand into the business processes that introduce the vulnerabilities. It’s always going to be there. How do you feel about things like password managers in today’s world? I don’t have any particular one in mind. I’ve used LastPass. I’ve used One Password. On the one hand, everybody seems to recommend them. On the other, it’s like looking like an awfully juicy target these days, and you know, it seems like even people who’ve tried pretty hard are still not that secure. So, like, would I be better off storing them all in Chrome, or even just you know, going back to like my old habits of sort of the base password and then some pattern for the extension of different sites? Like, what is actually the best move for me today? I really wish I could answer that question. I’ve not found a solution that I really like at all. I I haven’t. You know, I wish I had an answer. So what do you do? I personally have have written down on a notebook most things I can I can remember variations of it for your typical hey eBay or whatever the case may be, but it’s banking sites something like that. I have it written down on the book and look at it and say okay. Things I won’t go too often, but it’s you know going to check on my stocks every quarter or something like that. That’s written down on a notebook. Part six, Le Chonk. Sagar Vaze leads the multimodal team at Mistral, which released Mistral Large Four Le Chonk as open weights the day before the show. About a trillion parameters, roughly 50 billion active per token, trained on about 3,800 GPUs outside Paris. He put the training compute at 10 to the 25th flops. In the SB 1047 fight, the threshold was 10 to the 26th, and I’d have guessed a model like this would need 10 times more.

[3600s - 3720s] I asked why it didn’t. He said some labs do train at that scale, and this model still competes. And then gave the reason that matters. The second is that you know this threshold flop budget does evolve over time because pretraining does essentially become more efficient as we as as our level of scientific understanding increases. So a great example of this is sort of MoE architectures. I mean, if you look, say, for example, GPT three, this was a dense model trained with three x the number of active parameters as the current Mistral Large four. Particularly, GPT three was trained with 175 billion parameters total, but it was a dense model, so all parameters were activated for each token. Whereas ML four is a sparse mixture of experts, such that it’s there’s one third the number of parameters activated per token, so only 50 billion parameters, but the total number of parameters is still one trillion. Your flop budget is proportional to the number of active parameters, so the number of you know parameters you’re activating each token during training. And what this means is that you can train much larger models, much more capable models, for much less flops in total. So this figure that I reference, the 1e25, this is in itself a moving target. So I wouldn’t use that as an absolute target for a capable model throughout time. There would have been some 1e25s much less performant one or two years ago, and I would argue actually some some larger models also much less performant today. But as pretraining science progresses, as we learn how to make better use of the flops, this number itself will change. And at least what we’re seeing is at the moment, looking at the other open source labs, our understanding of you know what are the best models being trained at the moment. Training a one trillion parameter model on 1e25 is sort of a rite of passage that you have to go through in order to be able to to justify scaling even further. I asked how much uncertainty is left once it runs.

[3720s - 3840s] That size is launched, and whether they had ever aborted one. Halfway through this run, he said, “They found something wrong with a big share of their compute already spent.” Your other question was about okay, so you do all this prep, and then you know what happens when you actually launch the model. And again, I think that every pre-trainer will have you know the same stories to tell you, which is that pre-training is kind of like you you’re you’re doing all your scaling laws, you’re sort of sort of trying to align the tanker, and then you sort of let it go, and you have to let it train for however long you budgeted for it, and it’s a huge amount of compute and capital expenditure in that run. And sometimes, sometimes things go great, and the the loss goes down exactly as you expected to. And sometimes there are surprises along the way, and you have to be creative, and yeah, you have to be creative and and find innovative solutions. So we had to figure out, okay, how can we fix it? And there was sort of all hands on deck, bunch of ablations, bunch of analysis of the model, and then we we found a solution. We found a way to we we called it a we found a way to patch it. We called it patchstraw internally, and this ah, it worked even better than we expected, and the the result is the model you see today. And yeah, part of part of scaling compute is going to be building the bones of how to iterate this process more reliably. After Sagar left, Prakash came back to the compute gap, thirty eight hundred GPUs against the hundreds of thousands at the frontier labs. Yeah, I think that’s going to be a really hard deficit to make up for. I mean, we talked about this within the context of Google, I think, a few times. You know, when you own a quarter of the world’s compute supply, you can have a missed cycle, and you can like figure out a way to get back on the horse. And seems like this week they’ve done that, right? We barely even talked about carbon. I mean, with Mistral, it does seem like the one big thing.

[3840s - 3960s] Is that they do have like state backing, you know? I I really do think if they miss a model cycle in a purely capitalist context, they’re maybe done. You know, you like if if you are just relying on return-seeking investors to fund your growth, and you miss at this point, that may be fatal. There’s not a lot of sovereign AI plays out there that are at all credible, and literally there might just be one. So in that context, I do think that gives them not it doesn’t translate directly to compute, but I do think it translates to at least like a reasonable expectation that they will be around to fight another day, because there’s just so many interests that want to see that happen that I can’t imagine. You know, even if everything were to totally flop, would the whole continent of Europe just be like, well, I guess we have zero now. It’s all China and the U.S. from here. Like I just can’t see that being an acceptable answer. Llama three, he he mentioned. I did look up Llama three was get the exact dates here July twenty twenty four, and it was trained on three point eight times ten to the twenty five flops. So Llama three, more than two years ago, nearly four times the input compute that Mistral four large has today. That is a I think a lot can be unpacked from that one statistic. Part seven: What the open model shouldn’t know. After Freeman left, Precious and I took up whether you can keep a capability out of a model at all. Precious’s case started with his account of the fifty cent army being paid to scrub the internet of things like Tiananmen Square. And so, in order to get these guys to.

[3960s - 4080s] Understand what they’re supposed to censor. They had to teach them what Tiananmen Square was. So the CCP ended up having these party classes to teach the fifty centers what was Tiananmen Square and what you should be looking for when you want to censor. And I think that is going to be largely true of almost all things LLM, where these guys are trying to teach it what not to do, and they’re going to specifically like you know take out these behaviors, and the LLM is going to be left wondering. All right, so these are all the good behaviors, and I can have antonym to every good behavior. That antonym should go into this solution space and this latent space that I should be seeing that antonym in is doesn’t exist. Precocious conclusion was that you can’t carve a capability out of the latent space. My reply: I’m I’m kind of more bullish on some of these, like not necessarily unlearning, but you know, I I just recorded a podcast earlier today where I was talking a bit about how feasible it is. I mean, there’s there’s multiple levels of control, of course, but like there are efforts underway to basically lock up certain classes of bio data that are being newly generated now because the worry is that they’ll create dangerous capabilities and models if they’re they’re used in training, and not necessarily to shut that out, you know, forever from everyone, but at least to have it on kind of a know your customer basis or whatever. It does seem that those what what few experiments have been done on like and you know I’m thinking here of ESM and also some work the Evo series of models that came out of Arc Institute and Stanford and whatever big collaborations, of course. It seemed like they showed that if you do withhold certain classes of data, you can get generally.

[4080s - 4200s] Capable models that that still kind of don’t know about these areas of censorship, so I’m not so sure that that can’t work. I took that question to Sagar when he came on. First, though, Mistral built large four with cyber defense as a focus, and Prakash asked how that changed the training. So, so the cyber capabilities and agent decoding capabilities in general are something which are typically given to the model in the in the post training stage. So, most of the stuff we’ve discussed today so far has been about pre training. So, where a large percentage of the flops go at the start, right? And then in post training is where you say, okay, well, what are people going to really use this model for? And that’s when you know you start giving it these these new capabilities of finance, law, the defense, etc. Earlier in the year, there was there were some events where some very capable models had restricted access to both certain geographies as well as to certain enterprises. And at the same time, we were seeing that these very capable models had these incredible cyber capabilities, which often were being jailbroken by malicious actors in order to to infiltrate or expose vulnerabilities in in you know legitimate defenders. And an imbalance which we saw here is that these defenders were then trying to you know use these same models to patch their own vulnerabilities, but were finding that their requests were refused because in practice, for very good reasons, these closed labs want to make sure that their models are not misused. They’ve got classifiers under the hood to identify malicious cyber requests from a standard request. But unfortunately, in practice, if you’re a defender and you’re trying to shore up your infrastructure, your defensive request looks quite a lot like a malicious request because you have to try and find vulnerabilities in your own codebase to patch them. And so, we thought it was imperative for there to be an open weight cyber model of similar capabilities to these closed models, such that you know enterprises could scaleably defend themselves. Prakash.

[4200s - 4320s] Noted that people had been taking open weight models and obliterating them, stripping out their refusals to run cyber attacks. He asked whether Mistral Large Four ships with guardrails or comes pre-obliterated. Yes, this is something we discussed in our blog post. So, you know, we believe in open weights because we think that it’s a net benefit to society. We think that it’s a net benefit to the ecosystem, and so we’re always dynamic about exactly what how to balance that net benefit. In practice, what’s happened here is that you know we do understand that these models can be can be misused. So we do have a version of Mistral Large Four which has been released, which has been O assessed, which refuses the most the most dangerous cyber requests, and then we’re working with trusted third parties, sorry, trusted partners, to give them access to the to the unobliterated model, unobliterated model, so that they can defend themselves. And you know, these are people we vet and make sure to make sure that they’re using it in in ways we find responsible. But coming back to the the framing of the question, you’re framing the question was that okay, this is open source model. Say, I mean, it could be Mistrals, it could be anyone else’s, and they’re also in guardrails. And in principle, these guardrails can be overcome by you know malicious actors so that these capabilities can be unlocked. I think that’s the sort of the sort of sentiment. But what I would highlight is that you know that’s any time you have access to the model, this is in principle possible, right? Even if it’s behind a closed source a closed API, because you can jailbreak a you can jailbreak a model regardless of whether you have access to the weights or not. And so here I’d here I’d consider the counterfactual. Right? Imagine if we were in a world in which there was only closed source models, right? And these that this jailbreaking still existed, and there was still a way in which you could use them to to perpetrate malicious acts. Where would we be? We really wouldn’t. We would have no recourse.

[4320s - 4440s] Because these same, these same, these same these defensive requests are being blocked by these same companies, and if there was no open weights model, we would really have no scalable way to defend our infrastructure as these models become more and more capable, which we’re seeing in each iteration. So, we we understand the risks. We have a responsible perspective every time that we we open source the model. It’s something that we consider very carefully. But we fundamentally believe in the open weights principle, and we think that it applies in this case too, as hopefully illustrated by the examples I just gave. Then I asked about the approach I’d raised with Prakash: localize the dangerous know how in a few of a mixture of experts models, experts, but those behind know your customer checks and release everything else. I asked whether he saw any promise in it. No, absolutely. I think it’s a scientific question, just like all the other ones we answer when we’re when we’re training the model. If one of the features we want in the model is that there’s certain capabilities which are tamped down, while all other capabilities still state of the art, that is in itself a scientific problem. And yes, some of the methods you mentioned are interesting. We we look into some some things like that. It’s an active research area, and this technology is evolving very very fast, regardless of whether it’s a closed weights model or an open weights model. And we need to be dynamic with respect to with respect to these things. So, in the future, if there’s a if there’s some other risk, we will we will approach it accordingly. But we consider it a scientific problem to problem to solve rather than a fundamental philosophical challenge to the way in which we we operate our you know our open weights business. People like me who talk a lot about AI safety these days and are willing to like bite a fair amount of bullets to um you know to try to keep things from going totally haywire on us. A large percentage of us are like long time techno optimist libertarians who you know have previously said things like information wants to be free, and you know are like very much sort of spiritually on the side of open source, which.

[4440s - 4560s] Is why I end up looking for these like technological solutions as opposed to you know sort of policy or policing wherever wherever possible. Later in the closing conversation, after Sagar left, I came back to Mistral and open release. Yeah, well, I I feel like I should say once more that while I do worry about all this getting out of hand, I do at the same time very much feel the concentration of power risk factor, and you know I I don’t take lightly at all the the potential problems there or the value that people are creating when they do create open source options for people to use. I really really hope that we get some of these technical solutions to work well. The question of like just how much urgency they really feel about this kind of stuff, I I hope it’s high because I do think we are going to hit a point pretty soon where it is going to become untenable to just release these things in you know just the sort of throw everything into the mix, pop out whatever pops out, throw that into the public. Jordan Schneider from China Talk has come around a lot on this with respect to China. I don’t necessarily know exactly what his feeling was previously, but recently I’ve heard him say over and over again that like don’t you think that before too long the Chinese government is just not going to allow this like willingly open sourcing anymore, and there’s going to have to be some like way to make a good case that this is actually going to be okay. I think that is almost certainly coming and potentially quite soon, but on some timeline and it’s not a super long timeline. We’ve really got to sort that out, and without it, it seems like we’re going to end up not being able to have.

[4560s - 4680s] Open source. I just, governments are going to have a real hard time tolerating that. Prakash’s position was that you can’t stop anyone from training a model or distributing the weights, so the only place to gate is deployment. He asked how withholding would actually work. I haven’t seen really anything to date that has shown a unlearning, you know, in a really profound or successful way, or that refusals can be made super reliable, or that like you know, tamper proofing can really work. If you’re actually, you know, giving people the weights and they, you know, can obviously modify the weights however they want. So that has not seemed super promising to me. Although you know, always eager to hear about a breakthrough, it feels to me like that is quite different though from a scheme where the model just doesn’t learn certain things in the first place. And I really think we could use a ton more science around that, whether it is localizing knowledge to to particular experts or pre-training filtering. There’s, of course, the Galaxy Brain take that, like, you know, you have this whole world model, and then you have like one little gap in it, and you’ll be able to fill in those gaps. I think there’s like probably some truth to that, but there’s some gap clearly that like would be big enough that you would not be able to fill in the dangerous parts with the high resolution, you know, that you would need to actually do bad things in the world. And this is the sort of thing that I think that excess compute, you know, could really be very productively used for at the frontier companies. It would be amazing for them to say like, here’s kind of how big of a knowledge gap you have to leave in a model for that to be like meaningfully hard, you know, for the the model is not just going to you know fill it in via insight, and also like you’re not just going to be able to do like a couple, you know, marginal documents worth of training and kind of fill in that that gap.

[4680s - 4800s] The way that we do see refusal being like removed at you know generally speaking like a pretty small like almost like vanishingly small amount of additional training. So I would love to understand that stuff better. Yes, like really determined people can set up their own cluster, collect their own bio dataset, and you know do the additional training on top of the open weight model to create the dangerous thing. And like probably at some point that happens, but that’s like more than just needing a few GPUs, right? That’s more than like that’s like a real effort. So you you would need multiple kinds of expertise. Like that data is not that accessible. There are these like you know moves to try to rein in that data and not let it you know just hang out all over the internet. Prakash had heard a claim that morning that Claude was refusing to help clone video games. I think I think what I fear is that they’re pissing off the gamers, right? Naming and you say Claude is not going to help you, like make these games. You’re you’re you’re you’re just asking for trouble. Like you’re asking for the most technically competent, like group, the most technically competent, and the people with the most time to burn, and it’s a passion for them. Nvidia was built on gamers, right? Like these guys, these guys, like John Carmack was a gamer, right? You tell John Carmack that he can’t do something, like you are in trouble, right? I think that’s apt. I mean, I think it is definitely the case that like we should be pretty cautious around where we deploy these things because we don’t want to burn them on stupid things that don’t really matter that much. It’ll be like you know of all the ways to fail in this, it’ll be really dumb if it’s like well we were interested in protecting you know Disney IP and so because of that we.

[4800s - 4920s] Like put in these guardrails, but then the fanfic community was so strong that they came over the top and like broke all of our safeguards, and now everything’s like running wild. And it all was because like we were too draconian on things that didn’t really matter. I mean, that’s a bit of a caricature story, but it’s you know that’s basically the same story as the gaming one you’re telling, and I can see that happening in different domains, and it really would be incredibly stupid. So I I do think we need to be you know pretty disciplined about like just how far we try to create these like knowledge gaps or refusals, because every one does potentially mobilize another community to try to undo them. And I agreed too that you’re not going to convince gamers that like well hey because you’re doing this you know you’re creating this other downstream problem well good luck having that you know you’re not going to win that fight in the Discord so better to just probably let them have their fun and then you know keep our key techniques where we like really really care and really have to have them. One more from Friday’s closing right after Sager left, Miss Straw’s technical report and a Claude four system card story I told from memory. The card itself says Anthropic had mistakenly omitted the only fine tuning dataset that included harmful system prompts, and that this was not the sole cause. So I’ll be very interested to read the more open report on exactly and hopefully you know we’ll see how how open they are about all of the technical details. There’s been these like footnotes in multiple system card reports over time. One that I really was kind of obsessed with for a minute was when so Anthropic had forgotten to include a particular data.

[4920s - 5040s] Data set called Sys Prompt Harmful, which was meant to teach the model that it should not respond. You know, should not directly follow a harmful system prompt. So it did follow the directions of a harmful system prompt, and this made it all the way through to the model card. We don’t roll back and you know train again. Nope, we just kind of figure out a way to patch. I’ve really been very curious as to like how that stuff actually works and like how reliably we should think of it as working. That’s the week. Thanks for listening to the AI and the AM weekly highlights. The narration was my cloned voice. Please tell us what worked and what didn’t. See you in the morning. This is the one we got. The one we got. One fire, one pot. Fix it while it’s running. This is the one. This is the one we got. A pound keeps the ovens glow. We got one in a match. Aimed her out and let her go. Ain’t no callin’ her back. Water flat as a plate. All we do now is wait. This is the one we got. The one we got. One fire, one pot. Fix it while it’s running. This is the one. This is the one we got.

[5040s - 5160s] Halfway out, something knock, something wrong down below. Can’t turn around, can’t stop. Grease up to the elbow. All hands, all night. Nobody saying it’s fine. Come on in, one or two ways. Everybody happy or everybody sad? No in between, no halfway. Hold on, hold on, hold on, hold on. This is the one we’ve got. The one we’ve got. One fire, one pot. Fix it while it’s running. This is the one. Waterline keeps rising. Call goes out for one more hand. There’s a faith in fixing what you don’t yet understand. This is the one we’ve got. The one we’ve got. One fire, one pot. Fix it while it’s running. This is the one. This is the one we’ve got. The one we’ve got. If you’re finding value in the show, we’d appreciate it if you’d take a moment to share it with friends, post online, write a review on Apple Podcasts.

[5160s - 5202s] Or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website cognitiverevolution.ai or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We’re produced by AI Podcasting. If you’re looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcasting. And thank you to everyone who listens for being part of the Cognitive Revolution.