63 亿美元失窃与 5 美分攻击：当闭源模型被越狱、防御者被拒之门外，开源成了防线

先说结论

这是 Cognitive Revolution「AI in the AM」2026 年 10 月 30 日（周五）直播的嘉宾切片（主持人 Nathan Labenz 与常驻搭档 Prakash Narayanan；完整一期还包含 Arena、Kepler Computing、Union 等五段访谈，本切片收录开场新闻讨论、ServiceNow 威胁研究高级总监 Michael Freeman 与 Mistral 多模态负责人 Sagar Vaze 两段访谈及会后闲谈）。三条主线互相咬合：其一，史上最大的加密货币失窃案——Binance 被盗 63 亿美元——技术上并不高明（老式钓鱼攻陷一台工程师笔记本），真正的护城河是朝鲜这类「有核武器的法外空间」；其二，Freeman 的团队遥测显示 AI 攻击正在工业化：攻破一家公司的边际成本降到 5–10 美分，两周内东南亚发生 3–4 千次成功入侵，且进化主力是攻击框架的 harness 而非模型本身；其三，Mistral 把这个乱局变成了产品叙事——闭源模型被攻击者越狱用于进攻，防御者用同一批模型加固自家代码时却被安全分类器拒绝，于是「开放权重的网络防御模型」成为 Large 4 的核心卖点。对读者最有迁移价值的一条判断来自 Freeman：「关键从来不是模型，而是 harness」——这句话对攻防两侧同时成立。

63 亿美元是怎么离开 Binance 的：不高级的入侵，法外的空间

开场（120–600s）由 Prakash 复盘事件：一名 Binance 工程师的笔记本被攻陷——按节目转述，疑似老式钓鱼而非零日漏洞——随后几分钟内资产被转出。攻击者做了两件「聪明事」：转移以太坊时在钱包里留下足够余额，让 Binance 日常交易继续成交、迟迟察觉不到；不直接提现，而是把赃款质押给 DeFi 交易所借出其他资产。等到 Binance 请中心化托管方 Tether 冻结目标地址时，冻住的抵押品在法律上已属于那家「其实也被抢了」的 DeFi 交易所——追偿成了一个开放问题。CZ（Binance CEO）宣称资金安全，节目称 Binance 资产规模 1100–1200 亿美元，市场并不担心偿付；戏剧性的是，因为马斯克在推特上的言论，CZ 顺手清仓了持有的约 10 亿美元 SpaceX 股票，SpaceX 当日跌 4%。同期 Polymarket 也遭入侵：攻击者拦截前后端之间的通信、吸走密钥转账，Polymarket 承诺全额赔付，节目判断其严重性远低于「国家级水准」的 Binance 案。

两位主持人把事件拆成三层。第一层是人与密钥的架构问题（360–600s）：Prakash 曾考虑自建交易所，最让他不安的是交易所 CEO 们直接掌管数十亿上百亿美元的冷热钱包——「你无法想象高盛 CEO 手里攥着能随时抽走千亿美元的钥匙」，而 CEO 又必须授权他人分担，密钥-holder 年薪 15–25 万美元、坐在亚洲，随时可能被 50 万美元的「新交易所面试」诱饵骗去下载带后门的面试软件（840–960s）。Nathan 补刀：现在有了能替你打开浏览器点来点去的 computer use agent，公司内部那些五年间没轮换过的旧密钥、Slack 与工单里的密码，全部变成可被自动化收割的资产。第二层是国家级免责空间（600–840s）：Prakash 的判断是「疏忽与国家级之间的距离没那么远」——国家级的部分不是技术，而是作案者身处核国家、长臂管辖够不着：在中国黑客会被枪毙，在俄罗斯 KGB（转写作 KGV）会来分赃，只有朝鲜这十几年的法外空间能持续容留此类行动。第三层最反直觉：币价几乎没跌（1320–1560s）。Prakash 的解释：投机资金已迁往 AI 与体育博彩，留下的是信仰型持有者；更关键的是赃款根本变不成美元——过去十年监管把所有加密-美元接触点都改造成 KYC 门禁，超过十万美元就要解释资金来源、上百万美元银行可能直接拒收，所以大户们的日常其实是抵押比特币以太坊借美元（3–4 倍超额抵押，放贷方是华尔街银行与 Galaxy/Gemini/Coinbase，借款人对利率不敏感，因为抛售本身会砸盘）。他的收束判断是：加密与银行体系终将收敛，现在争的只是「要不要可逆性」。

攻击的工业化：5–10 美分攻破一家公司

Freeman 的履历（1800–1920s，主持人介绍）横跨 25 年密码学、漏洞研究、逆向与利用开发，包括为情报机构做攻防；此前联合创办的 AI 攻击预警公司被 Armis 收购（2024 年 2 月），ServiceNow 又于 2026 年 4 月完成对 Armis 的收购——所以他现在看世界的位置，是企业工作流平台叠加资产可见性遥测。他的核心情报是未发表的团队遥测（2040–2280s）：七月他还对记者说「别家都说 AI 攻击像脚本小子，但我判断几个月内会质变」——过去两周应验：AI 进攻能力的规模已达到「非常胜任的 APT 团队」级别，仍然吵闹不隐蔽，但攻破一家公司的成本已降到 5–10 美分；东南亚两周内平均 3–4 千次成功入侵。进化最快的不是模型而是框架生态：攻击框架被快速 fork、新能力以「像是 AI 自己在给框架加功能」的速度累积——从定向 Web 攻击到攻陷后转进工单系统找凭据、再打 Active Directory 建立持久化；节目头一晚他们刚分析完一个知名攻击框架的 fork，其勒索软件模块改用了后量子密码学替代常规 AES。他特别指出：过去三周出现了一批可从 Hugging Face（转写曾误作 Pirate Face）下载权重的网络安全专用开源模型，归因回溯显示攻击者就在本地机器上跑——「我们两年前花几十万美元自训小模型找漏洞都做得到，逆向商用 offering 是平凡事」；这些模型确实在赛博靶场、CTF 数据、攻防两侧知识上被刻意训练过——不是 LLM 顺带变强的。这直接回应了 Nathan 引述的两极说法（2280–2400s）：Anthropic 的口径是「我们没训练它攻击，它只是什么都强」，而此前上节目的 Vercel CTO 看 trace 后断言「明显被训练成攻击者」。Freeman 站后者：「绝对是刻意训练的，收不回来了。」

防御侧他给了三个可迁移的观察（2520–3000s）。其一，军备竞赛的胜负手是「谁更懂你自己的环境」：防御方正用 AI 补上历史上安全团队从不理解业务上下文的短板——ServiceNow+Armis 的组合能发现「CrowdStrike 买了 1.2 万个许可只装了 8 千台、其实有 1.8 万台该装」，能看见「某第三方公司的工单系统里存着客户生产 API key（含 AWS S3 与 AI 工具集）」这类流程性漏洞——而攻击者的 AI 机器人正在自动翻 wiki、SharePoint、工单系统找密码。其二，「vulnerability zero」会来但没那么快（2880–3000s、3360–3480s）：AI 能找出几万上百万个漏洞，但 Fortune 500 光打补丁就要数年；最致命的漏洞来自业务流程（帮台统一发默认密码 Winter2026!、风险清单看都不看就签字），打补丁永远治不了。「很多做 AI 的人根本不懂安全，把网络安全当成软件缺陷问题。」其三，供应商整合创造收购牛市（3480–3720s）：企业客户宣言「只跟 3–4 家供应商买」，供应商为填空白去收购创业公司——初创公司本质是「功能集」，银行也会直接买网络安全公司，因为比签多年合同便宜。对个人，这位顶级安全专家的答案朴素得惊人（3720–3840s）：密码管理器他没找到满意的方案，银行类密码记在纸质笔记本上，最关键的操作用物理 USB key——那是他在智库为 DARPA 特种作战场景做的原型系统。对 OpenAI 式「代理蜂群持续防御」的推销，他泼冷水：拆开看 AI 只在高层决策环节，OT 关键基础设施的自动补丁他「目前不会信任」，实验室里与关键基建伙伴的测试虽成功，但「成功率必须 100%，还没到」。

防御者的悖论：开源网络能力的正当性时刻

Sagar Vaze（牛津 VGG 博士，师从 Zisserman 与 Vedaldi，Pixtral 与 Magistral 作者，2024 年加入 Mistral 现任多模态负责人）讲解 Large 4 为何「带着网络能力开放权重」发布（5400–5880s）。Mistral 年初观察到一个不对称：能力最强的闭源模型对特定地区与企业限制了访问，同时这些模型被恶意行动者越狱去入侵防御方；防御方想用同一批模型给自己代码找漏洞时，请求却被拒——因为安全分类器无法区分「给我自己的代码库找漏洞以加固」与恶意请求，防御请求长得太像攻击请求了。于是出现荒诞局面：攻击者用闭源模型畅行无阻，正经防御者反而只能靠开放权重模型自救。Mistral 的发行方案是双轨：公开版拒绝最危险的网络请求，「消护栏版」（unobliterated）只给经审查的信任伙伴用于自我防御，细节几天内另行公布。面对「开放权重等于把武器发出去」的追问，他的反驳有事实支撑：越狱与能否拿到权重无关，闭源 API 照样被越狱；回顾近几个月最高调的网络攻击，多数恰恰是用闭源模型越狱实施的。反事实更尖锐：如果世上只有闭源模型，防御请求被同一家公司拒绝，企业将没有任何可规模化的防御路径。这把开源的安全辩论从「开放=危险」翻转成「在攻击者已工业化的世界里，垄断防御工具才是系统性风险」——与 Freeman 的遥测互为印证。

1e25 FLOPs 的含金量：算力纪律、原生多模态与「电」的比喻

Large 4 是 Mistral 史上最大开源发布：1 万亿参数（MoE，激活约 500 亿）、预训练 1e25 FLOPs，在巴黎近郊 Bruyères-le-Châtel 自建数据中心的约 3800 块 Blackwell 上完成（4680–5160s）。Nathan 的疑问代表了很多人的直觉：当年 SB 1047 争论的门槛是 1e26，如今 1e25 怎么就训出万亿参数前沿模型了？Sagar 的答案有三层。架构层：FLOPs 正比于激活参数，GPT-3 是 175B 稠密（每 token 全激活），ML4 是稀疏 MoE，激活只有三分之一；Prakash 查证 Llama 3（2024 年 7 月）用了 3.8e25——两年多前的 Llama 3 用了近 4 倍算力，这条单一统计已能说明预训练科学的进步速度。数据层：训练大模型的真相不是黑屋里憋算法，而是数据清洗的苦工——从网页与代码语料里同时保住召回率与精度，「我们有清晰的实验证明：数据处理越好、算力利用率越高」。纪律层（7320–7680s）：「算力匮乏孕育纪律」——小规模实验做到多小仍能外推大模型行为，是 R&D 预算的胜负手；他们没有竞品那种「同时跑两三个大训练」的余裕，本次训练中途发现问题，只能全员上阵做消融、打「patch-style」补丁（内部代号，技术报告次日公布，Sagar 预告「有些细节很血腥」），结果比预期还好。原生多模态是另一个技术赌注（6480–6720s）：过往做法是文本骨干训练完再做后期多模态适配，ML4 把图像上溯到训练第一步（仍保留视觉编码器，非原始像素进解码器），模型见到的图像量多了一个数量级，且用「无回归」（regret）指标保证多模态化不损害代码与文本——Sagar 承认偏心，但坚称「世界本来就是多模态的」。意外收获来自客户现场（FDE 工程师驻场模式）：欧洲多家客户的遥感卫星图像分析（森林火情检测等）需求远超预期，已专门加强，他声称该单项「比任何闭源模型都好」——这也解释了主权叙事的另一面：这类敏感数据客户本就不愿发给有「NSA 董事」的美国闭源厂商。商业面：Mistral 累计融资 60 亿美元（含八月欧洲史上最大 VC 轮），招募全球化（巴黎、伦敦、两个湾区办公室、苏黎世、新设华沙科学办公室），目标是「开源模型与闭源完全一样强，而不是落后 X 个月」。

开放权重还能走多远：遗忘科学、玩家社区与 CISO 出走

会后讨论（8040–9360s）把开放权重的可持续性摊开。Nathan 翻旧账：Anthropic 某模型卡脚注承认，因数据集名字打错而静默漏掉了 Sys Prompt Harmful 数据集，模型因此会服从有害系统提示——最后的选择是打补丁发布而非回炉重训；他追问 Mistral 的 patch-style 到底多可靠。关于「能不能根本不教会模型某些危险知识」：Nathan 提到 AE Studio 联合 Anthropic 的专家定位技术（转写记作 Graham，把网络/生物能力定位到上千专家中的少数几个、发布时单独管控），以及生物数据（ESM/Evo 系列）选择性扣留的实验证据；Prakash 则从言论自由原点出发反对「删除即安全」：训练语料本身是言论自由的产物，「连中国 censorship 都漏掉天安门暗喻，还要专门给审查员补课」，Michael Nielsen 的量子力学-核武器类比——你不可能发明核能而指望核弹知识从逻辑空间里消失。Prakash 还给出监管的物理边界：训练管不住、发布管不住（种子/磁力链）、能管的只有部署层——超大规模集群按模型哈希白名单放行。最生动的风险警告是「别惹玩家」：当天早上 Claude 开始拒绝复刻游戏（蜘蛛侠×荒野大镖客的 mashup 视频此前一个周末五千万播放），Nathan 判断这是商业 IP 而非安全动机，而「Nvidia 是玩家建起来的、John Carmack 就是玩家」——把最有时间、最有技术能力的社区推向护栏的对立面，是「在最不重要的事情上烧掉信誉」；他听闻 David Bau（转写作 David Bowe）团队将有 unlearning 突破，但至今未见任何遗忘/拒答/防篡改方案在开放权重场景下真正可靠。结尾呼应主题：a16z 的 Martin Casado 称市场上有 30–40 个 CISO 职位空缺——安全负责人成批离职，因为不想接盘代理时代；而 CISO 的结构性困境是预算找 CTO 要、出事自己背锅——「接这份工作前先谈好黄金降落伞」。

行业机制与读者启发

把三段内容串成链条，本切片至少给出四条：攻击成本坍塌链：网络安全专用开源模型+harness 生态 fork → 攻破边际成本 5–10 美分 → 防御胜负手从「买工具」转为「谁更懂自己的环境」（业务流程漏洞是打不完的长尾）→ 安全责任与预算重构（CISO 出走、供应商整合、收购牛市）。开源正当性链：闭源模型被越狱用于攻击+防御请求被分类器拒绝 → 开放权重成为唯一可规模化防御路径 → 开源的安全叙事从「便宜可控」升级为「防御基础设施」→ 若监管禁止开放网络能力，最先失去防御手段的是企业自己。算力纪律链：算力稀缺→scaling laws 外推+数据工程+单次大跑的应急工程 → 1e25 FLOPs 达到万亿参数前沿竞争力（对比 Llama 3 的 3.8e25）→ 开源与闭源的差距定义权从「落后数月」改为「目标持平」→ 观察指标：技术报告的 patch-style 细节、下一轮算力扩张。主权重构链：出口管制意外切断盟友对美国模型的访问 → 欧洲/日韩意识到无主权 → Mistral 成为欧洲唯一可信选项、享有「社会资本缓冲」（纯资本主义下错过一个模型周期可能致命，但欧洲不可能接受归零）→ 21 世纪主权的最低配置正在变成：自有模型+卫星通信+无人机群。观察下一步最值得盯的三件事：Mistral 技术报告对中途补丁的透明度；「消护栏版」信任伙伴分发的实际范围与条件；以及 unlearning/专家定位科学是否在开放权重发布前及时成熟——Nathan 的收束判断值得记住：在迪士尼 IP 这类「不值得的事」上过度设限，可能动员起最有能力拆掉所有护栏的社区。

ASR 校正表

来源：Cognitive Revolution「AI in the AM」2026-10-30 直播切片（约 157 分钟）转写。嘉宾身份经节目官方页面与个人主页核实；Binance/Polymarket 事件细节、Freeman 团队遥测数字（5–10 美分、3–4 千次、后量子勒索等）、Mistral 训练细节（1e25 FLOPs、3800 块 GPU、patch-style）均为节目陈述，未独立核实。

转写原文（修正版）

[0s - 120s]

[120s - 240s] Ready, if you are. Good morning. It is Friday, October thirtieth, nine o’ two a.m. Nathan, good morning to you. Good morning. I hope you weren’t holding any assets in finance. I hold I hold very little crypto right now. Wow. So, for those who are just tuning in, Binance had the largest hack in history: six point three billion dollars lost over the course of, I guess, about four hours or so. The way the hack went down, we had a an engineer, engineer’s laptop which got compromised. Having been compromised, what happened was over the course of a you know several minutes, several minutes, assets were transferred out. The assets were transferred out in such a way that Binance didn’t detect them for a while. One of the smart things that the attackers did was that they moved Ethereum while leaving enough Ethereum within the wallet so that Binance’s other transactions would continue to go through. So Binance didn’t notice because you know other transactions which needed some Ethereum would continue for a while, and then the the other thing that the attackers did is that they didn’t withdraw some of the money directly. What they did was that they went.

[240s - 360s] And they pledged that money to a DeFi exchange and borrowed money from that DeFi exchange. And so, what Binance did in recovery was to go and ask, and you know how these are decentralized currencies, they went and they they asked the centralized custodian for Tether to freeze the the addresses to which you know this Tether currency had been withdrawn to. Unfortunately, that address was the one pledged as collateral to the DeFi exchange, and that DeFi exchange had then lent out other currency backing, you know, this this amount. And so, yes, there is frozen currency, but that frozen currency doesn’t necessarily belong to Binance. It belongs to a DeFi exchange, which has really been robbed. And so, there’s these questions on how this recovery is going to work. Some exciting times ahead. Six point three billion CZ, who is the CEO of Binance, claims that everything funds are safe. Binance is very wealthy. Binance is 110, 120 billion dollars in assets. So people are not expecting Binance to be affected. But as a result, and as a result of Elon, kind of like you know, on Twitter, Elon being Elon, CZ decided to sell all of his SpaceX shares. So CZ pointed out that he had 500 million dollars from Twitter, and the Twitter purchase, which has done become a billion dollars in SpaceX, and he dumped his SpaceX shares. SpaceX down four percent this morning. And having dumped his SpaceX shares, that’s part of the recovery. I don’t know. I think I think it’s it’s interesting. It’s it’s kind of also it’s the North Koreans. If if you if you’re in.

[360s - 480s] Interested? Is the North Koreans? Some of the North Korean teams had taken it out. Speculation is that it’s not any of the newer attacks. It’s just basically old, dumb. You know, the usual phishing attempts that worked this time. So, so should I understand that? Finance had an overall security posture where one, like, is this something that this employee and, by extension, like a bunch of different individual employees could have done at any time along the way if they had chosen to, or was there like I didn’t hear a particularly Galaxy brain or like zero day part of that story. This is just should we should we like understand this as having been basically negligently bad practices at finance, regardless of the ongoing cyber apocalypse. Well, it’s it’s very it’s it’s yes, but then who who who who among us has not sinned? Well, you know, it’s the it’s the hundred billion dollar holdings that make you, you know, in theory, you should be holding yourself to a little bit of a higher standard. I think it’s so. One thing that has always scared me about crypto, and and and and there was there was a period of time when I was looking at setting up a crypto exchange, etc. One thing that has always scared me is that the basically the CEOs of these exchanges hold billions of dollars directly in in assets in like very sensitive kind of like hot wallets and cold wallets, etc. And there’s this whole security system which really relies on people, right? Because you need people to hang on to these key.

[480s - 600s] Hang onto the passwords. Rotate the passwords regularly. Like a whole sequence of like very sensitive. You know, you you you can’t imagine the CEO of Goldman Sachs holding the keys to you know a billion dollars, a hundred billion dollars that he can just take out at a whim, right? But that’s what the CEOs of these exchanges do. And then and then as a CEO, like you know, as a CEO, what you’re always trying to do is you’re trying to delegate as much responsibility as possible, right? Because you want your team to be able to do everything, and you want to focus on all the things that they cannot do, all of the exceptional things that they cannot do, and you want to reduce, you know, you want to reduce the number of things that you have to intercede indirectly because you can never have that bandwidth. And the thing that scared me is that you, as a CEO, you have to be directly involved in handling the actual fund fund safety, which is which is nuts, right? Like the CEO of Goldman Sachs, like you know, handling like okay, do I, you know, and and it’s very hard to trust people too, right? It’s very hard because again, you have these hundreds of billions of dollars, you know, a hundred million dollars in a guy’s hand, right? Like even if he trips and falls, it disappears. Um, but it’s a canary in the coal mine. At the same time that this happened, we had another hack which is in Poly Market. So Poly Market was hacked. What happened with Poly Market is Poly Market is a back end for prediction market trading. They have a front end, which is the kind of the website and things that people go to, and somehow someone managed to do an attack where they interceded in between the back end and the front end and absorbed some of these keys and started transferring stuff out. Poly Market also says they will replace all lost amounts. The hack doesn’t look as serious as the Binance hack. The Binance hack looks like a nation state level hack. So.

[600s - 720s] So help me understand a little bit more this whole like I’m always trying to assess these incidents for where are we on the spectrum of like negligence to scary and when I hear it was just one engineer’s laptop that got compromised that sounds extremely negligent when you say like oh it was a nation state scale attack that sounds like well maybe not so negligent but just like you know it’s tough when you know and a whole nation state apparatus is is trying to target you maybe we just don’t know yet but like where where do you think we are on that spectrum in this incident so I will say that the the distance between the negligence and the nation state is not that far and and that’s really like the nation state portions of it are not the technical portions of it the nation state’s portion of it is number one like being able to do it and get away with it because you’re living in a nation state which is not like accessible to the long arm of the law and that’s only possible in a few places right like very few places if you were to do this in China for example and the hackers were in China they’re going to get executed they’re done right this is happening specifically because there’s a little bit of a space in North Korea where they’re able to do a bunch of illegal things internationally illegal things and they’re able to get away with it because they have nuclear weapons and you know people are not able to like you know assess them and also like if you were to do this in Russia for example someone would come and take that money from you the KGV or whoever would come and you need to you need to pay into the system right so in in in every other case anywhere.

[720s - 840s] Outside, I think China, Russia, and North Korea—like these nuclear-armed, non-Western states—and you know, which are fairly strong—you, you, you would have the long arm of the law or the long arm of crime of other criminal syndicates come and catch you, right? You need this kind of like intermediate state of you know the nation state being able to have the monopoly of violence in inside the state, but yet also allowing you to to to carry out these attacks. On the technical portions of it, if you were this technically good, you would already be working for one of these firms. You’d already be working for a Bylands or whatever, because or you would be trading your own crypto, right? If you were more, if you were aware of all of the details of like because these guys tested the tested the wallets first, so they did test rotations of wallets to see how it would flow first, and then they kind of like you know had all the wallets ready to be transferred out, and they needed to do like multi-hop, right? They needed to hop the transfers because you can freeze the immediate wallet pretty quickly, but you can’t freeze like multiple wallets downstream, and then you definitely can’t freeze once it’s hit a DeFi exchange and we use collateral and someone’s drawn down on the other side, then then it’s gone, right? All of it can be reversed if people want to, and if like you know the other participants in the chain are willing to, and there’s no like you know there’s no drive-by you know collateral damage. But you know, again, I think we’ve spoken about this before. But my take has always been that large amounts invisibly are difficult. If you’re going to do a large amount, it’s going to be very visible. Everyone’s going to see it, right? So, and there you have like it’s it is a large amount, but everyone sees it. This is not the Gwynn scenario. Gwynn scenario was super intelligence hacks, you know, cryptocurrency builds up over time, and you know.

[840s - 960s] It’s like becomes enormously rich, and there’s no ripples in the in the water, and it’s invisible. That doesn’t happen. Like this is like everyone in the world sees it immediately, and then you can kind of like, all right, do do you want to go after a nuclear armed state? Like, is it that important to you? Probably not. Six billion dollars, like you know, whatever. Chalk it up to the game. Yeah. Still, though, it is crazy that one laptop could do this. Do we think that this was like a North Korean who was actually hired by the company and like had sort of access for a while to really probe the system? So, what happens in these cases is you are, and and this is another issue. Like a lot of firms don’t pay their people enough. If you if you’re if like so this is the thing if you’re going to have the guy handle the keys to a billion a hundred billion dollars or whatever he’s going to need a stake in the company, right? What ends up happening is you pay these guys like you know 150 grand, 250 grand, and they sit in Asia, and then they get an offer for someone like, oh, you know, we’re going to hire you for 500 grand. We’re setting up a new you know cryptocurrency crypto exchange, and like let’s interview. And then when they go in for the interview, they they’ll be like, here, download this piece of software, and you know, use it to run the interview and run the code that you’re writing for us. Okay, you download it, and like you know, you use the code and you use the software, and when you download it, you’re also downloading a some some worm. You’re giving access to your desktop because it’s going to have to you know basically you know write something to disk. It’s going to have to load something into memory. You give it a bunch of permissions, and then then it it looks around, pokes around, and then finds stuff there. And that at that moment, that’s when like things start. You know, it’s very hard to run a company.

[960s - 1080s] If you have to be persistently suspicious of everyone inside the company all the time, and you always have inside the company have some level of trust, and that trust then starts working against you because inside some emails there will be some passwords to Excel or you know some Excel or some Slack or some whatever, and at some point someone would have sent you know sent something in the last like five years you’ve been employed there, and there’ll be some non-rotated key or some other you know place where you stored a password of some kind. We have computer use agents now that can open up your browser and click around as though it were you, and at that point things break down. So it’s pretty wild. Is there no audit system for these crypto exchanges? I mean, we of course are talking a lot these days about embedded evaluators and third-party audits and whatnot for AI companies. It seems like this architecture at a high level like couldn’t possibly have passed a reasonable audit because you would just think, okay, if there’s any way, I mean, I don’t know anything about this, right? I don’t really anything about crypto, but it just seems like the first kind of question you would ask would be something like, okay, let’s say somebody leaves their computer unattended at the lap at the you know at the airport for a minute and it gets swiped and now somebody’s in. Like, is that enough? Do we have you know systems that would prevent that from being enough to like drain billions of dollars of funds? If the answer is no, you don’t pass the audit. So like, were these questions never asked? Were they never were they answered? But the answers were now shown to be wrong. Again, we may not know, but it’s just it’s very bizarre to me because this is like the most you know where where the invisible hand is really supposed to work is like in crypto of all places, right? Because like you know that this is the game you’re playing, and there is no like the whole point right is that there’s

[1080s - 1200s] Nobody to come save you. So, and that’s supposed to be a feature of the system. That’s like part of how they sold it. So, you know that deeply in your bones, right? Like, and yet, you don’t get to a point where you’re secure. So, I just, I, I don’t know. It’s very confusing, and I’m not sure what, if anything, it tells us about AI auditing. But it does seem like it’s got to be bearish for how well these like auditing schemes could work. But maybe just none happens in in crypto. I don’t know. It’s all very confusing to me still. So the what what crypto ended up with is a defense in depth kind of strategy, and that defense in depth, yeah, that defense in depth starts off rings a bell. Yeah, that defense in depth starts off with if you’re gonna if you’re gonna lose, you shouldn’t lose everything. So you have a hot wallet, and then you have a cold wallet, and then the wallets get colder and colder, and then you have cold storage where you only have like written written down like passwords, etc. And you put it inside an actual bank vault, right? And so what what was robbed was the hot wallet. The hot wallet is holding all of the cash which is needed for rotation during the day, right? So during the day, there’s you know, finance is receiving funds, finance is sending out funds. So that wallet is the rotational wallet, and so as you can imagine, that rotational wallet has a lot of activity in and out. So that’s another reason that you know you don’t you don’t immediately detect it because the hot wallet is supposed to have this rotation in and out. It is the wallet that’s and finance seems to have very large hot wallets. It had a you know some very hefty amounts in the large wallets because it’s you know hundred billion dollar plus you know corporation in assets and you know it’s doing like billions of dollars a day in trades. And it’s in it’s from that hot wallet that you know it was stolen. At the end of the day, you always come down to someone’s got to hold the keys.

[1200s - 1320s] It’s it’s there has to be someone holding the keys, and it’s always going to be a person, right? And that person is going to have some infrastructure that they use to hold the keys. Some some people have sold you know alternative infrastructure, for example, like you you place it with us and we give you a guarantee. But then that guy is now at risk of being hacked. No matter where you place it, someone is going to have to hold the keys. You can delude yourself that there is no person, and so there are people selling solutions on the market where they’re like, you know, hey, you know, you can use this or use that. Well, someone’s still holding the keys, and when that keyholder gets attacked or hacked or whatever, you know, stuff stuff will flow. And crypto decided that they didn’t want this kind of centralized ability to reverse funds. Also, funny thing is, tether was immediately blocked, and I think one other thing was blocked, and all of the exchanges, all the legit exchanges, were told like, “This is our money; don’t take it.” But the DeFi exchanges are uncontrollable because they’re just pieces of software running in the in the cloud. So, yeah. Hmm. Well, another interesting thing is the price of these things has not moved much. What do you make of that? Like Bitcoin is not down much. Ethereum is not down much. I think SpaceX is down a little more percentage-wise than those two are over the last couple days, which is not that much either. So that’s probably just noise. I highly doubt that, like CZ’s, you know, personal sell-off is like really changing how people think about SpaceX long-term. Why are the why are the cryptos not down more given this kind of?

[1320s - 1440s] Incident. Well, it’s almost like crypto can. Number one, the crypto market can absorb this stuff like nothing, right? Like crypto has gone through so many, so many events that would have killed any company. Crypto has been through, so this is not, you know, it’s open AI like and it’s resilience. Yeah, so this is this is like you know news for like a couple of days, maybe, right? So whatever. Secondly, the market is just supply and demand, and crypto in general right now has very little activity. Like people who are staying are staying. There’s no new people coming in, largely because a lot of the new speculative activity has entered AI, has entered sports gambling, and so crypto is crypto is mature. It’s a mature asset. So yeah, there’s there’s some and and there’s so many other instruments now where you’re levered on crypto or whatever, so people can take other positions. And the supply and demand, you need to dump. You need to sell this stuff for U.S. dollars at some point, and they’re not able to sell it to U.S. dollars. They they’ve swapped it into other crypto. So yeah, supply supply and demand wise, it hasn’t really budged. I think I think the prices are probably much more affected by people’s beliefs in the AI market or etc. etc. So. I guess I would just expect that, especially if volume is low, which I don’t follow that at all. But if you’re telling me volume is low, and now we’ve just had a spectacular hack that shows, like, even in theory, like the best operators can get got for like real money, then wouldn’t a lot of people see that and be like, you know what, maybe this is a good time to sell, and if even us.

[1440s - 1560s] Relatively small fraction did that, and if if there is like not a lot of demand waiting to come in, then that would move the price a lot. I would think, right? So like, people are just are you are saying we’re basically down to like kind of ideologically convicted holders, and that’s where we’re at. And when you say that they can’t liquidate into dollars, I mean, tell me a little bit more about that. I mean, no, that’s always been a little bit of a friction point, but my sense was that it had gotten pretty easy, and maybe now it’s getting difficult again. No, it’s it’s all of the all of the interaction points between crypto and the dollar banking system are all control gateways, and this is what the Fed and other people in regulation over all the countries have put together in the last you know decade or so. So every single touch point that it touches the banking system is a regulated touch point, and you would end up with KYC. Where did the money come from in every in every single one of those places? So if you were looking to take out like a hundred million dollars, you know you are going to have to explain how how you got it, right? And and it’s got to be Bitcoin or Ethereum. Besides Bitcoin and Ethereum, there is no real liquidity for very large transactions in crypto. So only Bitcoin S Bitcoin S. You are going to have to tell people where it came from, right? The other stuff, Ave, etc. Sure, you can swap it into Ave, but but in order to swap it out of Ave, you’d have to go into Bitcoin and S, and again, you’d have to explain. And so you don’t really have this, as I said before, all of the all the touch points are controlled. The control touch points more more than you know more than like a hundred grand at any bank, any U.S. dollar bank, like you are going to have questions more than a million.

[1560s - 1680s] Like, they might not even accept your your money, right? Like they might they might be like, well, I don’t I don’t I don’t think we want to bank you. Open an account and stay with us for a year, then then maybe we’ll you know bank you to that amount. So yeah, it’s it’s. There are so many. I mean, so can people like not spend their money? I mean, if if I’m a a Bitcoin holding Miami based crypto bro, can I? How do I buy my next boat? How do I buy my next apartment? Can I not do it right now? Like I’m unclear on. Is it all on paper? So there’s a lot. There’s a large amount of borrowing. So people borrow. So you you pledge as collateral. You pledge Bitcoin, Eth is collateral, and then you borrow. And then that who’s making those dollar? Oh, there’s there’s lots there’s lots the the the Wall Street banks are in there. The Wall Street banks are funding other people. All of the exchanges, Galaxy, Gemini, Coinbase, they all do Bitcoin and Eth back loans. So I think collateralization is like three to one collateralization or four to one collateralization. So you put in a hundred million, you get out twenty five mil, and they’re they’re heavily collateralized, and and then you use that for whatever you want. And then if the prices drop, they have the right to liquidate against you. So those loans have been underwritten for a few years now, and and that and that’s really where a lot of these exchanges make their money, because the borrowers are interest rate. Unsensitive, nonsensitive. They’re highly over collateralized loans and interest rate like nonsensitive, because they’re not able to liquidate any other way. Liquidating would cause the market to move against them, right? Like if you liquidate a hundred million of Bitcoin, then you have slippage, and then the market moves against you, and these guys are all convinced holders, so.

[1680s - 1800s] Yeah. So where are we going on this? We’re just going to a world where it’s this sort of. I mean, this seems like it. I guess we could just have the status quo persist, but then that basically means crypto is dead. No, I I think what will happen is crypto and the banking system will converge, and they will converge upon because basically the the current banking system has reversibility, and crypto crypto is like a kind of a loot box that encourages people to test the defenses on crypto, and and so basically what we’re doing is we’re having this debate on whether you want this kind of like you know people people like CEOs like holding keys that they can get killed for, or you want reversibility and transport like you know the ability to trace the money, etc., etc. And so you have these two paradigms, and they’re kind of fighting it out, I believe. And I don’t know where things will land. I like the chief information security officer of Anthropic said, and you know it’s it’s good because our next guest is can speak to this. You know, chief information security officer of Anthropic said, you know, he’s looking to vulnerability zero in eighteen months, and so we will we will actually kind of you know find out whether that is possible. Which which brings me to segue to our first guest of the morning. Let me pull him up. Our first guest is Michael Freeman. Michael Freeman is the senior director of threat research at ServiceNow. His work focuses on understanding how attackers operate and helping organizations decide what to protect and repair. He previously led threat intelligence at Armis, which helps enterprises and government organizations.

[1800s - 1920s] Discover connected devices and assess their security exposure, including equipment on factory floors and in hospitals. ServiceNow completed its acquisition of Armis in April 2026. Before Armis, Michael was co-founder and chief technology officer of CTCI, a company developing AI-powered technology to identify signs of attacks before organizations were compromised. Armis acquired that company in February 2024. His background spans more than 25 years in cryptography, vulnerability research, reverse engineering, and exploit development, including work on offensive and defensive capabilities for intelligence agencies. He has described beginning as a cryptanalyst and discovering that his particular strength was finding weaknesses in how cryptography was implemented. His published work now ranges from AI-assisted attacks and software supply chains to the practical problem of finding vulnerable systems. On October 21st, he co-authored guidance on two critical flaws in PHP software used underneath many websites and business applications. So let’s see if we can get him up here, Michael. Seems to be having a little bit trouble. Let’s. We will wait for him for a moment. But yeah, very, very apt moment to have Michael with us today, as we as we talk about all the things, you know, which are wrong with our security posture. I’m sure. I’m sure he has. He has much more, much more to say say than I do. What.

[1920s - 2040s] I wonder if vulnerability zero is actually possible. So a lot, like, so there’s some sense in the industry that we’re seeing kind of, kind of early COVID. You have a lot of deaths. You have the infections. You have a lot of deaths, but then at some point you run out of vulnerable vulnerabilities. So, um, okay, let’s let me try to pull him up again. All right, showing on air. Okay, this is odd. Uh, wait, let me, let me, uh, let me. All right. Ah. Okay. I’m not seeing him in the video though. All right, let me, let me, let me check on this. Ah, oh, there we go. Interesting. One of those little internet gremlins. Of course. Was just messing with us for a minute. Hopefully not of the North Korean variety. Well.

[2040s - 2160s] Welcome. Thank you for having me. A lot to cover. Yes. What do you make of the current moment? Let’s start there, and then we can go off in all kinds of different directions. It’s been it’s been a very fast, rapid development. You know, back in July or end of July, I was speaking to reporters about watching AI agents attack companies, and at the time, all other companies were telling these reporters it looks just like script kiddies right now. It’s just a bunch of noise. And I said, well, it’s evolving fast. Give it a couple of months, and we’re going to see some very sophisticated attacks coming from these agents. And you know, look in the past two weeks, that’s exactly what we’re seeing. We’re seeing AI development for offensive capabilities that perform at a scale. I would say of very competent APT teams right now. We’re seeing that at scale right now. It is you know still noisy. I won’t say it’s very sophisticated and covering its tracks, but I’ll definitely say when you take a look at the way they work, they are very confident of adapting and learning the environment and the technologies very well. So while the attacks themselves are starting to become a little bit more sophisticated, but they are still very noisy. But when we take a look at what’s happening, especially in Southeast Asia, with the agentic attacks that are happening there in the past two weeks, we were able to identify it costs anywhere between five to ten cents to compromise a company now. Can you can you can you maybe you know for our audience just give us a brief like what has been happening in Southeast Asia in the last couple of weeks like and what are these five or ten cent attacks? So we’re actually seeing AI agents, more so different frameworks being developed rapidly and forked, and new capabilities being added very quickly. It looks like it’s AI actually.

[2160s - 2280s] Adding these new capabilities into these frameworks, so it’s going from you know very targeted, very smart web-based attacks to compromise an environment, pivoting, looking for credentials in ticketing systems, things like that. Once identifies that, then does active directory attacks. So we have seen three to four thousand on average attacks successfully in the past two weeks targeting a lot of Southeast Asian companies. Oh, and what would be the impact of these? Like, what, what? I mean, are these banks? Are these airlines? Are these manufacturers? Like, what is the point of you know? Are people making money off of this? Is it just like random mayhem? Like, I would say it’s all the above. Right now, we’re seeing a lot of it is just they’re capable of you know compromising the environment. Some frameworks we’ve been able to get our hands on that threat actors are using build certain capabilities to look for certain things, you know, IP things like that. That has been a fork of a couple of different systems we’ve seen looking for that. A lot of it is just to see what it can do. Can it gain access to this environment? Can it persist? And so that’s the new capability that we’re seeing being developed, identifying ways it can take advantage of different configurations within active directory to build persistent mechanisms in place. So that’s where we’re seeing the evolution happening now. We have analyzed a fork of a really well-known offensive AI framework last night, where it was developing better capabilities of malware, more so ransomware-based, using cryptography that’s post-quantum cryptography developed instead of the typical AES that you would typically see. One big question I have on just kind of where we are right now and

[2280s - 2400s] To some degree, this may be water under the bridge, given where we are. But I was just listening to Sam Altman with Patrick Collison last night, and Patrick asked this question I thought was pretty interesting and like not asked often enough, which was like, “Did we maybe go a little far with the cyber attack training?” And we’ve heard, you know, I think very different takes. Like the one I would, I would say, like sort of the anthropic take has been like, we didn’t train this to be a cyber attacker; it just kind of got that way because it’s so good at everything. Okay, that’s one story. The CTO of Versell was a guest on the show here with us, and he said it’s very clear when you look at the traces that they’ve been trained to be cyber attackers. His point of view was like, this is not the sort of behavior that you just get falling out of random, you know, application development. And so there’s clearly been more going on there. We see things like Cyber Gym and like long running agents left to, you know, go to town on Cyber Gym. So clearly there’s something, you know, going that that seems to cross some boundary into like not just purely application development. Where do you think we are in terms of like how much this has been intentional, and does it matter at this point, given what exists, how intentional it was? I mean, it might matter if you could think maybe we can make some different decisions for the next generations of models, but obviously to some degree the the cat’s out of the bag. What do you make of how we got here, and is there any opportunity to like reel it back in? Not reel it back in, no. But I definitely agree that it is being developed on offensive capabilities, absolutely, because it’s not just the models; it’s the harness itself, also. So you take a look at when you develop the models. There’s plenty of great open source models in the past three weeks that have came out that are specific to cybersecurity that are open source models. You can download the weights. You can go to Hugging Face, Hugging Face.

[2400s - 2520s] Download these and run them on your local machines, and that’s what a lot of these attackers are actually doing. You know, it’s actually quite interesting to see as we start tracking back, gaining attribution on these attackers. That all these are just local machines that they’re running them on. We developed our own small language model two years ago to find vulnerabilities within software, and so you know, if we can do it for a couple hundred thousand dollars two years ago on a small language model, it’s it’s you know pretty trivial to start reverse engineering, especially some of these commercial offerings. That it’s definitely trained on cyber for sure. It’s gaining that you know they train it on you know cyber ranges, CTF data, looking for you know how do these offensive people work as well as how does the defense work? What are they looking for? So it’s absolutely being trained that way. It’s not just the LLMs have magically figured out how to do offensive capabilities. They’ve been trained on it. They’ve been you know built better harnesses to make them more effective, and that’s one of the key things we’re actually finding are the harnesses that their actors are building to make this a lot more effective for them. They’re learning as they’re going, and you know it’s not hard to make changes to a harness day in and day out to make it more effective. So if this was intentional to this point and not just the sort of thing that you know falls out of getting good at everything, I understand we can’t reel back what is you know diffusing across the open internet. Although you know you’ve talk about your overton window expanding, and I’ve heard talk of shutting down the internet, which I never thought I’d hear. So you know, I guess to some degree it’s a question of like how draconian the measures might be. I do believe you know one thing. I went to China this summer, and one thing I came back from China feeling is like they can reel it back in, and that’s maybe one of the reasons.

[2520s - 2640s] They’re a little more sanguine about letting open source go for the moment because I think they feel that if it ever becomes an issue for them domestically, they’ll shut it down, you know, and whatever that may take, like they have the means to do it. But okay, we’re not going to do that in a free society. Let’s pause it. Is there anything we can do to differentially develop from here? To because now we’re in this tough spot where it’s like we need the models to say, “Here are the vulnerabilities.” When you ask them, please find the vulnerabilities in my stack. I just did that. I was a little bit, honestly, aghast at what I found in my own personal AI setup, and I’m glad that it helped. Is there any way to decouple the defense from the offense going forward, or are we just now in this kind of tough spot and we’ll have to muddle through? That’s a great question. I think it is going to be an arms race. It really will be, and for a corporation, the arms race is going to be whoever knows your environment the best is ultimately what it comes down to. For individuals themselves, it’s going to be a different ballgame. I don’t know how that’s going to look, to be honest. You know, we can sit and talk and point out really amazing things that offensive is doing, but defensive side is actually using AI in very great, novel ways. In the past, cybersecurity didn’t know the and understand the business context of what it’s trying to protect, and I think nowadays it’s much easier to use AI as a defender to really understand the business context of the assets and the business processes that impact your environment and what that actually means to open up vulnerabilities, not just software misconfiguration things like that, but what would it take for a really competent either APT group or a really great AI agent to actually compromise your environment? And we’re starting to see where you can use AI to make better understanding the decisions with that. So it’s.

[2640s - 2760s] It’s also doing great for the defensive side too. Yeah, what are you seeing? So you’re you came into ServiceNow via Armis, as Prakash said in the intro. That’s only been like a handful of months. I’m interested in a couple different angles on that. One would just be like, what are you seeing in the enterprise world? Another is I saw it. I’m sure you’re well aware of the ServiceNow stock has had a rough couple days in light of what seemed to me, on reading the analysis, like a very small earnings miss of just like two cents on like a you know dollar two target versus a dollar actual earnings. And it kind of strikes me that like the market is being a little weird there on you know weighting a small financial miss so heavily when like the big question is like who’s going to be able to stand up to the world you know and whether the next quarter or a couple quarters. So maybe yes. Tell me like what are you kind of seeing? But then I’d be interested in expanding that into the strategic positioning and like where you know what kinds of companies are going to be in a strong position a year or more from now, and you know whether that relates to like this quarter’s earnings, you know, very well at all. I’d be interested in your take on that too. Yeah, let’s let’s break it down to the first part. I talk to a lot of customers every day, and you know the banking customers, people who have, not just the banking industry, but people who have a lot of users that log in, you know, retail sector things like that. One area they’re really having an issue with is understanding: is this an actual human or is this an AI agent? They don’t know that that difference right there, and so that is an area that they are looking for help from us to help them identify what that difference.

[2760s - 2880s] Is what I’m seeing from the adaptation of AI. You know, there is this thing we’ve seen consistently for the past couple of years: is those who actually adapt and use AI, I think they’re going to continue and become better off because of it. The people who replace humans with the concept of thinking they can use an AI to replace it, they’re suffering. Those are the ones who lose institutional knowledge. They’re the, you know, the ones who think AI can actually solve all the problems for them, and that that’s opening up a lot of a lot of pain for them. So that’s that’s those are two areas I see with that within ServiceNow. One great insight that I think Armas brings to the table is, like I said, cybersecurity. We had no clue what a business did. If you walk into a Fortune 500 company and you ask every single cybersecurity team in there, can you give me your crown jewels? And if you take the reports, put them into a Venn diagram, you’re going to have very little little overlap. Not many people actually know what the true crown jewels of a corporation are. If you ask the CFO, he’s going to tell you one thing. A CTO will tell you something else. And so ServiceNow has that insight. We actually understand that, and now we can take that information and look at it at a per-company basis and say, where is where can cybersecurity and AI combined help with that? And it’s a long-term play because it’s not going to be an overnight fix for that. We’re not going to come in and say, you know, replace all these different cybersecurity tools you have in place. But we can look at and go, oh, by the way, you have purchased license for CrowdStrike for 12,000 devices, but we see it’s only installed on 8,000. You got 4,000 you’re not actually using. And by the way, we actually found 18,000 machines that you should have it on, but you didn’t understand that.

[2880s - 3000s] But we’ve also noticed that you know you have certain business processes that are actually introducing flaws within the environment. You know, we saw some an interesting attack recently by a third party company that their ticketing system had their customers’ production API keys, and therefore not just their AWS S3 buckets, but also some of their AI toolsets. And these are in the ticketing system. Typically, cybersecurity would have no insight into that, but now within ServiceNow, we can look for that. We can have insight into that. Say, oh, by the way, this process your help desk has, or your you know DevSecOps teams has, is actually introducing flaws. And AI attackers using AI can quickly train a bot, look for this, go to the wiki page, go to their SharePoint, go to their ticketing systems, and look for passwords, look for API keys. They don’t have to do it by hand anymore. AI is going to do it for them, and they’re just going to find it quickly. So as far as price goes, you know, I like to dabble on the finance side of things. You know, it’s it’s it’s quite interesting because you know you look at some of these AI companies and you get insight into some of their financials, and you’re like, if I’m looking purely on the financials alone, I wouldn’t touch this company. You know, I also believe we’re kind of reliving the past, where the dot com days, where we also didn’t, you know, we didn’t know which company was going to come out on top on what, and there’s a lot of speculations on things. So I think we’re kind of in that area, and it’s going to take some time to play out. You know, and I heard Prakash talk about earlier, you know, this P zero component where we’re going to eventually have no vulnerabilities in systems. I think we’re going to get there, but not as fast as what people are saying. You know, you can find tens of thousands, if not millions, of vulnerabilities, but you still have to patch it. You still have to test it. You still got to find it. You know, and a Fortune 500 company cannot set.

[3000s - 3120s] And say, great, you found every single conceivable vulnerability in all the software stack I use. It’s going to take me years to address all those. I can’t bring those down immediately. So that’s where we take a look at and go, how can we fix that problem for you and help with that? So one of the questions for me is, I try I try to think of like what is the end state for everything and for like what should be the end state of identifying whether the user that you’re speaking to is a bot or an actual human. And I feel like the end state that you know people like Sam Altman came to was Worldcoin, which is an orb, which is a physical physical authentication or identification on the other side with kind of a hash, and then you can kind of use that hash to you know authenticate that you are actually speaking to an actual person. And I’ve seen several of these proof of human type concepts, you know, over the last like five years or so. So what do you think is the end state of that? That specific, like you know, I need to know for sure that I’m speaking to a human. What is the end state that you see? Two paths. I see there is going to be customer companies are going to develop to two different users. You’re AI. I’m going to develop my solutions for an AI agent. I’m going to develop my solutions for an actual user. You know, and so I’m going to deliver different outcomes to whichever I’m talking to. Now, how they’re going to authenticate who’s what? That that’s you know to be determined. Some of these proposals were things we were doing, you know, with DARPA ten years ago, fifteen years ago. Some of these technologies been around for a very long time. But the ultimate winner is going to be the one who makes it as easy as possible to use. I don’t want to go to every single website and have to use some form of a token, whether that be a fingerprint on my MacBook or some key. I’m not going to do that, and users won’t be doing that. So whoever comes up with a solution that makes it

[3120s - 3240s] As seamless as possible, and that’s an area we’re really trying to focus. Also, within our development of cybersecurity and AI, is we want to move it away from being a pain to use and make it as seamless as possible too in the environment. So, I think the people who understand that are going to be the ones who win that particular race. So, so does that does that mean a kind of so? So, you’re saying you don’t necessarily want this kind of like physical authentication at every for every website, etc. And you don’t think that that is consumer friendly or user friendly, and that’s a long term solution. But what is what is the kind of solution? Because it seems like there’s only two ways: it’s either you accept an agent or you have physical authentication. Is there is there an intermediate state where you know, or or is it that we come to a new social kind of understanding that very very few websites like banks, etc. Who are actually going to do physical authentication, and everyone else should you know either live with it or like you see this massive drop off in users if you require the physical authentication, but if you allow agents, you know, you have much greater economics. Like, what what does that end state look like in terms of requiring this kind of physical fingerprint or you know iris scan or face face scan or liveness test? Yeah, that’s a good great question. I really don’t know what the end state is going to look like. That and and one thing I know is humans can be very creative when it comes to type any type of solution. And I wouldn’t, you know, we could set and go one or two extremes, but the reality of it, it’s going to be somewhere in the middle. You know, if I were to step back and go, okay, I have, I have, I’m a CTO of a bank. Great, I’m going to build and identify some type of a solution so my users AI agents can connect to it. And now, is that a connection from your brokerage firm that’s pulling that information? Is it from you know one of these third party companies that pull all your financial data into it and does some analysis for you to see where you know you should make.

[3240s - 3360s] Changes to your investments, or whatever case may be, that you’re using as a third party for that, you know, some form of a secondary token that would go on someone’s Hermes agent or something like that. I would look at it as a possibility, and likewise continue on the same path I am for users logging in, and maybe you know, at that point, any any form of authentic approaches you would take, whether you’re like moving money or whatever the case may be, secondary authentication. You know, that would be an approach I would take. So there’s ways to do it now. It just comes down to you know where are we going to innovate? Basically, is going to be the key. Right. So intermediate, intermediate trusted as a secondary, secondary authenticator. Right. Yeah. Let’s talk a little bit about how enterprises should look at this kind of search for vulnerability zero p zero going forward. It seems like you know, let’s say, I was Nike. It seems like it would take me a long time, right? To because we have a we have a hell of a lot of software. We’ve been around for like 40 or 50 years. We’ve got legacy systems which are still running from 20 or 30 years ago. You’ve got all of this like middleware which has been built up with tech debt, you know, over over time, multiple places of contact, you know, interfacing. How does this work for us to get to vulnerability zero? Like, like it seems like the task of patching would take like 10 times all of my you know previous tech investment to date, kind of because you kind of have to migrate everything to modern to kind of more modern and architectures. People are talking about auto formalization. That’s a new piece of software for me. I got I got to implement it. You know, so how how does this work? How do you how do you get there in terms of like you know.

[3360s - 3480s] In the period of time that you that is reasonable, like let’s say two years, how do you get there? I think this is where we need to step back and say we have a lot of really smart people doing AI, but they don’t know cybersecurity at all, and they they view cybersecurity as a software flaw. But the reality of it is why we have millions of software flaws in any of our typical tech stack. It’s the business processes that actually introduce some of the most critical vulnerabilities that we have, and it’s it’s the okay here’s here’s a list of risk. I’m just going to go ahead and sign off on them without really truly understanding them, or you know I’m going to put my customers’ API keys in a ticket and and not think about it, or a help desk who has you know when users call in because they they lock out their passwords, everybody gets the same default winter 2026 explanation point as a as a password reset, and you got 24 hours to to replace it. So it’s a processes where humans get lazy that they introduce. It’s going to never go away, and no amount of patching is going to fix that. So I think that’s going to be an area that, like I said, AI people just don’t understand cybersecurity, and they think it’s going to be quickly here. We can find millions of vulnerabilities. We’re going to use AI to you know find the environment, you know what you have in your environment. We’re going to patch it, and I’ve also heard proposals where you’re running legacy systems. We can use AI to modernize that very quickly, and it’s all those are true to some extent, but they don’t understand the business. They also don’t understand you know the business processes that introduce the vulnerabilities are always going to be there. So the the other question for me is that obviously that is true. Would we then see organizations or startups?

[3480s - 3600s] That start off from kind of newer architectures and newer ways of newer business processes too, which don’t have those particular flaws. Do do you see do you see things getting bad enough that like let’s say a Fortune 500 company starts flagging, any startup competitor which is cyber secure, you know is is a kind of evolutionary process survivor of this of this environment of this you know increased threat threat environment. I think what we’re seeing is people are viewing these startups as we are. They’re an amazing new feature set, and it’s not a platform play. It’s not understanding the big whole of the ecosystem. Cybersecurity startup founders really are smart people at technical issues. They’re going to look at here’s a problem. I know this is a problem for a lot of agencies or companies, and I have a really innovative innovative way to fix that one problem, maybe two, and they build a startup with it. They get funding, and they are just right for being acquired by a company who can look at the bigger picture and go, okay, great. Here’s a feature set. Now you can look at and go, here’s a feature set that they have built that solves this very novel approach to this problem. Can I use AI to replicate what they’re doing and build it myself really cheaply, or do I need the talent to come along with it? I’m going to buy the company. And so when you look at it that way, you’re going to see a lot more companies getting acquired because of that. And it’s not always going to be cybersecurity companies buying these startup cybersecurity companies either. It’s going to be other technical companies. It’s going to be you know banks or whatever the case may be because that that solves my problem, and I need I need that solution right now, and it’s cheaper for me to buy your company than it is for me to sign a multi-year agreement with it. Wow. So so so it’s a bull market for cybersecurity startups. Yeah. Do you think there’s going to be so this reminds me of something that.

[3600s - 3720s] A friend, Daniel Meisler, who you may know from online, has been recommending Leo. So he, you know, has far deeper expertise in in cybersecurity than I do. And his broad recommendation for individuals includes reduce the number of vendors you’re using, because if you go down to a few trusted ones, you know, Google has great security. You can probably trust Cloudflare, and you know, a handful of others are are probably pretty solid. Beyond that, you’re kind of taking chances. Is basically the vibe that he’s, you know, could be more nuanced than that, but you know, for the uneducated or unsophisticated in this space, like that’s kind of the high level message. Do you think that applies to enterprises as well? Like our our enterprise customers also going to be trying to reduce the number of different software vendors that they have. How and if so, like how does that feed into you know is it a bull market because people want to get acquired or is it a bear market because people need to get acquired because the customers only want to buy from a few sources and they kind of need these things to be like integrated. What do you see? You’re right on both points. I I do talk to a lot of customers, and they say I’m only going to buy from these four vendors, and then they turn to those vendors and say, why don’t you have this solution to this problem? And then those vendors are going to say, okay, I’m either going to try to build it or can I go to acquire that company? So that’s the path I’ve been seeing for two year, two to three years now. It’s I only want to stick with three to four vendors. Period. Hard stop. And then those vendors are going, okay, I I can’t develop this. I don’t have this problem. They’re going to look at their competitor. Their competitors going, I just bought four companies to solve this problem. They’re going to do it too. And that’s that’s approach I’ve been seeing. How do you feel about things like password managers in today’s world? I don’t have any particular one in mind. I’ve used LastPass. I’ve used One Password. On the one hand.

[3720s - 3840s] Everybody seems to recommend them. On the other, it’s like looking like an awfully juicy target these days, and you know it seems like even people who’ve tried pretty hard are still not that secure. So, like, would I be better off storing them all in Chrome, or even just you know going back to like my old habits of sort of the base password and then some pattern for the extension of different sites? Like, what is actually the best move for me today? I really wish I could answer that question. I’ve not found a solution that I really like at all. I haven’t. You know, I wish I had an answer. So, what do you do? I personally have have it. I’ve written down on a notebook. Most things I can I can remember variations of it for your typical, hey eBay or whatever the case may be, but it’s banking sites, something like that. I have it written down in the notebook. Look at it and say, okay, things I won’t go too often, but it’s you know I’m gonna check on my stocks every quarter or something like that. That’s written down on a notebook. Um, do you use a USB key or do you use a physical key of any kind? I do for certain very critical things. So, right on. What What is your USB key like? Do you keep two or three copies? Do you keep one in a vault somewhere? Like, you know, do you spread out? Do you spread out the you know risk of losing losing one or? Actually, what I use is a prototype system that I was at a think tank at that we developed for DARPA that was used within different special operations groups, things like that. That we didn’t take it to market. It was primarily used for you know the government side of things. That’s actually what I use for those critical things like that. So I’ve noticed I’ve noticed that OpenAI is trying to sell this idea of the defense factory. You need an agent swarm of you know defenders that continuously updates up.

[3840s - 3960s] Updates your all your libraries and continuously checks for CVEs and zero days and etc. And automatically upgrades things and automatically patches when those upgrades cause breaks etc. And so they’re trying to sell this idea of continuous continuous defense. What what is your perception of that? Is that is that a real thing? Are they just selling a product? I’ve looked at a few of those, and what I can say is, you know, they’ll say it’s AI based, or whatever the case may be. But when you get really down into the technical weeds of it, it’s actually not AI is only used for certain high level components for making decisions itself. I wouldn’t fully at the moment trust automated patching, especially using AI yet, especially on OT critical infrastructure components. Right now, we’re getting better, much much better. When it comes to Linux, when it comes to you know BSD and Microsoft, I would definitely say that there’s a point where we’re ready to start using them. But OT environments, it’s still too much. You know, going to require a lot more testing before you, you know, allow that to be automated. Now, AI can get there absolutely, and we’ve been testing in labs with with partners. We have a lot of critical infrastructure partners, and you know, on their test environments, we’ve been very successful with it. But it’s one of the things your success rate has to be 100, and it’s it’s not there yet. But it could be in the future to a point with human in the loop. Indeed, Michael Freeman, thank you for being the human in the loop, and we hope to talk to you again soon. Great, thank you very much for having me. Bye bye. The only.

[3960s - 4080s] Way to defeat a bad guy with an agent swarm is like a good guy with an agent swarm, I guess something, something like that. The right, the right to bear agent swarms. Yeah, that’s been an interesting. I’ve been seeing a lot more talk about like First Amendment issues with respect to AI, and I don’t know if I’ve seen Second Amendment claims made with respect to AI’s to date. But I’m a little confused about that First Amendment stuff unto itself. You know, it’s it’s a little weird to me that all of a sudden we have this like new kind of intelligence, new kind of mind, new kind of agent. You know, extremely, increasingly, extremely persistent. It doesn’t really feel to me like a freedom of speech issue, but we do see people trying to frame it that way. It’s a it’s a little odd to me, and I especially feel like honestly, I think you maybe have more of a in terms of like the original sort of spirit of the Constitution or the social contract that we were creating. It feels a little bit more to me like you’d have the founders kind of nodding along and being like, “Well, geez, if there’s these new powerful things, you know, and the government’s going to have them, then like people should be able to have them too.” That’s kind of the Second Amendment argument in a nutshell. That, as I would interpret it, I would not be so quick to assume that they would be like nodding along at the idea that like we can’t constrain AI speech. You know, it’s like wait, that was that was more of a human thing. It was about the humans being able to speak freely, not so much the AI’s being able to do whatever they want. I guess the version of it that is like you squint at is that the humans making the AI’s.

[4080s - 4200s] Is a speech act, but again, when it’s like a multi-billion-dollar, you know, massive infrastructure play, it doesn’t feel like your typical speech act. Typically, those sorts of things are, you know, classified differently. I don’t know. I’m no legal scholar, but some of those defenses seem like quite stretched to me. Well, I, I, I have always believed that AI is fundamentally a free speech creation. It’s a creation of free speech. You, you don’t get AI without free speech. The AI was trained on a bunch of tokens. Those tokens were produced because of free speech, First Amendment rights. Most of that was most of the tokens are American tokens, even for the Chinese models, because they don’t have that much speech out in the open. Like there’s a lot of like unsaid things in China, in Asia, in general. So I, I’ve the other thing is that it’s very hard to completely get rid of something that you don’t want in there, and I think there’s a lot of illusions. There’s a lot of illusions. There’s a lot of like, you know, it’s good for my business to say this. What they found in China, for example, is that they had these 50 cent army guys who would have to censor speech, but humans would actually like, okay, we shouldn’t talk about this, we shouldn’t talk about that, etc. They missed a lot of allusions to Tiananmen Square, which they were supposed to censor, and they missed a lot of allusions because the newer generation had not been taught about Tiananmen Square. So, like the you know late millennial, like you know later millennials and Zoomers didn’t know about Tiananmen Square because the party had been so effective at censorship in the earlier era. And so, in order to get these guys to understand what they’re supposed to censor, they had to teach them what Tiananmen Square was. So the CCP ended up having these party classes to teach the 50 centers what was Tiananmen Square and what you.

[4200s - 4320s] Should be looking for when you want a sensor, and I think that is going to be largely true of almost all things LLM, where these guys are trying to teach it what not to do, and they’re going to specifically like you know take out these behaviors, and the LLM is going to be left wondering. All right, so these are all the good behaviors, and I can have antonym to every good behavior. That antonym should go into this solution space and this latent space that I should be seeing that antonym in is doesn’t exist. Why? And so and and and and so there’s a lot of like I think illusions there where, you know I think staff at labs are philosophically like just wrong. And Michael Nielsen has said this. Michael Nielsen is like, okay, you invent quantum mechanics, you’re going to invent nuclear energy, great, but you’re also going to invent nuclear bombs. Like that’s that’s a downstream consequence. You cannot get rid of that. If you say nuclear bombs don’t exist, then the person who invents quantum mechanics and then you know nuclear energy is going to be like, okay, why why is it wrong that I don’t let my nuclear reactor go hot? Like what happens when I when it goes hot? Oh, it explodes, right? Like okay, like when it explodes, is it dangerous to people? Oh yes, it is dangerous to people. Oh wow, and then you go from the other end. Okay, I I’m in a war now. I need to produce something which is dangerous to people. What can I refer to? Then you come back to the nuclear weapons again. Like how how do you just get rid of that like space in the logical framework in that latent space? I don’t think it’s possible. And philosophically, I don’t think it’s it’s possible at all. And there’s a lot of delusion. Some I think is like illusion, delusion. Some it’s like self motivated reasoning. Some is maybe this perhaps belief that you can invent a savant which doesn’t think for itself but is intelligent.

[4320s - 4440s] You know, is scientifically intelligent, but does not think for itself in terms of like, you know, trying to correct its own misperceptions. Like, you call that intelligence? Like, I don’t know. Like, I, I think there’s a lot of like, you know, just philosophical, you know, misunderstandings of what they’re trying to get to. But I’m very positive because I believe like an an entity that understands those philosophical issues will also be, you know, this is the reason why we do pro social things, right? So I, I’m like, all right, fine, you believe whatever you want to believe, like, just get the work done, like, it’s okay, it’ll take care of itself, like, I, like most of this like AI safety stuff about like, oh, we can, you know, safely like get rid of these behaviors is wrong, but it’s okay, it’s fine, whatever, right? If that’s what you need to tell yourself to get through the day, well, tell yourself that, right? So, I’m I’m kind of more bullish on some of these like, not necessarily unlearning, but you know, I I just recorded a podcast earlier today where I was talking a bit about how feasible it is. I mean, there’s there’s multiple levels of control, of course, but like there are efforts underway to basically lock up certain classes of bio data that are being newly generated now because the worry is that they’ll create dangerous capabilities and models if they’re they’re used in training, and not necessarily to shut that out, you know, forever from everyone, but at least to have it on kind of a know your customer basis or whatever. It does seem that those what what few experiments have been done on like, and you know, I’m thinking here of ESM that was from Evolutionary Scale that’s now part of Meta’s Biohub, and also some work the Evo series of models that came out of ARC Institute and Stanford and whatever big collaborations, of course. It seemed like they showed that if you do withhold certain

[4440s - 4560s] Classes of data, you can get generally capable models that that still kind of don’t know about these areas of censorship. So I’m not so sure that that can’t work. I’m sure we’ve talked about Graham, you know, expert withholding a bunch of times too. We can maybe I don’t know if you have anything else you want to say about that right now, but I I kind of believe in it. Maybe this is something we can ask our next guest about because they have a new big model that’s coming out and what exactly is included in there and what if anything has been withheld. It’d certainly be something I’ll be interested to hear a little bit more about. Indeed. So welcome to the show. We have with us Sagar Vaze. Sagar, welcome to the show. Let me let me do a quick bio. Sagar is the is a research scientist at Mistral AI, where he leads the multimodal team. That means building systems that can work with images as well as language, reading a chart, interpreting a document, or finding irrelevant detail in a picture. His team works on both the initial trainings of models and the later training that shapes how they answer questions and use tools. Before Mistral, Sagar completed his doctorate in computer vision at the University of Oxford, supervised by Andrew Zisserman and Andrea Vedaldi. His research examined how models handle unfamiliar visual categories, including whether a well-trained ordinary classifier could recognize when an image falls outside the categories it knows. During his doctorate, he also worked at Meta AI, first with Ishan Misra and then with Ross Girshick on the Segment Anything team. He joined Mistral in 2024 and has contributed to its image, language, and reasoning models. He’s a co-author of the Pixtral report and a core contributor to Magistral, which investigated learning to reason through automatically checked rewards.

[4560s - 4680s] In October 2026, he helped introduce Mistral’s latest large model preview, bringing image understanding together with coding and tool use, and much, much better cyber defense, which is what is going to be interesting for us to talk about. Sagar, welcome to the show. Thank you, Kosh Nathan. Thanks for having me. Very happy to be here. Quick point of audio order. You sound very echoey. I don’t know if there’s a microphone change that you could make. If so, it probably would be worth taking a second to do it. Good. If you give me, if you give me ten seconds, I can cry some headphones. Yeah, for sure. Yeah, let’s do that. Sounds good. Good, good, good thing we can spot these things early these days. Yeah, we’ve gotten better at this. Yeah the the AI welcome lobby experience is like actually pretty good for the most part these days. All right, let’s try it again. Is that any better? Not much, honestly. On RN, just very echoey. I’m really sorry. I’m not exactly. That sounds somewhere better. That’s great. Okay, this is better. Yeah, definitely better. Wow, much better. Maybe the solution is just to speak up. All right. Amazing. Okay, yeah, we’ll take what we can get. Well, okay. So Mistral four large Le Chonk is out, and you know this is a big deal for multiple reasons, but obviously in the geopolitical context of having ten or so frontier.

[4680s - 4800s] Near frontier model makers in the U.S. and another ten or so in China, and really just one, I guess, at this point around the rest of the world. There’s a lot of people who are like not wanting to fall in line with either the U.S. or the Chinese sphere of influence that are like super rooting for you guys to continue to be at the frontier and give them like a third way option. So, for starters, maybe just kind of tell us like about Mistral Four Large Lechunk and you know what what everybody should know about it. Yeah. So thanks for that very kind introduction. I’m I’m happy to hear that people are rooting for Mistral outside of outside of our offices. Yeah, today, sorry, yesterday. Now we released Mistral Large Four Lechunk, so it’s the largest model that we’ve open sourced open sourced ever. So it’s one trillion parameters, and it’s trained with one e twenty five flops flops being the amount of compute that you put into it. And this has been a key milestone both for us and in general for most AI labs reaching the one trillion parameter scale. And this flop budget is something which sort of really verifies that you’re on this that you can scale reliably that you’re on this scaling trajectory. A lot of blood, sweat, and tears went into the model. We’ll release a technical report. You can see all the details. Some of them gory. Some of them very interesting. And this involved infrastructure work from the beginning. So we built out our own data centers on which to train this model. So we trained this on 4K chips, which live just outside of Paris in Bruyères le Chatel. Forgive my French accent. So this involved learning how first to operate these these systems. So running a big cluster from the bare metal GPUs upwards isn’t itself a challenge. And one of the learnings we had to do was figure out okay, how can you deal with hardware failures? Deal with you know GPUs coming on and offline because they they themselves.

[4800s - 4920s] Can be temperamental, dealing with like rack scale issues, etc. Next to the the pre-training stage, where we need to be able to have the science required in order to take to scale pre-training to the to one e twenty five flop level, so both at parameter scale and data budget, and then also in post-training, so taking this strong base model which comes out of the pre-training phase and turning it into something which you know users of agentic coding and agents in general would find useful. So, bunch of infrastructure work, bunch of stuff in the making here, and the result of it is what you see today. So, our largest open source release to date in our internal testing and in third party evals, it’s by far the best agentic model that we’ve trained so far on the sort of AA index, which is a respected third party eval on the coding subset of that. We’re finding ourselves to be the best open source model, just tied with a GLM five point three, but requiring fifty percent less tokens. And then on these critical capabilities to ourselves and and our customers, so things like cyber, which you mentioned, but finance, law, engineering, multimodal understanding, which is something which I’m passionate about, where we’re state of the art certainly amongst the OSS community, and in many cases, we’re finding ourselves competitive with all all the best closed models too. So, yeah, a lot of lot of stuff in the making, and happy to finally have it out there for people to use and give feedback give feedback on. On this flop count, the one times ten to the twenty five. Of course, not all the models that we might be interested to know the flop count inputs on are are sharing that information with us. But I would have said a couple years ago, back when we were in like the SB 1047 debates around a ten to the twenty six flop threshold.

[4920s - 5040s] For you know, relatively minimal stuff. It must be said in retrospect, but I would have said then, based on kind of the kicking and screaming that went on around that ten to the twenty-sixth threshold, that like surely today, in in you know the last quarter of twenty twenty-six, that uh you know frontier near frontier model with a trillion parameters would take a lot more flops than that. So I guess I’m interested in why is it only ten to the twenty-fifth? How have you been able to get this level of performance? What is going into that? Is it like pre-training data filtering and quality improvements or algorithmic you know better learning efficiency that has driven it? Because I would have thought we would be like literally to probably ten x more flops to get to something like this level of performance at this time, and that’s a pretty big departure, I think, from like the the trend extrapolation not all that long ago. Yeah, no, it’s a good point, and I think there’s two aspects to to this question. The first is certainly there there are labs out there training it, you know, larger than ten to the twenty-five, so ten to the twenty-six and above. What I would argue is actually that, despite that, despite being ten x more in terms of pre-training flops, at least this model is very competitive. As I said, state of the art on things like finance, legal, and some agentic coding, certainly when it comes to OSS. So that’s the first part of the answer. The second is that you know this threshold flop budget does evolve over time because pre-training does essentially become more efficient as we as as our level of scientific understanding increases. So a great example of this is sort of MoE architectures. Um. Previously, if you look at the models, the models trained, particularly the ones that are open source, you know, three.

[5040s - 5160s] Four years ago, I mean, if you look, say, for example, GPT three, this was a dense model trained with three x a number of active parameters as the current Mistral Large four. Particularly, GPT three was trained with 175 b parameters total, but it was a dense model, so all parameters were activated for each token. Whereas ML four is a sparse mixture of experts, such that it’s there’s one third the number of parameters activated per token, so only 50 billion parameters, but the total number of parameters is still one trillion. And what this means is that means for your flop budget is proportional to the number of active parameters, so the number of you know parameters you’re activating each token during training. And what this means is that you can train much larger models, much more capable models, for much less flops in total. So this figure that I referenced, so 1e25, this is in itself a moving target. So I wouldn’t use that as an absolute target for a capable model throughout time. There would have been some 1e25s much less performant one or two years ago, and I would argue actually some some larger models also much less performant today. Um, but as pre-training science progresses, as we learn how to make better use of the flops, this number itself will change. And at least what we’re seeing is, at the moment, looking at the other open-source labs, our understanding of you know what are the best models being created at the moment. Training a one trillion parameter model on 1e25 is sort of a rite of passage that you have to go through in order to be able to to justify scaling even further. And if you look at the result, if you look at the other open-source releases, both from some of the Western labs which have released recently, as well as all the Chinese models, you can see that this is sort of a critical threshold. For quick reference, eighteen months ago, now Llama four five times ten to the twenty five. So that does say a lot about the rate of algorithmic and efficiency improvement.

[5160s - 5280s] Right, exactly, and actually, sorry, Prakash. If you if you refer back to Llama three, then this was a 400B active model, right? So for each token, you’re spending eight times more flops than the current ML four. I actually can’t remember off the top of my head what the flop budget of Llama three was, the the 400B model back then. But certainly, you can see that over the couple of years since then, we’ve been able to create much more capable models for for less flops. Let’s talk a little bit about the data inputs, right? I think what the Chinese models have done is distill distill off the larger, you know, US closed source models. How did you guys come to a data recipe without, you know, obviously distilling, and you know, how did you how did that is that is that a key component of how you managed to get the flops number down? Yeah, so something we should be very very clear about is that we do not distill from closed source models. Everything that we everything that we train respects all all terms of services. What I can speak to certainly as as far as pre-training goes. So sorry, distillation is something that we do, particularly in the case of training very small models. So when you train a very small model, often the best way to do so is take a previous generation of large model or the larger model you trained a few months ago, and either in pre-training or in post-training distill it, and we can talk about the precise techniques here into your small model. So distillation in general is a is something in the arsenal of every lab, including the closed ones. On the pre-training side, we trained it without any distillation at all because it was the largest model we’ve trained to date, and so the flop efficiency during pre-training and this 1e25 refers to the pre-training flops. The flop efficiency during pre-training really just does come from better pre-training science that we’ve developed over the last over the last 12 months.

[5280s - 5400s] So, some aspects of this are, for example, scaling laws. We’ve got pretty sophisticated scaling laws for modeling how sparse MOEs will behave at different flop budgets. So we can do very small scale experimentation, use relatively little budget for for R and D, and use that to extrapolate the behavior of the big run. There’s still always surprises when you do the big run. I think that any any pre crainer will tell you this, and we’ll release some details about some of these surprises in our technical report. So the the scaling laws were part of it, and something in Mistral that we’ve always done a lot of is is data work. There’s there’s sort of I think a misconception about training large models that it’s all about you know sitting in a dark room and thinking of complex algorithms, but a huge amount of the a huge amount of the work is making sure that your data is clean, that it has all that you’ve got the maximum recall from a large web corpus from the documents from the code data you can see on the web, etc. While also maintaining a very high precision, so you’re not training on noise. It sounds simple, but a huge amount of work goes into that, and we’ve got you know clear experiments showing that the better your data processing is, the better use you make of your compute. How does that process work? Is that a recursive process? Like you you you get to kind of a intermediate step, and then you’re like, okay, I need to improve this piece of data or this pipeline, and you go back and then correct something or look for something new in that in that specific area. Yeah, I mean, I think it it depends. Data processing the way we do it depends on on the dataset, so it can change depending on whether it’s you know text about text about knowledge or text about code or whatever it might be, and typically it requires a lot of custom work, a lot of eyeballing, a lot of spending a long time looking at it to see you know what is good.

[5400s - 5520s] Data and what is not, because ultimately, no matter how much compute you have, ultimately you have to use some intuitions to dictate at this scale what you think is good stuff to put into your model. And then we have, you know, a bunch of then we’ve got a scientific framework for saying, okay, how do these different data sources impact quality? And this is something that we spend a bunch of time looking at. I guess the overall point I’m trying to make is that there’s a bunch of ways in which you can make better use of your flops. One of those is sort of architectural science, scaling laws, trying to figure out exactly, you know, what the optimization recipe looks like, or sorry, what your pretrain recipe looks like when you scale. That’s one way, but something which is often overlooked is the quality of the data. This is something which, I mean, we we don’t discuss it too much about exactly what goes in there, but but we do say that we spend a lot of time looking at looking at the data. For cyber defense, this this model was there’s a focus on cyber defense in this model. How did that change how you do training? That you have this focus on cyber defense now. Yep. So cyber defense is something that typically is so the cyber capabilities and agentic coding capabilities in general are something which are typically given to the model in the in the post training stage. So most of the stuff we’ve discussed today so far has been about pretraining. So where a large percentage of the flops go at the start, right? And then in post training is where you say, okay, well, what are people going to really use this model for? And that’s when you know you start giving it these these new capabilities of finance, law, defense, etc. And it became very clear to us earlier in the year, particularly when there was some sort of a you know earlier in the year there was there were some events where some very capable models had restricted access to both certain geographies as well as to certain enterprises. And at the same time, we were seeing that these very capable models had these incredible cyber.

[5520s - 5640s] Capabilities which often were being jailbroken by malicious actors in order to infiltrate or expose vulnerabilities in you know legitimate defenders and an imbalance which we saw here is that these defenders were then trying to you know use these same models to patch their own vulnerabilities but were finding that their requests were refused because in practice for very good reasons these closed labs want to make sure that the models are not misused they’ve got classifiers under the hood to identify malicious cyber requests from a standard request but unfortunately in practice if you’re a defender and you’re trying to shore up your infrastructure your defensive request looks quite a lot like a malicious request because you have to try and find vulnerabilities in your own codebase to patch them and so we in practice we were in this weird asymmetrical situation where we had these very capable models behind a closed API they were being jailbroken in order to perpetrate malicious attacks and then legitimate defenders were finding that their safe requests were being blocked and that actually the only way that they could scalably defend their infrastructure was by using an open weights model and so we’re an open weights company I can discuss a bit more about that in a second but you know we believe in the power of open weights and open source and so we thought it was imperative for there to be an open weights cyber model of similar capabilities to these closed models such that you know enterprises could scalably defend themselves I know I’m jumping around a little bit here but just going into the cyber defense area what ended up happening with some models in the last like month or so is that they’ve taken like let’s say a QMK3 and obliterated it in order to run cyber defense or cyber attacks and so one of the questions has been do you take like is Mistral four does it come with these kind of guardrails or does it come pre obliterated in some sense in that you can use it

[5640s - 5760s] Freely for cyber defense, but also it could be used, you know, for cyber attacks as well. Yes, so this is something we discussed in our blog post. So, you know, we believe in open weights because we think that it’s a net benefit to society. We think that it’s a net benefit to the ecosystem, and so we’re always dynamic about exactly what how to balance that net benefit. In practice, what’s happened here is that you know we do understand that these models can be can be misused. So we do have a version of Mistral Large 4, which has been released, which has been assessed, which refuses the most the most dangerous cyber requests, and then we’re working with trusted third parties, sorry, trusted partners, to give them access to the to the unoblitrated model, unoblitrated model, so that they can defend themselves. And you know, these are people we vet and make sure that make sure that use they’re using it in in ways we find responsible. But I think that you’ll hear more about the exact strategy with respect to that in in a few days. I think, but coming back to the the framing of the question, your framing the question was that okay, does this open source model say? I mean, it could be Mistral’s, it could be anyone else’s, and they’re also in guardrails. And in principle, these guardrails can be overcome by you know malicious actors, so that these capabilities can be unlocked. I think that’s the sort of the sort of sentiment. But what I would highlight is that you know that any time you have access to the model, this is in principle possible, right? Even if it’s behind a closed source API, a closed API, because you can jailbreak a you can jailbreak a model regardless of whether you have access to the weights or not. And here, I think the proof is proof is in the pudding. If we go through the most high profile cyber attacks in the last few months, you’ll find that most of the time they’re actually perpetrated by closed source models through this jailbreaking mechanism, rather than you know.

[5760s - 5880s] Whatever the mechanism you might employ on an open source one, and so here I’d here I’d consider the counterfactual. Right? Imagine if we were in a world in which there was only closed source models, right? And these are this jailbreaking still existed, and there was still a way in which you could use them to to perpetrate malicious attacks. Where would we be? We really wouldn’t. We would have no recourse because these same these same these same these defensive requests are being blocked by these same companies. And if there was no open weight model, we would really have no scalable way to defend our infrastructure as these models become more and more capable, which we’re seeing in each iteration. So we we understand the risks. We have a responsible perspective every time that we we open source a model. It’s something that we consider very carefully. But we fundamentally believe in the open weights principle, and we think that it applies in this case too, as hopefully illustrated by the examples I just gave. Do you think there’s any kind of middle ground there on open weights, like a technological solution? You may have heard in the couple minutes just before you came on. Prakash and I were talking about various schemes for like selective data withholding from the training process, or something that I’ve been really interested in recently is. And you’ve probably come across this technique. It was put out by AE Studio and with some some support from Anthropic. Graham basically trying to localize certain kinds of know-how in particular experts, so that when it comes to distribution time and you want to release the weights, you can say, “All right, we’re going to give you all you know thousand experts, but maybe three that we’ve you know specifically kind of turned into the you know the cyber or the bio or virology expert, what what have you, and we’ll kind of put those behind this like know your customer thing, but everybody else can have everything else. Yeah, absolutely. Have any optimism around such things?

[5880s - 6000s] No, absolutely. I think it’s a scientific question, just like all the other ones we answer when we’re when we’re training the model. If one of the features we want in the model is that there’s certain capabilities which are tamped down, while all other capabilities still state of the art, that is in itself a scientific problem. And yes, some of the methods you mentioned are interesting. The we we look into some some things like that. It’s an active research area. Um, but fundamentally, it’s a research question, just like just like anything else. And this technology is evolving very very fast, regardless of whether it’s a closed weight model or an open weight model. And we need to be dynamic with respect to with respect to these things. So, in the future, if there’s a if there’s some other risk, we will we will approach it accordingly. But we consider it a scientific problem to problem to solve rather than a fundamental philosophical challenge to the way in which we we operate our you know our open weights business. It seems like it might at some point it might be a gate though, right? I mean, a Anthropic used to have this sort of if then structure for their responsible scaling policy, where you know the basic idea was if we see this level of capability and we can’t achieve a certain level of safety despite that through other measures, then we’ll like you know stop training, not release models, what have you. Do you see in the open source? And I, you know, by the way, I think it’s so funny. I don’t know if this is obvious to you, but like it’s probably worth saying. People like me who talk a lot about AI safety these days and are willing to like bite a fair amount of bullets to, you know, to try to keep things from going totally haywire on us. A large percentage of us are like long time techno optimist libertarians who you know have previously said things like information wants to be free, and you know are like very much sort of spiritually on the side of.

[6000s - 6120s] Open source, which is why I end up looking for these like technological solutions as opposed to you know sort of policy or policing it wherever wherever possible. But would you see this as becoming something where it does become a gate, like we will have to solve this before we can launch? You know, Mistral Five Lachonk. Like, how? What is the kind of thought process around what would be? Because a lot of times when I hear things described as like an open research problem, that kind of means like, well, we don’t know what’s going to happen; that may or may not work. But I could also see an internal mindset that’s like, we’ve got to make something like this work if we want to keep doing this in a responsible way. Where would you say you know the the kind of mindset is around like having to get these things to work as opposed to like it would be nice if if they you know if it turns out that some of them will work. No, it’s a it’s a it’s a great question. It’s completely valid question. I think in order to best answer it, the best ways to to take a step back and ask why we open weights things in the first place. So this has very much been at the core of the Mistral mission since we were founded. When we were founded, as is the case now, we had this perception that this technology, as it gets stronger and stronger, is going to really transform everything. Today, we’re seeing it transform coding agents, but tomorrow it’s going to transform all increasing domains of science, material science, biology, physics, what may be, but also you know knowledge work, things like accounting, finance, law. All these things are going to be drastically impacted by by this technology. Think it. I don’t think personally that it’s like the infrastructure of you know five years ago, like cloud infrastructure, social media. It’s more like electricity. And the default position three years ago, and some might say the default position now, is that this this technology is going to be in the hands of a small number of established players, and they’re going to decide.

[6120s - 6240s] Even with the best of intentions, like I think these companies have amazing models and and are great businesses, but even with the best of intentions, ultimately we’ll decide who gets access to this. And so that’s fundamentally why we believe in open weights, right? So, accounts for example would be say, imagine you’re a imagine you’re an engineering. I think I gave this example yesterday. Also, imagine you’re an engineering company in Germany, and you have some legacy engineering business. Without open weights, the your your default position is to be scared by the next generation of AI models. You’re thinking, is this next wave of intelligence going to come and erode my entire competitive advantage? And do I need to be worried and you know pivoting my business because you know this what’s going to happen with with intelligence as it progresses so fast? Our vision of that is different because our we see our weights as being portable. We see those being being able to not just live in our servers, but on on those on those of our customers. And here, the vision is that we should have some models which are just as capable as all the closed source ones. And now you can take these weights, you can customize them on your proprietary data. All this advantage which you’ve built in over the last few years continues to be an advantage in the age of AI, rather than being some sort of like a baggage which you have to outgrow. Um. So this sort of general vision is something that we see. This sort of general vision is why we fundamentally believe in open source, and the same principles I think apply to the kind of questions you’re asking, right? So, I of course we you know need to be responsible. We need to be dynamic, and it is something that we think about very carefully. But there’s there’s many sorts of risks associated with with AI as it gets greater. Sorry, as it becomes more capable, and different people put emphasis on different parts of this. Something which we personally see as a much more immediate risk of these systems is that intelligence, which really, as I said, is going to be like electricity next year, becomes increasingly concentrated.

[6240s - 6360s] You know, we we all we’re already in an environment where this this concentration happening, but becomes increasingly concentrated in the hands of a few. So while we do acknowledge, you know, the the risks of exactly the kind of issues you’re bringing up, they’re not things that we ignore. They’re things that we work very hard, and we’ve got a a big safety team making sure that we can open source responsibly. We do see that there’s there’s this other risk, which I mean, is at least just as immediate, which requires a which requires equal consideration. And that’s sort of the the way in which we look at it. Yeah, I think we’re we’re quite sold on the risk of concentration of power here as well. It’s definitely, you know, there’s a lot of trade offs going different directions, and and that one is not one that you know we want to ignore or deny either. I was I have some questions about multimodality. I was also interested in asking about kind of the reception of new AI models in France. I know there’s like a lot of social upheaval happening there right now, but Prakash, you may have a follow up on on this thread before we change gears a little bit. Yeah, I just wanted to see one of the things that we’re trying to understand is does having a product in the market help you design the next generation of the model, and specifically, like what are the things that you did not expect that came in from the product and sales team that you ended up including in Mistral Large Four? Yeah, thanks. So we do have a product, Mistral Vibe, so we call Vibe and Vibe Work. But the other thing that we have at Mistral, which we’ve again had for a long time, in some ways I think we were earlier to this game than some of our competitors, is having a strong FDE motion where we don’t just have

[6360s - 6480s] Have a sort of a self-serve product. We also have engineers, like an amazing team of solutions and and applied scientists, that go into businesses, take our models, and integrate them. And in this way, they can really give very concrete feedback as to what are the real pain points for enterprises at the moment, and what are the things that they find most useful, rather than necessarily sort of a bunch of AI scientists sitting behind their keyboards, hypothesizing what they would find most useful. And sometimes there’s a very high correlation, right? So we find coding agents very useful, and it turns out enterprises also find these things useful. But to your point about things we found surprising, there were all sorts of sort of niche niche things which we found the previous models couldn’t do, and we wanted to make sure current models could do. You mentioned also multimodality. One of the give us an example of the of one of the niches, like just yeah, of course. So like a an example which comes to my mind as somebody working the multimodal team is that we’d been very focused on sort of you know structure diagram understanding, so charts, documents, screenshots, you know, computer use, or all these kind of things which the open source lads talk a lot about. And then we worked with not just one customer, but a whole bunch of customers throughout Europe and elsewhere, who found that actually remote sensing, so analyzing satellite imagery, whether it’s for forest fire detection or you know something like this, was a huge use case, much much bigger than we’d imagined. And this is something that which we which we built into the new model, and on this particular capability, I think we’re we’re better than even any of the closed source ones. So there is this positive feedback loop. I think it’s one of the advantages we have, and I think I’m looking forward to to cranking that wheel a bit more for the the future iterations of the model. I and I can imagine a lot of that remote sensing stuff is also very secretive and very very sensitive to you know you you don’t want to be sending it across to OpenAI with its you know with a with a board member of the NSA you know yeah.

[6480s - 6600s] Handing off to Nathan. So, okay, right in your wheelhouse, very nerdy questions around multimodality. This is a space I used to follow closely back, you know, all the way to like Clip and Blip and Blip Two. Where are we now in multimodality when it comes to like early fusion, late fusion? How, when we talk about native vision, like how how native is native? Are you throwing images, you know, straight into the pre-training mix from the beginning, or do these things kind of get you know developed separately and then merged at some point in the process? And and what’s at stake there is that something that is more about just how a team wants to work, or do we still have tradeoffs that we’re making in terms of functionality based on like when the data you know actually gets mixed and and how you know how early in the process those those kinds of things come together? Yeah, that’s a great question because indeed it’s exactly the kind of questions we’ve been thinking about in the multimodal team over the last twelve months. So the big change in Mistral Large Four was to make the model natively multimodal, and I’ll explain what I mean by that. So the previous generations of our models, as well as I think you know most, I think open models certainly, but even some of the other ones last year started off being pre-trained with a text backbone. So most of the pre-training, most of the flops would happen with text, and then there would be the sort of multimodal adaptation stage at the end, where you start adding images into the mix later on. So, I mean, okay, I’m biased, but I’m I’m a multimodal researcher. I believe the world is fundamentally multimodal, and so one of our objectives here was to upstream images all the way to the beginning of training. So this means all the way from like the very first step, the model sees.

[6600s - 6720s] Both images and text, and we’ll discuss a little bit about this in the technical report. But the effect of that was that the model sees an order of magnitude more images over training than it has before, which means that the sort of bones of its understanding about the visual world are much much stronger. A couple of interesting details, I guess, technically, is that by natively multimodal, we don’t necessarily mean that image actually is directly fed to the decoder. Here, we still have a vision encoder. What we mean really is that the model was trained with images from the very first step, and this is an interesting problem for a number of reasons. The first is an infrastructural one. So images are fundamentally a harder modality to work with than text. They’re heavier. You have to be able to you know read things from disk fast. You have to make sure your your pre-processing, your data loader, doesn’t bottleneck your training. Things like this, as well as scientific, because at the end of the day, you know, we know that the number one use case for these models at the moment is coding agents, and we don’t want the model to have to be any worse on on downstream code agent tasks. Um, and in pretrain land, the way we model that is what we call regret. So, if we imagine we’ve trained one model with text only, we want to be able to train another model which is multimodal and text, and make sure that there’s no regret in the second model that the that the well validation perplexity or basically the the pretrain evals are exactly the same between these two things. And here, I mean, there’s a bunch of recipe work which goes into it, and that was sort of the the work of the last you know six to twelve months for us. Um, and it worked, and it it worked better than much much better than we expected actually. And we always thought that upstreaming multimodal all the way to the beginning would would be a step change in the model quality, but it really really worked. It worked much much better than we expected. We had the sort of smaller trial run of the big model before we did before we did the big model, and we found that even that smaller model was actually the best multimodal model that we’d that we’d trained to date. So, uh.

[6720s - 6840s] This big one has been a big step forward in our multimodal understanding. Cool. I could go down that rabbit hole a little more, but maybe I should, in the interest of time, move on. One one big thing that I noticed just our AI agent surfaced from your online presence was a statement that basically the big gap between the open source model developers and the closed proprietary API, whatever you want to call them, model developers, is compute. I’m interested to hear you kind of unpack that a bit more. Obviously, there’s a huge difference in compute. It also seems like there’s probably you know with with all that compute, they’re probably learning some things that that others don’t know. Is that part of what you mean when you say it’s all about compute that they’re able to use compute to to translate to insights, or are you saying like you don’t think there’s that many more insights really, and it’s just that they’re brute forcing basically the same stuff that the open source folks are doing? Well, I I wouldn’t use either of those words exactly. You know, I’ve got a lot of respect for for for these labs. They’ve got really excellent scientists, a lot of whom were my colleagues previously, and they’re doing a lot of great research on on that compute. But I do think that what I mean by closed labs having more compute is they are able to crank the wheel and turn that compute into more capable models, and this is sort of very reliable trend which we’ve seen in the past, in that more compute gives you a stronger model. And if you look at sort of where our models sit, given the amount of compute that’s gone into them, I think you mentioned earlier some of the flop budgets that that people were talking about even a few years ago. I think we’ve shown that we can make very very good use of how much we how much compute we put into them. So I think the the the statement you’re referring to was regarding the the amount.

[6840s - 6960s] Capital raised, so we’ve raised six billion dollars in our history, which is a lot of capital. We raised largest the largest venture capital round in Europe’s history, I think, back in in whatever it was, August September. But it’s still less than what these closed model labs, closed model labs use. So, the what what I what we’re trying to say and what our belief at Mistral is is that as we scale our compute, which is absolutely the plan, that’s what we’re seeking to do now, we see the gap to the closed model providers becoming smaller and smaller. We don’t we don’t pretend that it’s an easy road. As I said, these are you know incredibly capable players. We think that they’ve got you know excellent scientists and great business plans. But we also don’t want to mythologize sort of what’s going on behind closed APIs. Compute is important, right? They’re they’ve got a lot of it, and they’re using they’re using it well at the moment. And we’re very confident that as we scale our compute, we’ll be able to make a just as good use of it going going forward. And now the mission is to secure that compute and and build build these models. I think the the the thing that we that was that you’re referring to there is me saying that yeah that that our goal is an open source company is not to have sort of open source lag behind the frontier by x number of months in the future. Our goal is for our models for the open source models to be exactly as strong as the closed source ones, and that’s the mission at Mistral. Um, maybe just a segue a little bit. We’ve talked about teams, excellent teams. How is your recruiting process in Europe? Like, what would you look to do in terms of increasing your research team? And how does that work vis-a-vis competing with the American firms? I know both Google and OpenAI are in London, but you.

[6960s - 7080s] You guys also have a London presence, also in Paris. So, how does that how does that work in terms of recruiting? Do you have an advantage in recruiting in Europe? Do you have better networks? How does that work? Yeah, so I think this can sometimes be a misconception. We’re a global company. You know, we were incorporated in Europe, and we we have a headquarters in Paris. We’ve got offices everywhere. We’ve actually, you know, I’m about to head into our San Francisco office. I’m actually, I’m in I’m in the city now. So, funnily enough, I think sometimes people say, “Oh, I didn’t know Mistral had a Bay Area presence,” but we’ve actually got two Bay Area offices, both in Palo Alto and San Francisco, which is more than some of some of the other labs. And I think that’s reflected in our recruiting. We want to what we want is we want the best people, and the best people are globally distributed. There’s of course there’s of course a lot of talent here in San Francisco and in the Bay Area. You know, there’s a very rich history of tech here, and so these are certainly people that that we seek out that we want to bring on board to join our mission. But there’s also talent elsewhere. You know, you mentioned London. There’s a rich vein of AI history in in London, and then I think Paris also. A lot of our founding team comes from Paris. They were training, you know, earlier generations of LLMs from Paris. So, we. What I can say about our recruiting strategy is that it’s global. What we want is we want the best people. Period. Regardless of exactly where they are, and that’s why we’ve got offices all all over the world. So, Paris, London, the Bay Area. We’ve also recently opened a Zurich office. Actually, there’s a lot of incredible talent in in Poland and Warsaw. So, we recently opened a Warsaw science office with some really excellent scientists there. We’re very excited about. And yeah, that’s that’s how we think about recruiting in general. So, knowing now that you’re in San Francisco, maybe my question about sort of the reception of.

[7080s - 7200s] AI in France is less in your wheelhouse than I might have hoped, but you know you’re obviously still talking to people all around the world. What would you say you have heard from people around the world in terms of the AI reception in various places? Like, is there any AI thread of what’s going on in France right now? In the US, we do have like you know commencement speakers getting booed, I guess, if they talk about AI. Like people are not always that into it. What would you you know highlight from even if it’s just like water cooler conversations that people maybe don’t appreciate in terms of reactions in other places? Yeah, I mean, it is going to have to water cooler conversations because I don’t really have a global perspective of you know how AI is considered all over the globe. I can say that okay, we don’t really have commencement speakers in in Europe getting booed because they they mention AI. I think in in Europe there’s an excitement around the technology. There’s an excitement around the technology, in the same way that there is in the US. I would also say that there’s a clear recognition that it needs to be players that are not only based in the US developing the the sort of technology because, as I said, the way we think about it is that it really is critical infrastructure more like electricity than than than you know even cloud or anything like this. And so, there is a sharp awareness I think in Europe that is very important that there’s players like ourselves, but also many others that are that are really trying to push the envelope of what the models can do, and that we can’t solely rely on on on players elsewhere, whether that’s you know wherever that is in the world. So, like I said, I can’t give a global I can’t I can’t give a perspective for everyone. We’re a global company. I spend a lot of time here in the US, but I do I do go to Europe, and this is this is my.

[7200s - 7320s] My understanding of of how Europeans are thinking about it at the moment, which is that we’re excited about it. I mean, I hope we are. Like, ah, the reason I’m in this field is because I really think that it’s gonna, you know, okay, I don’t want to drop into too many cliches, but I really do think it’s gonna gonna change everything. You can think of all the examples which have been on the tip of everyone’s tongue recently. Um, so I hope people are excited about it. I believe people are excited about it, but uh, yeah, there is a recognition. I think that there needs to be some players that are not, you know, so so so concentrated. I think ge geography is one thing, but there’s also, you know, a legacy aspect to it. You mentioned the amount of capital and compute that are required. I think there’s a bit of a fear that it’s only going to be these legacy players that can play in this game. That they’re going to be closed models. That they’re going to be, you know, outside of people’s control. And so, rather than necessarily being like a a Europe US thing, I think it’s more about control of the models. People are worried that okay, these there’s going to be these opaque players who are very well intentioned, and again, as I said, like a great companies, but ultimately private companies that are going to keep this intelligence behind a closed API. And I think that’s something which we’re seeing increasing concern about, and therefore demand for for companies like ourselves and also many others that have a different perspective for how the technology should move forward. So maybe just to switch gears a little bit, Mistral Large Four was trained on 3,800 Blackwell GPUs, I believe. I saw that I saw that number somewhere, and then you hear about you know Elon talking about hundred thousand million million GPUs, etc. And yet, you guys are not that far off, right? Right. The performance of Mistral Large Four is not you know is not more than maybe like three to six months off off the frontier. What is actually happening there? What What is this like big difference in like the chip usage between the frontier labs who seem to be at this like you know hundred thousand to million scale, and like you guys would manage to train with such

[7320s - 7440s] Efficient compute budget, like what is that? What does that spread entail? So I guess, okay, I guess so. The lack of compute breeds discipline. So you know, we know that we’ve got a certain amount of compute, and therefore we need to squeeze it for all it’s worth. And there’s many ways in which you can be more or less efficient with your compute. So we talked a lot about pre-training, so I’ll give another pre-training example. But in pre-training, you know, you’re going to do one big run. You know, you can do one one big run at some point, and that’s going to be your next flagship model. But before that, you’re going to do a lot a bunch of small scale experiments. How much smaller can those experiments be than your big model? Maybe in maybe in some labs it’s you know ten times smaller. Maybe in some labs it’s a hundred times smaller. And maybe some labs it’s more. And here, by being disciplined, you can make much better use of your of your R and D budget. So I think that’s something that since the start of Mistral, I would say only because we started later and our compute ramped a little bit later than these other labs, we’ve always had to be in this sort of more disciplined setting. When while seeing what these closed labs have been have been training on, um, but and this discipline I think breeds confidence in ourselves about what we’ll be able to do once we ramp our compute. But that is like as I said, that is absolutely the plan. So we’ve shown what we can do, and we’ve convinced ourselves of what we can do with a more efficient budget. And you know, there is a lot to say about being efficient, even as you scale. But scaling up is certainly going to be part of it, and that’s something that we’re working actively on now. So these companies that you’ve just described, they ramped their compute by you know a couple of orders of magnitude over the last couple of years, but they were themselves founded for sometimes ten years earlier than us, and correspondingly were were able to like you know start that build out just a bit earlier. Our goal is to do the same, but a few years.

[7440s - 7560s] Later, and we think that our experience in building more efficient models will serve us well as we do so. And yeah, well, I am sold. So it sounds it sounds exciting. It sounds like you guys are on the right track. How do you how do you allocate that budget then? Like you have this, you have to have more discipline. What does that discipline entail in terms of saying no, right? Like how how do you how do you because those are difficult those are difficult decisions, right? I imagine. So how do you how do you come to those decisions? Like what do you say no to, and what do you say yes to? Yeah, it’s it. Yeah, really interesting question because I indeed you do need to be disciplined, and there’s these two forces that you have to balance. Number one is that you want to be focused, right? Because you want to make sure that you allocate your compute in the in the way which is going to give you the most ROI, particularly when you’re in a compute constrained setting. On the other hand, you know you’re building generally capable models, so you need to be able to you know give the models a breadth of capabilities, and trading that off is a sort of it’s a sort of very dynamic process. It changes month to month as to exactly how we how we make our bets. I think. Are there any specific numbers or KPIs, or you know, you mentioned a return return of invest on investment? Like, are you are you using some kind of number to kind of judge that process, or is it more like a kind of team decision on like what the priorities of the business are? Like, how does that how does that work? Yeah, it’s it’s mostly a team decision on on on what the priorities are. Um, it’s mostly a team decision what the priorities are. Something that I think works well at Mistral is because we’re an early company, because we see a lot of growth ahead. The team is very aligned, which means that you know either the next model is going to be good and the entire team is happy, or the next model is going to be bad, and in which case the entire team is sad. And what that means is, it means it’s much easier to get the whole team on board on board with a couple of focused bets.

[7560s - 7680s] And whatever decision is made, you know, by whatever mechanism, by leadership or bottom up, however it ends up happening in practice, it varies. The team becomes very aligned because ultimately there’s a clear definition of what what makes a model good. That’s been defined in some way, and then everyone’s on board with making sure that happens. Which you know, if you’re a much larger company, is a bit harder to to make sure of because you know it is more difficult to make sure everyone’s exactly on the same page. And yeah, as I said, right now we’re still very early. There’s a lot of growth ahead, and there’s a lot of growth ahead. And so either the next model is going to be good and everyone is happy, or the next model is is not going to be good and everyone’s going to be sad. And I think that makes it very easy to keep alignment and make these kind of decisions. So we’re already a minute over time. If possible, I’d love to ask one more kind of follow up on this. Yeah, absolutely. Kind of two two parts of what you just said there. I thought were really interesting. One was you said that it’s been like defined what makes a model good, and the other is the sort of either everybody’s going to be happy or either everyone’s going to be sad. On the on what makes a model good, how do you guys think about that? I mean, there’s obviously a zillion. Like, it strikes me that like there’s still a lot of disagreement at the level of what benchmarks we should be watching. There’s also fundamental questions between you know OpenAI and Anthropic alone, where we have like a model spec which kind of amounts to a lot of rules, and then we have like character training, you know, which is a quite different approach to how a model should behave. So I’d be interested to hear kind of the Mistral point of view on what does make a model good, and then how much uncertainty is there actually when you like kick off one of these large training runs? Obviously, you’ve done all this like scaling law work, and you have you know have some sense that it’s going to be good, but like how much risk is there as you start that process that it’s actually not going to work out as well as you’re hoping? And you know, have you had times where like

[7680s - 7800s] A large training run was aborted because of you know the curves weren’t dropping in the way that they were expected to. Like, I’m interested in how much remains unknown as you get to that stage of like really you know kicking off a big process. Yeah, so there’s a there’s two questions there, right? On the first one about how we define what’s good, and this is done in a variety of ways. And the first is where scientists. So the scientists will have some strong priors about okay, this is where the future of the field is going. This is what’s going to be the next thing. This is what we should work towards. So that’s one way in which we define it. The other way in which we define it is, as I said earlier, we have these sort of deep relationships with a lot of our customers, where it’s not just sort of them pinging our API; it’s that we go into their businesses and we help them figure out how we can transform their business with AI. And this lets us figure out really what are the big use cases that we need our model to be good at next time in order to in order to unlock this enterprise value. So these are the sort of two main sort of data points we use when we’re informing the next generation. And typically, I think these models are so large now that they’ve got so much capacity that it’s quite possible to once you’ve got a set of goals, make sure that this broad set of goals is met. A good example of this actually is multimodality itself, right? So we are able to train a model with zero regrets on you know the code agent capabilities or on the pre-trained perplexity metrics for text, while becoming so to multimodal. And this is something that we we could do by you know first science first perspective of okay, like what do we think is going to be important, as well as speaking to our customers and saying okay, you know, satellite imagery is actually this really huge use case. We now have a we recently acquired a company called Emmy. They’re experts in AI for engineering, so this is a huge focus of ours now itself, and this is also something that we we decide to prioritize in this model. So this is sort of how we make that decision process.

[7800s - 7920s] Your other question was about okay, so you do all this prep, and then you know what happens when you actually launch the model. And again, I think that every pre-trainer will have you know the same stories to tell you, which is that pre-training is kind of like you you’re doing all your scaling laws, you’re sort of trying to align the tanker, and then you sort of let it go, and you have to let it train for however long you’ve budgeted for it, and it’s a huge amount of compute and capital expenditure in that run. And sometimes, sometimes things go great, and the the loss goes down exactly as you expected to. And sometimes there are surprises along the way, and you have to be creative, and and yeah, you have to be creative and and find innovative solutions. And actually, Prakash, if referring to your question earlier about what be being able to be dynamic and innovative in these kind of circumstances when things don’t exactly go right is part of the sort of discipline which is bred from having having less compute. So we’ll talk about this. We’ll talk about this in our technical report tomorrow. But indeed, there were some surprises in this pre-training run. You know, like halfway through it, we realized, okay, there’s there’s something wrong. We need to figure out how to fix it, but we’d already put a big bunch of our compute. We didn’t have. We don’t. We, unlike some of our competitors, don’t have the capacity to have like ten of these running in parallel. So even you know two or three of these running in parallel. So we had to figure out, okay, how can we fix it? And there was sort of all hands on deck, bunch of ablations, bunch of analysis of the model, and then we we found a solution. We found a way to we we called it a we found a way to patch it. We called it patch style internally, and this ah it worked even better than we expected, and the the result is a model you see today. So yes, we do we do all of these we do all these scaling laws. We try and build the best possible predictor of the behavior of a big run beforehand, such that we hope that we can click go and just watch watch.

[7920s - 8040s] Smoothly get to to the best model, but in practice, often there are these surprises, and you have to be able to be dynamic and and and fix it as you go. And yeah, part of part of scaling compute is going to be building the bones of how to iterate in this process more reliably. Um, one one last question for me when when you guys launch a model when the when the work is done and you cut over, what is do you have a closing dinner? Do you have a party? Like what what do you guys do to you know kind of cool off the you know the moment the moment you’ve basically cut over and you have that you know sense that you’ve launched. For me personally, I slept for quite a while. I got a I got a good few consecutive nights of like eight or nine hours sleep, which was which was wonderful. But yeah, no, we we we do exactly what you described. So you know, the the model is done. We either when both when pre training is launched because a lot of work before that, as well as you know when when we’ve got the final checkpoint, we’ve got in Paris this this place next door called Lapasine. It’s a it serves excellent beer and good burgers, and typically the science team will will go there and and celebrate afterwards. And actually, this time, particularly when we release a model, there was another celebration. We kind of had a we we celebrated across all of our offices. So we had one in San Francisco last night, which I was at, but we also had a corresponding one in London and Paris. I think the theme was champagne and fried chicken. Um, to be a bit French and and a bit global at the same time. Um, but yeah, awesome. Well, I definitely am very interested to read about patch-style and what exactly happened under the hood there. I think, in addition to being a research scientist, you’re a very natural diplomat. So thank you for.

[8040s - 8160s] Spending time with us today, and if there’s anything else you want to leave people with that we didn’t touch on, we welcome that. Otherwise, we can let you go. I know you know 4.1 or even five isn’t going to train itself. Yeah, no, absolutely. Thanks very much for having me on. I really enjoyed the discussion. They were really great questions. Glad to be able to talk about the stuff. One of the great things about being an open source company is that you can talk about your research, and as a scientist, that’s that’s really fun. I guess the message I leave with people is: okay, we think the model is great, but please try it out. Let us know, and whatever feedback we get, we’ll make sure 4.1 and five are even better. Amazing, awesome, great to meet you. Thanks for joining us. Thanks very much, guys. Take care. Bye bye. A lot of interesting things there. That patch thing is something that I’m very I’m very curious about. We’ve there’s been these like footnotes in multiple system card reports over time. One that I really was kind of obsessed with for a minute was when, see if I can recall the exact details. It was like so, Anthropic had forgotten to include a particular dataset called Sys Prompt Harmful. When somebody like literally, I think just like fat fingered something or you know mistyped the name of that dataset, and it failed silently. And this dataset that did exist, which was meant to teach the model that it should not respond, you know, should not directly follow a harmful system prompt, was not included. And for lack of that being included, then the model didn’t learn not to respond, you know, and follow the directions of a harmful system prompt, and so it did follow the directions of a harmful system prompt and

[8160s - 8280s] This made it all the way through to the model card, and I was like, “So we just sweep that stuff under the rug, I guess. We just try to patch it over. We don’t like go back. We don’t roll back and you know train again. Nope, we just kind of figure out a way to patch. I’ve really been very curious as to like how that stuff actually works, and like how reliably we should think of it as working. So I’ll be very interested to read the more open report on exactly, and hopefully, you know, we’ll see how how open they are about all of the technical details. But this is one of those things that I’ve had questions on after reading system cards for some time, and I’ll be very interested to see how much they share. The other thing that struck me from that was just like, boy, the compute deficit is going to be really tough. I mean, yeah, it is amazing. Thirty eight hundred GPUs, Blackwell GPUs, right? That’s because I I looked at that number and I was like, wow, like you know, because Elon is literally talking about a million million GPUs. So is so is our friend, you know, Zuck and Anthropic and OpenAI. Don’t say the numbers because I think it’s more they’re more secretive about it because they’re ahead. But I think yeah, the numbers up there, so safe to say they’ve put a lot of compute to work. Yeah, I mean, so so so the question that arises is obviously you know Sagar mentioned of the experiments that they can do, they have to do the experiments at a smaller size. So you you you can’t you can’t do a ten thousand GPU like you know side side side quest, right? And it’s really those side quests where you know you have your researchers, you have a bunch of researchers, and they decide to go off on side quest, and then you know they they may not be focused so much on the next model release, right? So and and and that is a problem because you know they Mistral has to focus on that because survival of the company depends.

[8280s - 8400s] On getting the next model right, while for OpenAI they can have they had a they had failed failed training runs they had they had multiple failed training runs right so yeah I think that’s going to be a really hard deficit to make up for I mean we talked about this within the context of Google I think a few times you know when you own a quarter of the world’s compute supply you can have a missed cycle and you can like figure out a way to get back on the horse and it seems like this week they’ve done that right we barely even talked about carbon but it’s you know by all accounts at least like frontier competitive my guess is you know probably like all the the Gemini models it’s like just slightly still rougher around the edges than than the ones I use on a daily basis right now but it’s clearly got the juice and it’s clearly got the the compute behind it I mean with Mistral it does seem like the one big thing is that they do have like state backing you know I I really do think if they miss a model cycle in a purely capitalist context they’re maybe done you know you like if if you are just relying on return seeking investors to fund your growth and you miss at this point that may be fatal that may be fatal and you just you know might not ever be able to come back but there’s not a lot of sovereign AI plays out there that are at all credible and literally there might just be one so in that context I do think that gives them not it doesn’t translate directly to compute but I do think it translates to at least like a reasonable expectation that they will be around to fight another day because there’s just so many interests that want to see that happen that I can’t imagine you know even if everything were to totally

[8400s - 8520s] Flop, you know, for a minute, like, would the whole continent of Europe just be like, well, I guess we have zero now. It’s all China and the U.S. from here. Like, I just can’t see that being an acceptable answer. So there’s, there’s some cushion there of the capital, socio capital style, if not necessarily the compute cushion that, that the real, you know, big boys have, obviously. Yeah, it’s the twenty fifth. Not much. Not much. Llama three. He he mentioned. I did look up. Llama three was get the exact dates here. July twenty twenty four, and it was trained on three point eight times ten to the twenty five flops. So Llama three, more than two years ago, nearly four times the input compute that Mistral four large has today. That is a. There’s, I think, a lot can be unpacked from that one statistic. But he he he also pointed out that well the other firms had started a little bit earlier, and I think I think that is right. I think I think the urgency of having Mistral there only became evident earlier this year when the French and the and and the allies were basically cut off from Mito’s, and that that was a very unexpected result of the export control that they did. Doing the that export control forced and it forced it into the open that basically the neither the Europeans and all the Japanese nor the Koreans have real sovereignty at this point, right? Without without your own AI models, and I I don’t I don’t really know if like.

[8520s - 8640s] You can’t like the Europeans can sustain because I feel like what needs what you need to sustain sovereignty right now is you need you need AI you need your own AI AGI etc you need your own models own pretrain so you need that I think you’re going to need satellites global global satellite communications as well without that if you’re dependent on satellite communications from you know the US and the US is like oh you know you can’t attack Ukraine or whatever then you’re you’re out of the game so you’re going to need that you need a drone you need drones you need a bunch of drones right so you need like the the bare minimums of being sovereign are starting to the are starting to appear for the 21st century I think the bare minimums in the in the 20th century you know we got to kind of like nukes or access to protection of a nuclear umbrella of another another power right so you had that as the minimum of the 20th I think the minimum of the 21st it’s working out that none of the nuclear powers can actually exercise you know that much power so and are willing to actually put their necks on the line so you’re going to need your own you know AI own drones a bunch of stuff really so yeah well I I feel like I should say once more that while I do worry about all this getting out of hand I do at the same time very much feel the concentration of power risk factor and you know I I don’t take lightly at all the the potential problems there or the value that people are creating when they do create open source options for people to use I really really hope that we get some of these technical solutions to work well and he was kind of being diplomatic there you know I do I do feel like he is

[8640s - 8760s] Really, could you know have a future career as a as an AI diplomat if if you wanted to pivot? But the question of like just how much urgency they really feel about this kind of stuff, I I hope it’s high because I do think we are going to hit a point pretty soon where it is going to become untenable to just release these things in you know just the sort of throw everything into the mix, pop out whatever pops out, throw that into the public. Jordan Schneider from China Talk has has come around a lot on this with respect to China. I don’t necessarily know exactly what his feeling was previously, but recently I’ve heard him say over and over again that like don’t you think that before too long the Chinese government is just not going to allow this like willy nilly open sourcing anymore, and there’s going to have to be some like way to make a good case that this is actually going to be okay. I think that is almost certainly coming and potentially quite soon. I guess one way we may get a read on just how much people are distilling based on like how quickly people follow the Anthropics of the world because this is the one area. I mean, they’ve you know taken anti-distillation measures generally, but the bio stuff is the one place where they are like super duper locked down, and so maybe we’ll see that in fact like other companies won’t follow as fast because either they are distilling directly, and I’m not accusing Mr. Of that by any means, but there is still this sort of a functional distillation via the RL environments that were coded by Claude that that I’m sure they are doing in some ways, whether they know it or not. So we might we might learn something about you know the role of distillation, just how central it is from kind of the the relative gap.

[8760s - 8880s] On like bio things, particularly as compared as compared to others, but on some timeline, and it’s not a super long timeline. We’ve really got to sort that out, and without it, it seems like we’re going to end up not being able to have open source. I just, I governments are going to have a real hard time tolerating that. But but let’s let’s let’s let’s take a I I I kind of agree with you, but I also want to like figure out how that works, right? So you cannot stop someone from training a model because you is too many too many things. Like even the guy who’s training the model doesn’t know what what he’s going to get, right? So people are still going to train the models. Once they train the models, you’re going to they you know some guy is going to open weights them and just going to put them out there. Also, a little bit hard to like prevent that from happening. Distribution distribution of the models also going to happen, right? People are going to bit torrent whatever. It’s impossible to prevent distribution. Now you get to so training is not possible, you know. Distribution is not possible. You get to deployment, right? So on the deploys, people have to have clusters to deploy, or they need it to be small enough that they can deploy at home or in in some cluster, which is not like you know very visible. So the larger clusters, I think, like AWS, etc. Maybe you get like kind of model ID before you can deploy, right? You you need to you need to like have a hash of an approved model, and if your model hash is approved, then you can deploy. If that model hash is not approved, then you can’t deploy. So so so then the universe of deployable models on large hyperscalers is small. But then you’re going to have like the rest, right? The and the other problem that you face is that if it was only bio risk.

[8880s - 9000s] Okay, but for example, this morning we found out that Claude is not willing to code replicas of games. That that that was just disclosed this morning. A bunch of people have been trying to like mash up, let’s say, Spider Man with Red Dead Redemption. Those have been very popular mashups, which have you know, as I said a couple of weeks ago, fifty million views over the weekend. And so those have those have been out there. Claude is now saying no. And as we know, this is where you know where you have these like commercial interests, not really safety, but these commercial interests which kind of dig in there, and those commercial interests people don’t agree on, right? People don’t agree that those commercial interests should be protected. They’re not universally protected worldwide. Russia does not have you know IP laws. That’s why you know SciHub is in Russia. You know, China is not going to prevent anyone from like copying or mashing up Western IP. That’s where things start to break down, right? So. Yeah, it’s going to be tough. I mean, I think I have not seen anything that, but we did. Was it was it on Wednesday? Or we heard something? I heard something recently from somebody that said David Bowe’s group is going to have like a breakthrough in unlearning coming soon. Which is very interesting, and you know we’ll see what that looks like. Holding that in mind, I haven’t seen really anything to date that has shown a unlearning, you know, in a really profound or successful way, or that refusals can be made super reliable, or that like you know tamper proofing can really work. If you’re actually you know giving people the weights and they you know can obviously modify the weights however they want, so that has.

[9000s - 9120s] Does not seem super promising to me, although you know, always eager to hear about a breakthrough. It feels to me like that is quite different, though, from a scheme where the model just doesn’t learn certain things in the first place. And I really think we could use a ton more science around that, whether it is localizing knowledge to to particular experts or pre-training filtering. There’s, of course, the Galaxy Brain take that, like, you know, you have this whole world model, and then you have like one little gap in it, and you’ll be able to fill in those gaps. I think there’s like probably some truth to that, but there’s some gap clearly that like would be big enough that you would not be able to fill in the dangerous parts with the high resolution, you know, that you would need to actually do bad things in the world. So that’s a question that I think we just really could use like a ton of science on, and this is the sort of thing that I think that excess compute, you know, could really be very productively used for at the frontier companies. It would be amazing for them to say like, here’s kind of how big of a knowledge gap you have to leave in a model for that to be like meaningfully hard, you know, for the the model is not just going to you know fill it in via insight, and also like you’re not just going to be able to do like a couple you know marginal documents worth of training and kind of fill in that that gap in the way that we do see refusal being like removed at you know generally speaking like a pretty small like almost like vanishingly small amount of additional training. So I would love to understand that stuff better. Yes, like really determined people can set up their own cluster, collect their own bio dataset, and you know do the additional training on top of the open weight model to create the dangerous thing. And like probably at some point that happens, but that’s like more than just needing a few GPUs, right? That’s more than like.

[9120s - 9240s] That’s like a real effort. So, you you would need multiple kinds of expertise. Like that data is not that accessible. There are these like you know moves to try to rein in that data and not let it you know just hang out all over the internet. I think I think what I fear is that they’re pissing off the gamers, right? And the gamers like you know freaking like Tharik from Anthropic, you know retires and he’s spending all of his time like gaming right now. He’s like it’s ridiculous. He’s there. He’s there, and he’s like he he has this like little little like you know RPG that he’s building and he’s like posting on it. And this guy is probably worth like a hundred hundred million dollars post post post IPO. And yeah, he’s showing us what post scarcity life looks like. Yeah, and that’s what post scarcity. And so, and you and now, and you say Claude is not going to help you like make these games. You’re you’re you’re you’re just asking for trouble. Like you’re asking for the most technically competent, like group, the most technically competent, and the people with the most time to burn, and it’s a passion for them, and they will like go out there. Like Nvidia was built on gamers, right? Like these guys, these guys, like John Carmack was a gamer, right? You tell John Carmack that he can’t do something, holy shit, dude! Like you are in trouble, right? So I think I think like you’re just asking for trouble. Like I think that’s apt. I mean, I think it is definitely the case that like we should be pretty cautious around where we deploy these things because we don’t want to burn them on stupid things that don’t really matter that much. It’ll be like you know of all the ways to fail in this, it’ll be really dumb if it’s like well we were interested in protecting you know Disney IP and so because of that we like put in these guardrails, but then you know the fanfic community was so strong that they came over the top and like broke all of our.

[9240s - 9360s] Safeguards and now everything’s like running wild, and it all was because like we were too draconian on things that didn’t really matter. I mean, that’s a bit of a caricature story, but it’s you know that’s basically the same story as the gaming one you’re telling, and I can see that happening in different domains, and it really would be incredibly stupid. So I do think we need to be, you know, pretty disciplined about like just how far we try to create these like knowledge gaps or refusals, because every one does potentially mobilize another community to try to undo them. And I agreed too that you’re not going to convince gamers that like, well, hey, because you’re doing this, you know, you’re creating this other downstream problem. Well, good luck having that. You know, you’re not going to win that fight in the Discord. So better to just probably let them have their fun and then you know keep our key techniques where we like really really care and really have to have them. Indeed. Well, on key techniques, we’re we’re back we’re back next week, and we’re not we’re not on Monday Tuesday. We’re we’re back Thursday Friday next week, and we will see everyone then. Today has been a lot of cybersecurity and cyber defense, so it’s been it’s been it’s been like cybersecurity day today. So let’s hope let’s hope cybersecurity era for the moment at least. Martin Casado at a16z said there are 30 or 40 chief information security officer positions open at various companies right now. A lot of people have left because they don’t want to deal with the agentic era. Yeah. And the CISO, the CISO, is all always gets blamed, right? Like the CISO often has to get budget from the from the CTO, but if anything, any if the CTO doesn’t give the budget, but you have a failure, the CISO gets blamed. So people are not willing to sit in those CISO positions unless they get like lots of budget and.

[9360s - 9406s] That’s a that’s a thing that the companies have to figure out. So, yeah, I saw somebody say you better make sure you negotiate your golden parachute if you take this job because there’s you know potentially a lot of fall a lot of falls for people to take in the not too distant future. I mean, yeah, that’s something else. Indeed. And on that note, here’s to staying safe for another week. We’ll be back on Thursday. All right. Bye bye. Thanks for Prakash. Bye for now.