As much as I hate everything about the rise of LLMs, saying this isn’t impressive because it can be matched by “an elite security researcher” isn’t very reassuring to me. It’s still an agent being pointed at a codebase and finding hundreds of vulnerabilities. Even if only a twentieth turn out to be exploitable in practice, that’s still a terrifying tool to imagine in the hands of hackers who might otherwise lack the skills to find these vulnerabilities.

Most hacking groups buy exploits off of dark markets and indiscriminately target servers until they find one that’s vulnerable. The number that can actually develop those hacks is far smaller, but if you can simply ask an LLM to find a vulnerability then that bar is lifted. Hell, you could probably coerce it into writing the actual exploit too by claiming you need a proof-of-concept for a CVE writeup.

This is the meat of the headline:

For example, the Anthropic-claimed 181 Firefox exploits ran with the browser sandbox turned off and the FreeBSD exploit transcript “shows substantial human guidance, not autonomy.”

Additionally, the “‘thousands of severe vulnerabilities’ extrapolates from 198 manually reviewed reports. The Linux kernel bug was found by Opus 4.6, the public model, not Mythos,” Devansh said.

Another researcher, Davi Ottenheimer, pointed out that the security section (Section 3, pages 47-53) of Anthropic’s 244-page documentation “contains no count of zero-days at all. With no CVE list, no CVSS distribution, no severity bucket, no disclosure timeline, no vendor-confirmed-novel table, no false-positive rate.”

Ottenheimer likens it to “the ending of the Wizard of Oz, a sorry disappointment about a model weaponizing two bugs that a different model found, in software the vendor had already patched, in a test environment with the browser sandbox and defense-in-depth mitigations stripped out.”

Marketing outstripping reality? Is that even allowed to happen? I hope nothing like that happens with the dick pills I just ordered.

Exactly everyone but AI bros saw that coming.

More analysis on Anthropics claims:

https://www.flyingpenguin.com/the-boy-that-cried-mythos-verification-is-collapsing-trust-in-anthropic/

no!

So far we’ve found no category or complexity of vulnerability that humans can find that this model can’t

This seems like a statement designed to deceive. I would like to see comparisons like the percent of vulnerabilities Mythos found that an expert also found (and somehow make sure those vulnerabilities weren’t leaked into the training data and the prompts don’t hint at the answer).

Better portrayal of Mozilla’s response from this ars technica article:

The vulnerabilities identified by Mythos could have also been discovered either by automated “fuzzing” techniques or by having an “elite security researcher” reason their way through the browser’s complex source code, Holley writes. But using Mythos eliminated the need to “concentrate many months of costly human effort to find a single bug” in many cases, Holley added.

The key part there is fuzzing. These tools have existed for a while. (and many are free!!!) Mythos just does what most AI tools do: Take something that requires more understanding or effort, and condense it into a prompt. Instead of starting a fuzzing tool, configuring its scope and some parameters, then letting it go hog wild for a bit, you just tell the AI model with a prompt to perform similar functions. (while costing more money and taking more time due to inevitable overhead from running a whole LLM)

If anything, this points more towards Mozilla not using existing fuzzing tools to find flaws in their code because they were too lazy, not that Mythos is magic and superior to all else.

It’s funny to think that Whiskey Pete may have cost the US military access to the most powerful cyber warfare tool in history all because he’s an incompetent drunkard and immature man-child. But I also don’t want to be giving these AI companies the benefit of the doubt, especially when it comes to marketing claims.