A Hacking Tool Built With A.I. Can Breach Phones Without a Click
An attack discovered by A.I. researchers could have compromised hundreds of millions of devices within hours, experts said.
A friend you haven’t heard from in a while suddenly calls. You don’t pick up, but in a matter of seconds, the damage is already done.
The call wasn’t actually from an old companion looking to reconnect, but from a hacker who had hijacked that number. Worse, the assailant now secretly has control over your account, including access to your private messages, and is already using your number to reach your saved contacts and compromise them, too.
Like a raging, highly contagious virus, the attack keeps spreading. Within hours, millions of people have suffered the same fate.
A year ago, such a cyberattack might have sounded far-fetched to even the most paranoid digital experts, or at least like something only the world’s most elite spy agencies could have pulled off.
An artificial intelligence researcher quit his job at Anthropic on Tuesday and accused the company, and its chief rival, OpenAI, of acting irresponsibly, igniting a frenzy of concern on social media about the rapid pace of the technology’s development.
Jacob Coxon, who has worked as a researcher at both companies, said in a post on X that he resigned out of concern that Anthropic and OpenAI are “gambling with our lives.” He said the people building AI “earnestly believe that it could kill us all by the end of the decade.”
From the OP article: “We have a big challenge in front of us,” Sam Altman, the chief executive of OpenAI, said last week at a Group of 20 nations meeting in North Carolina. “Some things are going to go very wrong with cybersecurity unless some people act quite urgently.” … When Anthropic’s new Mythos model was announced in April, the company said it had identified thousands of zero-day vulnerabilities “in every major operating system and every major web browser,” including many that were decades old."
The cyber world has gotten more dangerous. Why are the AI companies focused on finding vulnerabilities? We know vulnerabilities exist, but training the frontier models to find them is odd.
The OP article doesn’t give any advice on personal protection. The standard advice to keep your system software updated probably still applies, as the attacks and the defences both get stronger. I expect the attacks will win one day, and we will wake up to our phones and the internet not working. So keep a few paper books around! A local high school had to close for a week because of a cyber attack. Unclear why the school could not operate without computers.
Cal Newport as usual had a good take on this. The issue is a specific type of LLM. The whole article is worth a read, but here is a very quick synopsis:
We can call these (LLMs) long-horizon, dangerously equipped unsupervised LLM-powered agents…
…There’s an obvious solution here: stop racing to amplify this very specific type of particularly unstable system.
This wouldn’t even necessarily entail much financial sacrifice. Most of the things people already like doing with LLMs, or hope LLMs will enable soon, don’t require these haphazard and unpredictable setups. Anthropic and OpenAI could stop working on them today with essentially zero impact on their projected revenue.
Not much. Fortunately a company that specializes in detecting hacking attempts alerted the company that experienced the breach and it was quickly contained.
My theory is that we should simply pierce the corporate veil and make the executives of these companies personally liable for any harm they produce, up to and including criminal charges. They are going to be richly rewarded if they succeed, perhaps they should also have some downside to think about. Maybe Safety Officers with actual power, testing suites, stuff like that?
In a paper published this summer we find that AI has designed 16 new viruses that can easily infect bacteria (E. coli). Presumably But if someone wanted to make a virus that could infect humans, it would be a lot easier and faster than it would have been back in the prehistoric era of, say, 2019.
“A cocktail of the generated bacteriophages rapidly overcame bacteria that had evolved resistance to a natural bacteriophage. This work lays a foundation for AI-guided design of biological function at the whole-genome scale.”
This reminds me to, again, recommend several good books on the subject.
“Click Here to Kill Everybody” by Bruce Schneier. Primarily about cyber security but also talks about the danger of AI and the likely need to bring it under the military.
“Empire of AI” by Karen Hao. Mostly about the formation of OpenAI, but generally about how the industry has skirted copyright law, exploits cheap African labor, the issues with the data centers.
“The Nerd Reich” by Gil Duran. This chiefly talks about how and why Silicon Valley veered so hard right in the last 10 years, what their goals and plans are for civilization and how they intend to bring it about. And whether the 98% of us should want it.