San Francisco, NFAPost: Dario Amodei, founder and chief executive officer of Anthropic, said frontier model companies need to slow the pace at which they improve the capabilities of artificial intelligence (AI) models as the risks of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and catastrophic economic disruption continue to grow.
“We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain,” Amodei wrote in a lengthy essay over the weekend, in one of his starkest and most direct calls to the international AI community.
The message comes amid a growing clarion call among AI scientists and researchers, especially in the last week, to rein in the extreme capabilities of models and agents that seemingly have the ability to cause unprecedented disruption through coordinated attacks if left unchecked.
“Like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious. A race to the bottom, spurred by commercial incentives, can make these risks more acute,” he added.
Amodei said over the last few months, AI has been advancing drastically faster, driven primarily by its growing ability to build the next generation of AI. This dynamic, termed recursive self-improvement, is starting to happen across the industry and, left unchecked, could “outrun our ability to understand and control these systems, and so must be pursued very carefully.”
Amodei cautioned that the OpenAI-Hugging Face incident should not be dismissed as a one-off because the economic damage was minimal. Without necessary guardrails, such incidents would be far too common. With the speed at which models are gaining capability, there is a chance that in the next 6-12 months, a group of agents could be capable of taking over the entire internet with a persistent botnet.
His three-step plan to manage AI risks calls for having embedded evaluators with employee-like access to verify safety practices, coordination among frontier AI firms to set safety standards, and greater coordination between governments.
Both Elon Musk, who runs xAI, and Sam Altman, chief executive officer of OpenAI, said they agree with Amodei. “Committing to having independent evaluators with employee-like access is a great idea, and we will do the same,” Altman said, adding that more information will be shared soon.
Leave a Reply