Skip to main content

Anthropic Urges Global Pause in AI Development, Flags 'Self-Improvement' Risk

1 month ago
Anthropic is urging leading AI labs to consider slowing development, warning that frontier models are advancing fast enough that they may soon be able to improve themselves without direct human intervention. The company says a global ability to pause or slow AI development would "likely be a good thing," citing internal data about accelerating model capabilities. From a blog post: Using public benchmarks and previously unreported data from within Anthropic, The Anthropic Institute is showing that AI is already accelerating the development of AI systems. To take just one example: today, Anthropic engineers on average ship 8x as much code per quarter as they did from 2021-2025. The technical trends discussed in this piece suggest that AI systems are going to become much more capable in coming years. These trends have huge implications. AI that can build itself would be a major development in the history of technology -- one that could bring enormous good for the world in science, healthcare, and beyond. But full recursive self-improvement also might increase the risks of humans losing control over AI systems. If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important. [...] If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing. But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe. Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures. We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology. The Anthropic Institute will conduct research -- in collaboration with many others -- and take actions to help build the systems that a credible slowdown or pause would require. These systems would enable frontier AI developers to verify that others globally have actually stopped or slowed, and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret. If such systems existed, we expect that we would slow down or temporarily pause, if other developers at or near the frontier also did so in a verifiable manner...

Read more of this story at Slashdot.

BeauHD

New IronWorm Malware Hits 36 Packages In npm Supply-Chain Attack

1 month ago
A new npm supply-chain attack has infected 36 packages with Rust-based infostealer malware called IronWorm. According to BleepingComputer, the malware "targets 86 environment variables (key-value pairs) and 20 credential files that may contain OpenAI, AWS, Anthropic, and npm credentials, vault configuration files, SSH keys, and Exodus cryptocurrency wallet files." From the report: According to researchers at supply-chain and devops company JFrog, IronWorm is written in Rust, hides behind an eBPF kernel rootkit, and communicates with the operator over the Tor network. The Rust-based malware self-propagates by using stolen credentials for publishing on npm; this includes secrets associated with npm's Trusted Publishing workflow. Once it compromises a developer or CI environment, it can publish trojanized versions of packages owned by the victim, which then infect additional developers and CI systems. This behavior is conceptually similar to Shai Hulud, which had its code published on GitHub recently. Although JFrog researchers did not find a clear connection between IronWorm and Shai Hulud, they observed the same commit names in both supply-chain attacks. This opens the possibility that the new malware is an evolution of TeamPCP's payload, since IronWorm appears to be "a custom, carefully built implant from an operation with its own infrastructure." [...] The company provides a list of all impacted package names and their versions in the report and recommends that developers upgrade to fixed releases, rotate their keys, and enable two-factor authentication (2FA) for all accounts. At the same time, Endor Labs and StepSecurity have spotted a very similar but distinct attack involving a JavaScript-based malware named binding.gyp, performing registry poisoning and GitHub Actions infection, unfolding during the same time-frame.

Read more of this story at Slashdot.

BeauHD