Skip to main content

Anthropic Urges Global Pause in AI Development, Flags 'Self-Improvement' Risk

1 month 3 weeks ago
Anthropic is urging leading AI labs to consider slowing development, warning that frontier models are advancing fast enough that they may soon be able to improve themselves without direct human intervention. The company says a global ability to pause or slow AI development would "likely be a good thing," citing internal data about accelerating model capabilities. From a blog post: Using public benchmarks and previously unreported data from within Anthropic, The Anthropic Institute is showing that AI is already accelerating the development of AI systems. To take just one example: today, Anthropic engineers on average ship 8x as much code per quarter as they did from 2021-2025. The technical trends discussed in this piece suggest that AI systems are going to become much more capable in coming years. These trends have huge implications. AI that can build itself would be a major development in the history of technology -- one that could bring enormous good for the world in science, healthcare, and beyond. But full recursive self-improvement also might increase the risks of humans losing control over AI systems. If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important. [...] If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing. But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe. Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures. We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology. The Anthropic Institute will conduct research -- in collaboration with many others -- and take actions to help build the systems that a credible slowdown or pause would require. These systems would enable frontier AI developers to verify that others globally have actually stopped or slowed, and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret. If such systems existed, we expect that we would slow down or temporarily pause, if other developers at or near the frontier also did so in a verifiable manner...

Read more of this story at Slashdot.

BeauHD

New IronWorm Malware Hits 36 Packages In npm Supply-Chain Attack

1 month 3 weeks ago
A new npm supply-chain attack has infected 36 packages with Rust-based infostealer malware called IronWorm. According to BleepingComputer, the malware "targets 86 environment variables (key-value pairs) and 20 credential files that may contain OpenAI, AWS, Anthropic, and npm credentials, vault configuration files, SSH keys, and Exodus cryptocurrency wallet files." From the report: According to researchers at supply-chain and devops company JFrog, IronWorm is written in Rust, hides behind an eBPF kernel rootkit, and communicates with the operator over the Tor network. The Rust-based malware self-propagates by using stolen credentials for publishing on npm; this includes secrets associated with npm's Trusted Publishing workflow. Once it compromises a developer or CI environment, it can publish trojanized versions of packages owned by the victim, which then infect additional developers and CI systems. This behavior is conceptually similar to Shai Hulud, which had its code published on GitHub recently. Although JFrog researchers did not find a clear connection between IronWorm and Shai Hulud, they observed the same commit names in both supply-chain attacks. This opens the possibility that the new malware is an evolution of TeamPCP's payload, since IronWorm appears to be "a custom, carefully built implant from an operation with its own infrastructure." [...] The company provides a list of all impacted package names and their versions in the report and recommends that developers upgrade to fixed releases, rotate their keys, and enable two-factor authentication (2FA) for all accounts. At the same time, Endor Labs and StepSecurity have spotted a very similar but distinct attack involving a JavaScript-based malware named binding.gyp, performing registry poisoning and GitHub Actions infection, unfolding during the same time-frame.

Read more of this story at Slashdot.

BeauHD

Companies Are Using Reddit To Manipulate ChatGPT and Google AI Search

1 month 3 weeks ago
An anonymous reader quotes a report from 404 Media: The moderators of the biohacking subreddit say that peptide and hormone replacement therapy companies have been surreptitiously spamming Reddit in an attempt to get their posts scraped by AI chatbots. The strategy is an effort to systematically manipulate the answers provided by chatbots by manipulating the underlying source material that those chatbots will scrape -- in this case, a popular Reddit community. In a post last week, the moderators of r/biohackers said they would be banning new posts about peptides and hormone replacement therapy (HRT) because of attempted manipulation by the companies that make, market, and sell them. [...] "As AI search engines increasingly pull answers from Reddit, companies are using us for AEO. On top of that, there's been an explosion of peptide interest and AI usage flooding the sub. Together, this has put serious pressure on content quality," a post by the moderators read. [...] It has become incredibly difficult to stop Reddit manipulation, because the firms doing it are getting more sophisticated. The moderator said that there are really standard and long-running strategies where brands will hop in the comments and suggest their products: "That type of marketing has always existed and if people want to try something new because the brand resonated with them, cool. That's the way marketing should flow in my mind," they said. "But what I'm seeing that is way scarier to me is that there are companies that will reverse-engineer the actual prompt patterns that are prioritized by LLMs, and so you'll see someone post a super clickbait, high-traction, vague question like 'Is all the hype around Vitamin D actually worth it?" they added. "And that thread will do really well because everyone on biohackers actually has an opinion, so it gets engagement and prioritized by LLMs, and then brands will sneak in and they'll embed their brand mentions in those threads in the exact right places in a seemingly organic way. But none of it is organic, the entire thing is a strategy by an agency to prioritize brand mentions or a narrative within an LLM." The Reddit accounts that are doing this are "warmed up" or are made to seem human, meaning they have a posting history that is not just promotional. This makes them much harder to detect and moderate against. Some of the agencies doing this are paying real people to post promotional content, or have built communities where people are incentivized to post promotional content. The moderator said that Reddit's automated moderation tools have been helpful, but that the type of promotion happening has become so sophisticated that it has become more of a you-know-it-if-you-see it kind of thing. "A lot of it has become pattern recognition," they said. "You literally just sort of know what to look for. But the problem is you don't want to become punitive to the people who aren't doing this maliciously, and so I think the over-moderation risk is very real."

Read more of this story at Slashdot.

BeauHD

Meta Keeps Delaying the Release of Its New AI Model to Developers

1 month 3 weeks ago
Meta has reportedly delayed the developer release of its Muse Spark AI model API multiple times, and as of Tuesday, had no scheduled launch date, according to the Wall Street Journal (paywalled). Reuters reports: A Meta spokesperson told Reuters on Wednesday that the company is already testing the Application Programming Interface (API) with some early partners and is looking forward to releasing it this month. "The muse spark API will be coming soon," Meta AI Chief Alexandr Wang announced in a post on X in April. Meta unveiled Muse Spark in April as the first model built to close the gap with rivals. Muse Spark is the first in a new series of models created by the company's Superintelligence Labs. Earlier on Wednesday, Meta unveiled an AI agent aimed at helping businesses carry out day-to-day operations, hinting at the company's ambitions to compete with rivals such as OpenAI, Anthropic and Alphabet's Google.

Read more of this story at Slashdot.

BeauHD