X

Aggregator

AI Insiders Issue New Warnings - Including Former Anthropic Engineer Jacob Coxon

Slashdot
1 day 17 hours ago
Neither OpenAI or Anthropic is acting responsibly, warned AI researcher Jacob Coxon when resigning last week from Anthropic. But just hours earlier, OpenAI VP of Research Aidan Clark had posted "For the first time, I am asking myself if things are moving too fast. I'm honestly not sure, but I am sure that it would be good for us to have an answer to 'What would a successful pace look like?'" CNN noted Wednesday they're just some of the many AI insiders who are now concerned about the speed of research. Coxon even wrote that "The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately." In 10 days since, Coxon's post has been viewed more than 170 million times, warning that OpenAI and Anthropic are "racing straight to self-improving superintelligence and gambling with our lives." It's part of what CNN now calls "pressure on AI companies and governments to do something about the pace of development and safety," where "much of that pressure is coming from staffers inside the companies." One staffer at a top AI company told CNN the fears of how AI could hurt humanity keeps them up at night. Another researcher who recently left a different AI company said it's a common subject of conversation at parties and social events in Silicon Valley. "You can't spend more than a few hours in this community without realizing that a very substantial number of people are really pretty worried about these sorts of outcomes," said the researcher. "A majority would say there's some chance of it killing everyone...." Dozens of Coxon's colleagues in the AI industry publicly supported his statements, with some making even more dire predictions... OpenAI CEO Sam Altman and SpaceXAI CEO Elon Musk agreed to Amodei's proposal to embed independent watchdogs at the AI companies... "I would burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive. I expect a great many of my colleagues across the industry would as well," wrote Drake Thomas, who works on AI safety at Anthropic. "I promise you, we are actually just f**king scared, it's not galaxy brained marketing..." AI staffers told CNN that they fear that as AI gets better at training and improving itself, their leverage goes down, prompting today's urgency. "As we get into this recursive self-improvement loop, I think that might substantially reduce staff's bargaining power, because frankly, you'll be able to replace many of the staff with models that can do as good a job," the researcher who recently left a top AI company said. Coxon posted Tuesday on X that Anthropic "largely initiated" the race to recursively self-improving AI, justifying it with "a belief in its inevitability." And then OpenAI "had to shed a bunch of dead weight like Sora," as he sees it, "because Anthropic was going for the jugular." (In fact, his specific disagreement with Anthropic's leadership was whether China and the U.S. could ever negotiate an alternative to their current race towards self-improving AI...) In an informal "Ask Me Anything", Coxon responded to a question about when we'd see a Terminator-like malevolent AI by saying that "Skynet could go live in the 2030s if we aren't careful. ai-2027.com is a modern skynet story written a year ago and it's on track so far." Yet while AI development risks an end to humankind, "I do think that if we go slower we can take risk to 0%... But this requires radical action." He acknowledged there was still a possibility that the steady increases to model intelligence could suddenly plateau, but "They haven't so far, and it's just a few more steps up the ladder to hit the finish line." To avoid stifling innovation, he recommends "prioritizing applications that actually improve people's lives [like healthcare discoveries], rather than immediately going for raw economic value or intelligence." But isn't mass unemployment a more pressing threat? "Things are coming so fast that unemployment would be a brief preliminary to deadly superintelligence." To people who feel disempowered, Coxon offered his solidarity. "I also feel disempowered. Part of resigning was a feeling of hopelessness about the future. I would say — keep your eyes open as things get crazier and advocate for increased transparency into AI companies." When asked if he'd start his own company now, Coxon said he had "No idea what I'm doing next."

Read more of this story at Slashdot.

EditorDavid