Skip to main content

FDA's New Drug Approval AI Is Generating Fake Studies

3 weeks ago
An anonymous reader quotes a report from Gizmodo: Robert F. Kennedy Jr., the Secretary of Health and Human Services, has made a big push to get agencies like the Food and Drug Administration to use generative artificial intelligence tools. In fact, Kennedy recently told Tucker Carlson that AI will soon be used to approve new drugs "very, very quickly." But a new report from CNN confirms all our worst fears. Elsa, the FDA's AI tool, is spitting out fake studies. CNN spoke with six current and former employees at the FDA, three of whom have used Elsa for work that they described as helpful, like creating meeting notes and summaries. But three of those FDA employees told CNN (paywalled) that Elsa just makes up nonexistent studies, something commonly referred to in AI as "hallucinating." The AI will also misrepresent research, according to these employees. "Anything that you don't have time to double-check is unreliable. It hallucinates confidently," one unnamed FDA employee told CNN. [...] Kennedy's Make America Healthy Again (MAHA) commission issued a report back in May that was later found to be filled with citations for fake studies. An analysis from the nonprofit news outlet NOTUS found that at least seven studies cited didn't even exist, with many more misrepresenting what was actually said in a given study. We still don't know if the commission used Elsa to generate that report. FDA Commissioner Marty Makary initially deployed Elsa across the agency on June 2, and an internal slide leaked to Gizmodo bragged that the system was "cost-effective," only costing $12,000 in its first week. Makary said that Elsa was "ahead of schedule and under budget" when he first announced the AI rollout. But it seems like you get what you pay for. If you don't care about the accuracy of your work, Elsa sounds like a great tool for allowing you to get slop out the door faster, generating garbage studies that could potentially have real consequences for public health in the U.S. CNN notes that if an FDA employee asks Elsa to generate a one-paragraph summary of a 20-page paper on a new drug, there's no simple way to know if that summary is accurate. And even if the summary is more or less accurate, what if there's something within that 20-page report that would be a big red flag for any human with expertise? The only way to know for sure if something was missed or if the summary is accurate is to actually read the report. The FDA employees who spoke with CNN said they tested Elsa by asking basic questions like how many drugs of a certain class have been approved for children. Elsa confidently gave wrong answers, and while it apparently apologized when it was corrected, a robot being "sorry" doesn't really fix anything.

Read more of this story at Slashdot.

BeauHD

Leading 3D Printing Site Bans Firearm Files

3 weeks ago
Thingiverse, a popular 3D printing file repository, has agreed to remove downloadable gun designs following pressure from Manhattan DA Alvin Bragg, who is pushing for stricter moderation and voluntary cooperation across the 3D printing industry. "However, it's unlikely to slow the proliferation of 3D printed weapons, as many other sites offer downloadable gun designs and parts," reports The Register. From the report: Earlier this year, Bragg wrote to 3D printing companies, asking them to ensure their services can't be used to create firearms. On Saturday, Bragg announced that one such company, Thingiverse, would remove working gun models from its site. The company operates a popular free library of 3D design files and had already banned weapons in its terms of use, but is now promising to improve its moderation procedures and technology. "Following discussions with the Manhattan District Attorney's Office about concerns around untraceable firearms, we are taking additional steps to improve our content moderation efforts," Thingiverse said in a statement. "As always, we encourage our users to report any content that may be harmful." [...] At any rate, while Thingiverse may be popular among 3D printing mavens, people who like to build their own guns look to other options. [...] Bragg's approach to 3D printing sites and 3D printer manufacturers is to seek voluntary cooperation. Only Thingiverse and YouTube have taken up his call, others may or may not follow. "While law enforcement has a primary role to play in stopping the rise of 3D-printed weapons, this technology is rapidly changing and evolving, and we need the help and expertise of the private sector to aid our efforts," Bragg said. "We will continue to proactively reach out to and collaborate with others in the industry to reduce gun violence throughout Manhattan and keep everyone safe." But it seems doubtful that the sites where Aranda and other 3D gun makers get their files will be rushing to help Bragg voluntarily.

Read more of this story at Slashdot.

BeauHD

UN World Court declares countries must curb emissions or be held responsible

3 weeks ago
But it has no way to compel scofflaws to comply

In a sweeping and unprecedented legal opinion, the United Nations' highest court has decreed that "The consequences of climate change are severe and far-reaching" and constitute an "urgent and existential threat." What's more, it stated that action must be taken to not only ameliorate that threat but also to determine the legal consequences for those states whose actions harm others.…

Rik Myslewski