Skip to main content

OpenAI Announces Benchmarks for AI Life Sciences Research. Its Best Model Failed 63.9% of the Test

2 months 1 week ago
This week OpenAI announced a 750-task test to to measure "whether AI systems can support realistic life science research tasks, not just answer biology questions." But while OpenAI's top-performing GPT-Rosalind model led the rankings, Slashdot reader BrianFagioli notes that "it achieved a pass rate of just 36.1 percent, failing nearly two-thirds of benchmark tasks." Nerds.xyz points out that means "the best-performing model failed nearly two-thirds of the benchmark's tasks." The benchmark also revealed a familiar weakness. AI systems generally perform better when everything is presented as text. Once they are forced to work with supporting documents, figures, or complex datasets, performance drops noticeably. GPT-Rosalind's pass rate fell from 45.1 percent on text-only tasks to 28.1 percent on tasks involving artifacts or URLs. To be fair, the benchmark is not intended to suggest AI is useless in research. Quite the opposite. OpenAI found that models are becoming increasingly capable of scientific communication, evidence synthesis, and translating research findings into practical explanations. Those are valuable skills, particularly for researchers drowning in information. But LifeSciBench serves as a useful reminder that today's AI systems are still far from autonomous scientists. They can help. They can assist. They can sometimes provide surprisingly useful insights. What they cannot reliably do, however, is replace the expertise, judgment, and skepticism that real scientific research requires.

Read more of this story at Slashdot.

EditorDavid

Remembering When Alan Turing Developed a Portable Voice Encryption Device

2 months 1 week ago
Long-time Slashdot reader smooth wombat writes: Alan Turing, one of the more famous people who worked at Bletchley Park to decipher the German Enigma coding machine, was also working on a separate project. His private papers, known as the Bayley papers for his assistant Donald Bayley who held onto the papers until his death in 2020, reveal Turning had produced a working model of a portable voice encryption device. He even demonstrated it by using a Winston Churchill speech recording. "Weighing just 39 kg, including its power pack," Jack Copeland wrote in an article for IEEE Spectrum, "Delilah would be at home in a truck, a trench, or a large backpack." More from Popular Mechanics: Turingâ(TM)s work at Bletchley Park actually informed the Delilah experimentation he was doing at Hanslope Park, and not just because he used Red Forms, the Army-issue sheets Hanslope staffers were meant to use to alert Bletchley staffers to enemy signals, as his personal scrap paper for Delilah experiments. He drew inspiration from one of the German cipher machines they had decoded at Bletchley; not the famed Enigma machine, but rather the SZ42. While the former relied on Morse Code, the latter utilized a 5-bit telegraph code, which Copeland notes âoewas a forerunner of ASCII and Unicode and is still used by some ham radio operators.â The SZ42 produced an obscuring key of telegraph characters, with an identical key produced to both the sender and receiver. If it could be done for text, Turing reasoned it could be done for sound as well... [T]he reason Delilah fell to the wayside of history isnâ(TM)t because it was a failure, but rather because it simply wasnâ(TM)t needed anymore. By the time Turing had built and demonstrated his device, the war was over. What good was a portable voice encryptor if you had no major enemies trying to intercept your calls, the government reasoned. So funding for the project stopped, and Turingâ(TM)s two-year experiment ended with a whimper. Turingâ(TM)s time as an electrical engineer at Hanslope Park became a footnote in his story, if even that.

Read more of this story at Slashdot.

EditorDavid