A study led by researchers at Harvard Medical School and Beth Israel Deaconess Medical Center, published in the journal Science, found that OpenAI’s o1 reasoning model outperformed physicians at diagnosing patients in emergency room scenarios, and the findings are still generating debate months later, including in a Reason magazine piece published August 9, 2026 that revisited the study amid broader questions about whether AI will replace doctors.
The study’s most closely watched test used 76 real cases drawn from Beth Israel’s emergency department. The AI model and two attending physicians were each given identical inputs — electronic medical records, vital signs and a few sentences written by the intake nurse — with no data cleanup or preprocessing done before the AI saw the records.
How the AI and doctors compared
At the triage stage, the o1 model correctly identified the exact or a near-correct diagnosis in 67% of cases, compared with 55% and 50% for the two attending physicians in the comparison. By the point of hospital admission, when more clinical information had accumulated, the AI’s accuracy rose to about 82%, versus roughly 79% and 70% for the two physicians. A separate panel of attending physicians graded all the diagnoses blind, without knowing which had come from the AI and which from a human colleague.
Beyond the flagship test
The 76-case emergency department comparison was just one of six separate experiments the research team ran, pitting the o1 model against hundreds of physicians across different levels of training and specialty, including residents, specialists and family physicians. According to the researchers, the model outperformed human physicians in every one of the six experiments, not just the marquee ER test, suggesting the result wasn’t a fluke tied to one particular dataset or specialty.
What the lead researcher said
Arjun Manrai, an assistant professor of biomedical informatics at Harvard’s Blavatnik Institute and a senior co-author of the study, said the AI surpassed both earlier AI models and physician benchmarks across nearly every test the team ran. Manrai and his co-authors emphasized that the model and the physicians received exactly the same raw information, with no advantage given to either side in how the case data was prepared or presented.
The pushback
The study stopped well short of recommending that hospitals deploy o1 or similar models for actual emergency diagnosis, calling instead for formal prospective clinical trials before any such tool touches real patient care decisions. Critics have raised sharper concerns: emergency physician Kristen Panthagani, among others, has argued that comparing an AI model to non-specialist physicians, and treating diagnostic pattern-matching on a written case as equivalent to genuine emergency care, understates what real ER medicine involves — physical exams, patient communication, triage under time pressure, and responsibility for outcomes that a model reviewing a case file afterward never has to bear.
Why this matters beyond one study
The Harvard-Beth Israel results land amid a broader wave of research testing reasoning-style AI models against physicians on diagnostic tasks, part of a larger argument in health policy circles over how quickly — and how safely — AI could be integrated into frontline clinical decision-making. Reasoning models like o1, which work through problems step by step before producing an answer, have shown stronger performance on complex diagnostic reasoning than earlier generations of AI, and studies like this one are being used by advocates on both sides: those pushing for faster clinical AI adoption, and those warning that lab benchmarks built on retrospective case files don’t capture what happens when an algorithm is thrown into a chaotic, understaffed emergency department in real time.
What comes next
Manrai’s team and other researchers working in this space are expected to pursue the prospective clinical trials the study called for — testing, in real time rather than retrospectively, whether AI-assisted diagnosis actually changes patient outcomes when deployed alongside working physicians rather than compared against them after the fact. Until that evidence exists, the Harvard study is likely to remain a talking point rather than a deployment blueprint, cited by AI optimists and clinical skeptics alike as they argue over how much weight a single, well-designed but retrospective study should carry.
