Microsoft published results for MAI-DxO, its Diagnostic Orchestrator, in Science on Thursday. The tool coordinates several AI agents like a panel of doctors: they question the patient, order tests, argue about a diagnosis. On complex New England Journal of Medicine cases it was right 83% of the time, against 36% for generalist physicians working alone.
Key takeaways
- 83% on NEJM cases versus 36% for doctors with references and 22% without
- 80% versus 47% on emergency department cases from Beth Israel Deaconess
- Best results came with GPT-5 and GPT-5.5, with Google, Anthropic, Meta and DeepSeek also tested
- Gemini 2.5 Flash: 46.4% alone, close to 69% with orchestration
- SDBench is to be open-sourced
Orchestration helps weaker models most
The study's data shows most of the gain comes from the base model, with orchestration adding the rest. GPT-5 with a baseline prompt reached 78.9% on hard cases at roughly $8,400 each. DxO cut that cost by close to $3,000 but raised accuracy by only four points.
With the newer GPT-5.5 the results split. On complex cases orchestration lifted accuracy from 72% to nearly 87%, and slightly cheaper. On emergency department cases there was no real difference — both scored around 85%. The biggest jump came with weaker models, as gains shrink near the ceiling.
| Model | Without orchestration | With DxO |
|---|---|---|
| GPT-5 | 78.9% | about 4 points more, roughly $3,000 cheaper |
| GPT-5.5 | 72% | close to 87% |
| Gemini 2.5 Flash | 46.4% | close to 69% |
Cases of this class are normally handled by specialists or physician teams, and neither was in the comparison. Microsoft measured its system against primary care doctors and internists working alone. The company joins Google and OpenAI, both of which claim their models match or beat doctors.
Why it matters
The result measures something narrower than "medical superintelligence" suggests. Orchestration looks more like a cost optimiser than a source of new diagnostic capability — with a strong model and simpler cases its contribution vanishes. The open SDBench benchmark will matter more than the headline percentage, because it lets the experiment be re-run independently. Without it this stays a vendor-chosen comparison, and the most interesting question — how the system fares against a specialist team — is unanswered.
What's next
- Microsoft is training a healthcare foundation model with Mayo Clinic, release planned next year
- SDBench is to be released as an open benchmark
- Suleyman projects streaming predictions of how a medical record will unfold by 2027
