medqa
**MedQA** is the **medical question answering benchmark derived from the United States Medical Licensing Examination (USMLE) and equivalent exams in China and Taiwan** — testing whether AI can answer the multi-step clinical reasoning questions that physicians must answer to obtain medical licensure, requiring integration of basic science knowledge, pathophysiology, clinical presentation, and treatment guidelines.
**What Is MedQA?**
- **Origin**: Jin et al. (2021).
- **Scale**: 61,097 multiple-choice questions (4-5 options) across three datasets.
- **Sources**: USMLE Step 1/Step 2/Step 3, Chinese National Medical Licensing Examination (CNMLE), Taiwanese Medical Licensing Exam.
- **Format**: Clinical vignette (2-8 sentences describing a patient presentation) + question + answer choices.
- **Languages**: English (USMLE), Simplified Chinese (CNMLE), Traditional Chinese (Taiwan).
- **Difficulty**: Requires qualifying for medical licensure — questions test clinical reasoning at the level required of practicing physicians.
**The USMLE Clinical Vignette Format**
A typical MedQA question:
"A 58-year-old woman presents with 3 days of progressive shortness of breath, orthopnea, and bilateral leg edema. She has a history of hypertension treated with lisinopril. On examination, JVP is elevated, crackles bilaterally at both bases, and an S3 gallop is present. BNP is 1,240 pg/mL. Which of the following is the most appropriate next step in management?
A. IV furosemide
B. Cardiac catheterization
C. Echocardiography
D. Metoprolol titration
E. Digoxin loading dose"
Answering correctly requires: recognizing acute decompensated heart failure from the constellation of signs, knowing that diuresis (furosemide) is first-line acute management, and ruling out premature invasive investigation.
**Why MedQA Is Hard**
- **Multi-Step Clinical Reasoning**: Questions require recognizing the diagnosis, understanding the pathophysiology driving each finding, and applying treatment guidelines — not just recalling isolated facts.
- **Synthesis Across Disciplines**: USMLE Step 1 integrates biochemistry, anatomy, physiology, microbiology, and pharmacology in single vignettes.
- **Distractor Quality**: Wrong answer choices are common traps — correct drugs for the wrong indication, appropriate management for the misdiagnosed condition.
- **Context Sensitivity**: The same symptom constellation has different correct answers depending on patient age, comorbidities, and acuity.
**Performance Timeline**
| Model | MedQA (USMLE) Accuracy |
|-------|----------------------|
| Human (passing threshold) | 60% |
| Human (first-time takers) | ~67% |
| GPT-3 (few-shot) | 44.7% |
| PubMedBERT fine-tuned | 55.9% |
| GPT-3.5 | 57.6% |
| Med-PaLM | 67.6% |
| GPT-4 | 86.7% |
| Med-PaLM 2 | 86.5% |
GPT-4 and Med-PaLM 2 exceeded expert physician performance on MedQA — a landmark result that triggered significant discussion about AI-assisted clinical decision support.
**Why MedQA Matters**
- **Clinical Decision Support Validation**: A system that scores above 80% on MedQA can assist physicians with differential diagnosis and treatment selection at near-expert level.
- **AI Medical Licensing**: MedQA provides the objective standard for "can AI practice medicine?" — a question with profound regulatory and liability implications.
- **Multilingual Medical AI**: The Chinese and Taiwanese versions enable medical AI for 1.4 billion people in a healthcare system with different epidemiological patterns and drug formularies.
- **Reasoning Chain Development**: MedQA vignettes are ideal training data for medical CoT — step-by-step clinical reasoning chains derived from USMLE explanations.
MedQA is **the medical licensing exam for AI** — measuring whether language models can reason through the complex clinical scenarios that define physician competence, with performance crossing the passing threshold representing a genuine milestone in AI-assisted healthcare.