General LLMs tested in orthopedic paper screening

Sakamoto and colleagues compared nine general-purpose large language models on medical questions and orthopedic literature screening. Top models approached the agreement between expert screeners in the literature task. Spine Surgery and Related Research published the study.

Models and tasks

The models reflect availability as of December 2025: OpenAI's GPT-5, GPT-5-mini and GPT-5-nano; Google's Gemini 2.5 Pro, Flash and Flash-Lite; and Anthropic's Claude Opus 4.5, Sonnet 4.5 and Haiku 4.5.

The team used 1,273 MedQA-USMLE medical examination questions and 384 papers from a published orthopedic meta-analysis. In the second task, models replicated initial screening of papers for inclusion in research. All models used the same automated pipeline and structured output format.

Results

MedQA-USMLE accuracy reached 94.27% for GPT-5, 93.40% for Gemini 2.5 Pro and 90.26% for Claude Opus 4.5. These scores measure answers to knowledge questions, not care of patients.

For literature screening, agreement with expert judgments measured by kappa was 0.8881 for Claude Opus 4.5 and 0.8805 for GPT-5. Agreement between the two experts was 0.8951. Kappa accounts for chance agreement; values nearer 1 indicate closer agreement.

The authors also report trends suggesting that model confidence scores could support a two-stage screening process with human review. The abstract does not report time savings or reductions in missed papers from such a workflow.

Scope and limitations

The screening evaluation used one orthopedic meta-analysis. It does not establish performance across other specialties or inclusion criteria. Scores near inter-expert agreement do not establish equivalence between AI and experts.

Here, screening means selecting research papers. The researchers did not evaluate patient diagnosis or disease screening.

Study notes

This validation study used existing questions and research articles. Reporting relies on the public abstract, which does not provide funding or conflict-of-interest details.

Sources

1. [1]
   論文：Sakamoto et al. Performance Evaluation of the Latest General-Purpose Large Language Models in Medical Reasoning and Clinical Screening Tasks
2. [2]
   データベース：PubMed 公開要旨

Images: generated with GPT Image 2.5 (illustrative).