Large language models encode clinical knowledge

This paper investigates whether large language models encode clinical knowledge and can be applied to medical question answering. To move beyond limited automated benchmarks, the authors assemble MultiMedQA, which combines six existing medical question-answering datasets spanning professional medicine, research, and consumer queries with a new dataset of medical questions searched online, HealthSearchQA. They also propose a human evaluation framework that rates model answers along axes including factuality, comprehension, reasoning, possible harm, and bias, and evaluate the 540-billion-parameter PaLM model and its instruction-tuned variant Flan-PaLM.

Using a combination of prompting strategies, Flan-PaLM achieved state-of-the-art accuracy on every MultiMedQA multiple-choice dataset, including 67.6% on the USMLE-style MedQA, surpassing the prior state of the art by more than 17%. However, human evaluation revealed key gaps, prompting the authors to introduce instruction prompt tuning, a parameter-efficient alignment method using few exemplars, producing Med-PaLM. Although Med-PaLM performed encouragingly and improved with scale and tuning, it remained inferior to clinicians, reinforcing the importance of rigorous evaluation frameworks and method development for safe clinical LLMs.