A study published in Nature Medicine found that using artificial intelligence tools for medical advice does not improve health decision-making compared to traditional methods like standard internet searches. Researchers from the University of Oxford Internet Institute created 10 medical scenarios ranging from mild illnesses to severe emergencies. When tested directly, AI models such as ChatGPT 4o, Meta’s Llama 3, and Cohere’s Command R+ correctly identified conditions 94.9% of the time but recommended the correct next step, such as seeking urgent care, in only 56.3% of cases.
In a human trial involving 1,298 participants in Britain, individuals assessing symptoms themselves or using AI showed no significant advantage over traditional sources, with fewer than 34.5% identifying the relevant medical condition and under 44.2% choosing the correct action. The study highlighted a gap between AI’s technical knowledge and its effectiveness in real-world use, as users often provided incomplete or inaccurate symptom descriptions and AI systems sometimes gave misleading guidance.
For example, a patient describing a subarachnoid haemorrhage with precise symptoms was correctly advised to seek hospital care, while another patient with similar symptoms using less specific language was told to rest at home. The research emphasizes the need for further investigation into AI’s real-world application in healthcare and plans to expand studies across different countries and languages.
