Google Scales Back AI Health Summaries After Safety Fears Over Misleading Advice

By Staff Reporter

Google has removed or modified a number of its AI-generated health summaries from search results after a Guardian investigation found the feature was serving false and potentially dangerous medical advice at the top of key health queries, prompting warnings from charities and patient advocates about risks to public safety.[2][3][4][6]

AI Overviews Under Fire for Misleading Health Advice

The controversy centres on Google’s AI Overviews – automated summaries that appear at the top of search results and are designed to provide quick, conversational answers to user questions by drawing on material from across the web.[2][3] Health organisations and experts say that when these summaries are wrong, the prominent placement risks amplifying misinformation and giving it an unwarranted aura of authority.[2][4][5]

A Guardian investigation, published this month, identified multiple instances in which AI Overviews offered advice that conflicted with accepted clinical guidance and could lead to harm, delay in diagnosis, or inappropriate self-management.[2][3][6] The findings triggered a wave of criticism from patient groups, with some describing the errors as “dangerous and alarming”.[2][4][6]

Examples of Risky and Inaccurate Guidance

Among the most concerning examples highlighted by charities and experts were summaries affecting patients with serious or life‑limiting conditions.[1][2][3][6]

• Pancreatic cancer dietary advice: In one widely cited case, Google’s AI Overview advised pancreatic cancer patients to avoid high‑fat foods.[1][2][6] Pancreatic cancer specialists and charities, including Pancreatic Cancer UK, warned that this is not standard advice and could be actively harmful.[1][2] For many patients, maintaining weight and calorie intake is critical to surviving chemotherapy and major surgery; restricting fat could make it harder to consume enough calories and may jeopardise treatment.[1]
• Misleading liver blood test explanations: Investigators also found AI Overviews providing inaccurate explanations of liver function blood tests, mischaracterising what abnormal results might mean and what steps patients should take.[2][3][6] Experts cautioned that such misinformation could cause people either to dismiss worrisome results or to become unduly alarmed, leading them away from appropriate medical follow‑up.[2]
• Errors on women’s cancer screening: Other AI summaries were found to contain false or incomplete information about screening for women’s cancers, including who is eligible, how often screening should occur, and what abnormal results imply.[2][4][6] Cancer charities warned that errors in this area can directly influence whether people attend screening or seek further evaluation.

Health professionals and digital health experts noted that, in some cases, AI Overviews appeared to mash together fragments of online content without sufficient clinical context or nuance, producing advice that sounded confident but was incomplete or misleading.[2][3]

Charities Warn of Delayed Diagnosis and Harm

Patient advocates and health information specialists reacted sharply to the revelations, warning that the design of AI Overviews magnifies the impact of any inaccuracies.[2][4][5]

Sophie Randall, director of the UK Patient Information Forum, said the investigation showed that false or context‑free information at the top of search results can “put people at risk of harm” by shaping decisions at highly anxious moments, when users may be searching in the middle of the night without immediate access to professionals.[1][2][4]

Charities raised particular concerns that misleading assurances might cause people to delay seeking care for serious symptoms or to disregard professional advice, while other flawed summaries risked prompting unnecessary worry or self‑treatment.[2][4][6] Experts also criticised the lack of clear signalling around uncertainty and the absence of consistent prompts to consult qualified clinicians for serious or persistent symptoms.[2][4]

Inconsistency and Opacity Fuel Concerns

Beyond specific factual errors, the investigation found that AI Overviews could give different answers to the same health query at different times, even when the underlying evidence base had not changed.[2][3] Researchers and clinicians argued that such variability undermines trust and makes it harder for people to know whether the information they see is reliable.[2]

Digital health commentators noted that, unlike traditional search snippets that clearly link to specific documents, AI Overviews often blend material from multiple sources, making it difficult for users to scrutinise where particular claims originate or what level of evidence supports them.[2][3] That opacity, they said, raises questions about accountability when errors occur.

Google Responds, Removes Some AI Summaries

In response to the criticisms, Google has acknowledged issues with some of the health‑related AI Overviews surfaced in the investigation and confirmed that it has removed or adjusted a subset of summaries that were found to be inaccurate, misleading or lacking critical context.[2][3][5]

The company insists, however, that the majority of AI Overviews are “accurate and helpful” and emphasises that it is investing heavily in safeguards for sensitive domains like health.[1][2][5] Google said that, when problems are identified — for example, where the system misinterprets underlying web content or fails to flag that users should seek medical care — it takes action to refine the models and improve prompt handling.[2][5]

Google also highlighted that AI Overviews are generally linked to what it describes as “well‑known and reliable sources” and that users are encouraged to click through to those sources and consult health professionals rather than relying solely on concise summaries for clinical decision‑making.[1][2]

Broader Questions About AI in Health Information

The controversy around Google’s AI Overviews comes amid wider debate about the role of generative AI in healthcare communication and decision‑support. Recent research has shown that large language models can misinterpret or inconsistently apply medical risk terms, further complicating efforts to use AI safely in clinical contexts.[7]

A study reported by Vanderbilt University Medical Center in JAMA Network Open found that several leading AI models — including systems from multiple major providers — frequently failed to align with established definitions when describing how common side‑effects or outcomes are.[7] Where regulators define “very rare” events as affecting up to 1 in 10,000 people, some models used figures orders of magnitude higher, and they often avoided giving numbers at all, especially when questions were phrased anxiously or involved severe conditions.[7]

Experts say these findings, combined with the Guardian’s investigation into AI Overviews, underscore the need for robust oversight, transparent evaluation, and closer collaboration between technology firms, clinicians, regulators and patient groups when deploying AI tools that touch on health.[2][3][7]

Calls for Stronger Safeguards and Clearer Labels

Following the revelations, charities and digital safety advocates have urged Google and other tech platforms to introduce stronger guardrails for AI‑generated health content.[2][4][5] Proposals include clearer labelling that information is AI‑generated, explicit prompts to seek medical advice for worrying symptoms, and tighter restrictions on AI summaries in areas such as cancer, rare diseases, mental health crises and paediatrics.

Health information organisations argue that AI systems deployed at internet scale need evaluation frameworks comparable to those used for patient leaflets or clinical decision tools, with input from medical experts, user‑testing with patients, and ongoing monitoring for harmful outputs.[2][4]

For now, clinicians continue to advise the public to treat AI‑generated health summaries — whether from search engines or chatbots — as starting points for discussion rather than definitive sources of medical guidance, and to seek personalised advice from qualified professionals before making decisions about diagnosis, treatment or changes to medication.[2][4][7]