Medical overtrust — chatbots are not your doctor

People sometimes ask chatbots for diagnoses, drug advice, or emergency guidance. Models can sound confident while being wrong, outdated, or incomplete. Overtrust—believing fluent answers—is the hazard.

This page teaches caution. It does not provide medical protocols, and it is not medical advice.

Clinicians, researchers, and regulators have publicly discussed risks of consumer AI for health advice: incorrect recommendations, missed emergencies, and privacy issues when people paste sensitive records into tools. Newsrooms and medical journals have published evaluations where models fail medical exam-style or triage-style questions in important ways. Treat any single headline as one study; the stable lesson is: fluency ≠ clinical care.

Health systems that deploy AI officially do so under governance, validation, and liability frameworks—very different from an unsupervised chat window.

A wrong answer about symptoms can delay real care. A privacy leak of health details can follow you for years.

• Emergencies: call local emergency services—do not negotiate with a chatbot.
• Use clinicians and licensed resources for diagnosis and treatment.
• If you use AI at all for health literacy, treat it like a rough encyclopedia, then verify with trusted medical sources.
• Do not paste identifiable medical records into consumer tools.

Useful questions to ask any “health AI” product:

• Was it evaluated for this use? By whom?
• What happens when it is unsure?
• Where do I see sources?
• Is my data stored or used for training?

⚠️ Myth: If the answer is detailed, it must be right.

✓ Reality: Generative models can be detailed and wrong (hallucinations—Course 08).

⚠️ Myth: Disclaimers make unsupervised medical chatbot use safe.

✓ Reality: Disclaimers help legally; they do not remove clinical risk.

• FDA / health authority pages on AI in medical devices (for regulated clinical AI context; start at https://www.fda.gov/ ↗ and search current AI/ML medical device guidance)
• World Health Organization materials on AI ethics/governance in health (verify current documents on https://www.who.int/ ↗)
• Course 08 Hallucinations; Course 20 Society