What happens when a chatbot follows the wrong voice in the room? In this episode, we explore the hidden vulnerabilities of prompt injection, where malicious instructions and fake signals can mislead even the most advanced AI into offering harmful medical advice.

We unpack a recent study that simulated real patient conversations, subtly injecting cues that steered the AI to make dangerous recommendations—including prescribing thalidomide for pregnancy nausea, a catastrophic lapse in medical judgment. Why does this happen? Because language models aim to be helpful within their given context, not necessarily to prioritize authoritative or safe advice. When a browser plug-in, a tainted PDF, or a retrieved web page contains hidden instructions, those can become the model’s new directive, undermining guardrails and safety layers.

From direct “ignore previous instructions” overrides to obfuscated cues in code or emotionally charged context nudges, we map the many forms of this attack surface. We contrast these prompt injections with hallucinations, examine how alignment and preference training can unintentionally amplify risks, and highlight why current defenses, like content filters or system prompts, often fall short in clinical use.

Then, we get practical. For AI developers: establish strict instruction boundaries, sanitize external inputs, enforce least-privilege access to tools, and prioritize adversarial testing in medical settings. For clinicians and patients: treat AI as a research companion, insist on credible sources, and always confirm drug advice with licensed professionals.

AI in healthcare doesn’t need to be flawless, but it must be trustworthy. If you’re invested in digital health safety, this episode offers a clear-eyed look at where things can go wrong and how to build stronger, safer systems. If you found it valuable, follow the show, share it with a colleague, and leave a quick review to help others discover it.

Reference:

Vulnerability of Large Language Models to Prompt Injection When Providing Medical Advice
Ro Woon Lee
JAMA Open Health Informatics (2025)

Credits:

Theme music: Nowhere Land, Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 4.0
https://creativecommons.org/licenses/by/4.0/