The AI Security Review Every Healthcare CTO Should Run
Most healthcare AI features ship without a real security review. Here is the checklist that catches exposure before a customer or regulator does.
This is some text inside of a div block.
Most healthcare AI features ship without a real security review. Here is the checklist that catches exposure before a customer or regulator does.
There is a moment in every healthcare software company's AI journey where the feature works, the demo lands, and the pressure to ship becomes hard to resist. The security review that would normally precede a launch gets compressed, deferred, or skipped entirely, mostly because no one on the team is quite sure what a security review of an AI feature actually looks like. This post is that review, laid out in phases. It is the sequence we run when evaluating AI integrations in behavioral health and healthcare platforms, adapted into something a CTO can drive internally. It will not replace a full engagement, but it will surface the exposures most likely to cause damage, and it will tell you quickly whether you are looking at a manageable punch list or a structural problem.
Healthcare remains the costliest industry for data breaches for the fourteenth year running, averaging 7.42 million dollars per breach and taking longer to detect than almost any other sector. AI adds a specific version of this risk. Unauthorized or ungoverned AI tools were involved in roughly 20 percent of breaches studied in one recent analysis, nearly all of them at organizations without proper access controls in place. A rushed AI launch is exactly the kind of gap that shows up in numbers like these later.
Everything starts with a truthful map, not the architecture diagram from the design document but the actual paths PHI travels once AI is in the loop. For each AI feature, trace what data enters the prompt or context, where it comes from, which vendor or model receives it, what that vendor retains and for how long, what the model produces, and where the output goes afterward. Do this by reading code and configuration, not by interviewing the team, since the gap between what engineers believe an integration sends and what it actually sends is where most findings live. Pay close attention to context assembly, the code that gathers relevant records into a prompt, because that is where scope creep hides, and to any retrieval or embedding pipeline, since vector stores are among the most commonly unmapped PHI repositories in modern healthcare stacks. This phase is done when there is a data flow diagram covering every AI touchpoint, every vendor, and every storage location for prompts, outputs, embeddings, and logs. If that cannot be produced in a few days of focused work, that delay is itself a finding.
With the map in hand, interrogate every external dependency. For each AI vendor, is there a signed BAA, and does it cover the specific service, endpoint, and tier actually in use? Do the technical configurations, retention settings, training opt outs, region and deployment choices, match what the agreement assumes? If an intermediary platform sits between the product and the model provider, is their subprocessor chain papered all the way down? Then check the shadow layer: internal tools, engineer workflows, and support processes that touch production data. A pasted stack trace containing patient data, a support workflow that runs tickets through an AI assistant, an analytics notebook calling a model API, these are disclosures too, and they almost never appear on the official integration list. This phase is done when every PHI touching vendor relationship has an agreement, a verified configuration, and a named owner.
Now test the AI layer the way an adversary, or an unlucky user, eventually will. Four categories matter most in healthcare. Isolation asks whether one patient's data can appear in outputs about another. Test retrieval filters under adversarial queries, session handling across a clinician's caseload, and caching behavior under concurrency. In behavioral health, cross patient contamination is the nightmare scenario, and it needs to be confirmed with actual tests, not just an architecture review. Injection asks whether untrusted content, in intake forms, uploaded documents, or patient messages, can steer the model into disclosing context, ignoring constraints, or producing harmful output. Assume users' documents will eventually contain adversarial text, because eventually they will. Extraction asks whether a feature reveals more than intended, system prompts, other users' history, or upstream data it should not expose. Verbose failure modes and overly helpful assistants are the usual culprits. Output safety asks how wrong a clinically consequential output can be, and what actually catches it before a clinician acts on it. A hallucinated medication detail in a summary a clinician skims quickly is a patient safety issue wearing a product quality costume. This phase is done when there are documented test results for each category, with failures ranked by severity and reproducibility.
Compliance is the ability to demonstrate all of the above, not just to have done it. Confirm the HIPAA risk analysis explicitly covers the AI data flows found in phase one, since an analysis written before those features shipped does not cover them. Confirm audit logging captures AI feature usage, who invoked what, on which patient's data, and when. Confirm privacy notices and customer facing security documentation describe the AI processing accurately, because an inaccurate disclosure compounds any later incident. And confirm the incident response plan actually contemplates AI specific scenarios, a vendor side exposure or a contamination bug, with named owners and communication paths.
A review that ends in a fifty item spreadsheet changes nothing. Rank findings on two axes, severity of exposure and effort to fix. Unagreemented PHI flows and isolation failures go first, since they represent active exposure. Configuration corrections, retention settings and logging scope, usually follow quickly because they tend to be high severity and low effort. Architectural work, segmentation, output controls, monitoring, gets sequenced into the roadmap with owners and dates rather than parked in a backlog. The deliverable that makes the whole exercise real is a build sequence engineering can actually execute alongside feature work, this sprint, next sprint, this quarter. If a review does not end in committed engineering time, it was theater. This is the structure our AI Acceleration Sprint follows end to end.
For a focused team with access to code and vendors, phases one and two run about one to two weeks, phase three another one to two depending on feature surface, and phases four and five about a week. Call it four to six weeks of real elapsed time, which is exactly why waiting for a customer's security questionnaire to force the issue is so expensive. Under deal pressure, this work gets done badly, in a panic, with a prospect watching. Teams in the strongest position ran the review on their own clock, found their own exposures, and turned the results into a security story that accelerates sales instead of stalling them. Our pricing page breaks down what that looks like when someone else runs the sequence for you. One last note on scope. Resist the urge to review only the flagship AI feature. The marquee summarization tool usually gets scrutiny. The quiet integrations, an AI assisted search added last quarter, the internal tool support uses, an autocomplete an engineer wired up in an afternoon, usually do not. The serious finding is rarely in the feature everyone is proud of. It is in the one nobody remembered to mention.
Can our existing security team run this, or do we need AI specialists? A strong team can execute most of this with the structure above. Where outside depth pays off is phase three adversarial testing and recognizing what normal findings look like across many AI healthcare stacks, pattern recognition that shortens the whole cycle. Our FAQ page covers a few more questions teams ask before deciding to run this internally or bring in outside help. We already passed SOC 2. Does that cover us? SOC 2 evaluates controls against a framework that was not designed for LLM data flows. It is necessary and genuinely useful, and it will not catch over scoped prompts, retrieval contamination, or vendor retention misconfigurations. What if we find something serious mid review? Fix it immediately and keep going. Finding it yourself, first, is the entire point of running this on your own schedule instead of a customer's.
This checklist is the shape of the Resolve Health Tech AI Acceleration Sprint: six weeks, a first critical finding and fix underway within 14 days, and a 90 day build sequence a team can actually execute. Contact us to run this before a customer forces the issue.
Sana is a Technical Content Specialist at Resolve Health Tech. She specializes in breaking down complex architectural patterns, nearshore hiring trends, and software engineering workflows into actionable, human-friendly guides. Working alongside Resolve Health Tech's tech team, Sana ensures every piece of content is both highly readable and technically precise.