Healthcare AI compliance is a layered problem: classify the tool, lock down patient data access, and run continuous monitoring after deployment. Three actions matter right now. Classify every AI system in use against medical device criteria, enforce minimum necessary access for any AI agent touching patient records, and execute business associate agreements before protected health information ever reaches a vendor's model.
TL;DR:
- Most healthcare AI systems must be classified according to medical device regulations to determine their compliance obligations.
- Data protection laws like HIPAA and GDPR apply whenever patient information is processed, regardless of whether the AI tool is classified as a device.
- Continuous governance, including risk assessment, post-market monitoring, and lifecycle management, is essential for long-term AI safety and compliance.
- Proper vendor management requires signed Business Associate Agreements, documented security evidence, and independent verification of regulatory claims before data flow begins.
- Endpoint discovery and real-time monitoring of AI tools are necessary for effective oversight and preventing unmanaged PHI access in healthcare organizations.
Table of Contents
- What is healthcare AI compliance and which rules actually apply?
- When is an AI tool a regulated medical device?
- Data protection and privacy requirements for AI handling patient data
- Building lifecycle governance for AI in healthcare
- Technical controls that protect PHI and prove compliance
- Vendor management and contracting safeguards for AI suppliers
- When does 21 CFR Part 11 and GxP apply to AI systems?
- A 6–12 month roadmap to healthcare AI compliance readiness
- How endpoint discovery and runtime guardrails support these controls
- Where healthcare AI compliance is actually heading
- A practical next step for AI visibility and control
- Sources
- FAQ
What is healthcare AI compliance and which rules actually apply?
Healthcare AI compliance sits at the intersection of three regulatory traditions that rarely speak to each other: medical device law, data protection law, and general compliance programme governance. A single AI tool can trigger all three at once, depending on what it does and what data it touches. That's the trap most organisations fall into: they treat AI compliance as one checklist when it's actually three overlapping ones.
In the United States, the FDA regulates software that meets the definition of a medical device, and it has been doing so at scale for years. The agency has cleared or authorised over 1,000 AI and machine learning enabled medical devices as of March 2025, spanning radiology, cardiology and diagnostic support tools. That number tells you something important: a huge share of clinical AI already sits inside a known regulatory pathway, and the FDA has built specific guidance for AI/ML-based Software as a Medical Device (SaMD), including expectations around predetermined change control plans for adaptive algorithms.
In the European Union, the EU AI Act classifies most clinical AI as high-risk, which brings obligations around risk management, technical documentation, human oversight and post-market monitoring. The European Health Data Space adds another layer specifically for health data reuse, including rules on secondary use of patient data for AI training. National device regulators, such as the UK's MHRA or Australia's Therapeutic Goods Administration, run their own device classification and conformity assessment processes that often mirror, but don't always match, FDA or EU categories exactly.
Then there's data protection, which applies regardless of device status. HIPAA governs protected health information in the US; the HHS Office for Civil Rights makes clear that covered entities and business associates must protect electronic PHI whether a human or an algorithm is processing it. GDPR does the same job across the EU and treats health data as a special category requiring a stronger legal basis and often a Data Protection Impact Assessment.
The practical takeaway: classification decisions come first, because they determine almost everything downstream.
- FDA/device pathways apply when software's function meets the SaMD definition, regardless of the underlying AI architecture.
- EU AI Act high-risk rules apply broadly to health AI systems that influence clinical decisions or patient safety.
- HIPAA and GDPR apply whenever PHI or personal health data is processed, independent of device status.
- National regulators (MHRA, TGA and equivalents) run parallel but distinct classification and approval regimes.
A tool can clear every device hurdle and still create a serious compliance failure if the data-handling side was never assessed properly.
When is an AI tool a regulated medical device?
Not every clinical AI tool is a medical device, and getting this wrong in either direction creates real exposure. Deploy a device-grade tool without clearance and you risk enforcement action; wrongly assume a genuine device is exempt and you skip premarket evidence obligations and post-market surveillance duties entirely.
The core test hinges on intended use and clinical function. Software that analyses medical images, calculates risk scores that clinicians act on directly, or provides diagnostic or therapeutic recommendations tends to fall inside device definitions. Software that simply organises information for human review, without drawing clinical conclusions, often qualifies for clinical decision support exemptions, provided the clinician can independently review the basis for any recommendation.
Work through classification in this order:
- Document the intended purpose. Write down exactly what the tool does, who uses it, and what clinical decision it informs or automates. Vague purpose statements are the single biggest reason classification gets challenged later.
- Map the clinical workflow. Identify whether a clinician reviews the underlying data independently before acting, or whether the AI's output is presented as a final answer. This distinction is often the deciding factor for CDS exemptions.
- Assess the risk classification rationale. Grade the tool against the harm that would follow from an incorrect output, then record why you landed on that grade. Regulators and auditors both expect to see the reasoning, not just the conclusion.
- Decide device status and record the decision. Whichever way you land, write a formal classification memo with the rationale, the assumptions, and the date. This document becomes your primary defence if the classification is questioned during an audit or incident review.
- Plan for the consequences of your answer. A device classification means premarket evidence expectations, potential submission requirements, and an ongoing post-market surveillance obligation. A non-device classification doesn't exempt you from data protection or governance duties. It just changes which rulebook applies.
The National Commission's review of AI regulation in healthcare argues for exactly this kind of proportionate, lifecycle-based approach rather than a one-time premarket gate, and its recommendations push regulators toward stronger post-market surveillance requirements generally. If your organisation treats classification as a one-off paperwork exercise rather than a living decision revisited whenever the tool's function changes, you're already behind where regulators expect you to be.
Data protection and privacy requirements for AI handling patient data
HIPAA obligations don't disappear because an algorithm rather than a person is reading the chart. Covered entities and business associates must protect electronic PHI under the HIPAA Security Rule, and that duty extends fully to any AI vendor or platform touching that data. If a third party processes PHI on your behalf, you need a signed Business Associate Agreement before a single record flows, not after you've discovered the vendor doesn't have one.
Pro Tip: Treat "does this vendor have a signed BAA" as a hard gate in procurement, not a follow-up item. No signature, no data, no exceptions.
GDPR imposes a parallel but stricter regime for EU-linked patients. Health data is a special category under GDPR, requiring an explicit legal basis and, in most cases, a Data Protection Impact Assessment before deployment. The European Health Data Space adds specific constraints around secondary use of health data for AI training, meaning data collected for treatment can't automatically be repurposed to train a model without separate legal grounding.
A frequently missed risk sits outside device regulation entirely. Administrative AI tools, scheduling assistants, coding copilots, transcription tools, aren't medical devices, but they still handle PHI and carry real enforcement exposure. One review of compliance risks found that administrative AI tools remain subject to serious data-protection scrutiny even when they never touch a clinical decision, because PHI leakage and fraud exposure don't care whether the tool is classified as a device.
The technical answer to minimum necessary is attribute-based access control (ABAC) enforced at the operation level, not the system level. Rather than granting an AI agent broad database access and trusting policy documents to constrain it, ABAC binds each agent's permissions to the specific PHI fields its function actually requires. One analysis of layered AI compliance requirements makes the point plainly: policy without technical enforcement of minimum necessary is not compliance, it's a hope.
Several controls matter here in combination:
- Operation-level ABAC that scopes access per function, not per system.
- Tamper-evident audit logging that records every AI access to PHI, not just human access.
- De-identification applied with a clear understanding of re-identification risk, especially for smaller datasets or rare conditions.
- Breach notification procedures that explicitly cover AI-related incidents, including model output leaks.
- Cross-border data flow mapping wherever a vendor's infrastructure or subprocessors sit outside the patient's jurisdiction.
Get the GDPR and AI obligations wrong and you're not just exposed to HIPAA enforcement. You're carrying EU regulatory risk on top of it, often for the same underlying dataset.
Building lifecycle governance for AI in healthcare
Premarket clearance is a floor, not a finish line. A senior FDA advisor interviewed on the state of medical AI oversight put it directly: authorisation sets a minimum standard, and the organisation deploying the tool carries ongoing responsibility for monitoring, optimisation and safe use in its own clinical context. That's the single biggest mindset shift compliance teams need to make.
Building that lifecycle governance takes a defined structure, not good intentions.
- Stand up a multidisciplinary AI governance committee. Include clinical leadership, compliance, IT security, legal and a data science or biostatistics voice. A committee made up entirely of IT staff will miss clinical risk; one made up entirely of clinicians will miss technical drift.
- Run risk assessment before deployment, not after. Score each tool for clinical harm potential, data sensitivity and vendor dependency, and revisit that score whenever the tool's function or training data changes.
- Require clinical validation evidence proportionate to risk. A high-risk diagnostic tool needs stronger validation than a low-risk scheduling assistant, but both need something documented.
- Set deployment controls before go-live. Define who can approve production use, what rollback looks like, and who is accountable if the tool underperforms after launch.
- Run continuous monitoring, not a one-time check. Track output drift, demographic performance gaps and error rates on a schedule, not only when someone raises a complaint.
- Maintain post-market surveillance formally. Log incidents, near misses and performance anomalies in a system built for audit, not scattered across emails.
Model drift is the quiet failure mode nobody budgets for. A diagnostic model validated on one population can degrade silently as patient demographics, imaging equipment or clinical protocols shift over eighteen months. The HAARF framework for healthcare AI agents proposes exactly this kind of continuous verification, including ongoing equity monitoring across patient subgroups, as a standard governance category rather than an optional add-on.
Pro Tip: Write your predetermined change control plan before your first model update, not after. Deciding retraining thresholds under pressure, mid-incident, is how governance programmes quietly fall apart.
Evidence collection ties the whole lifecycle together. Every risk assessment, validation report, monitoring log and incident review needs to be retrievable on demand, because regulators increasingly ask not just "was this tool safe at launch" but "how do you know it's still safe now."
Technical controls that protect PHI and prove compliance
Policy documents don't stop a misconfigured AI agent from pulling more patient data than its function requires. Technical controls do, and regulators increasingly expect to see them, not just read about them in a policy binder.
Operation-level ABAC remains the practical foundation. Rather than granting a coding assistant or clinical summarisation tool blanket read access to a patient record, ABAC scopes each permission to the exact fields the function needs, binding access to purpose rather than convenience. That single design decision closes most of the gap between "the policy says minimum necessary" and "the system actually enforces it."
Tamper-evident audit trails come next. Every AI interaction with PHI, every query, every output, every access attempt that gets denied, needs to be logged in a form that can't be quietly edited after the fact. Auditors and regulators treat gaps in AI audit logs the same way they treat gaps in human access logs: as a finding, not an oversight.
Several other controls round out a defensible technical posture:
- Data loss prevention tooling that catches PHI heading into an unsanctioned AI tool or an external API call before it leaves the environment.
- Encryption at rest and in transit for any data an AI system touches, including intermediate outputs and cached prompts.
- Endpoint-level discovery and monitoring of AI tools in active use, including browser extensions and copilots that IT never formally provisioned.
- Secure integration patterns, airlocks and middleware that mediate what an AI agent can actually call or touch, rather than granting direct database access.
- Least-privilege connectors for any AI agent using Model Context Protocol or similar integration standards, so a compromised or misbehaving agent can't cascade into systems it was never meant to reach.
Pro Tip: If you can't answer "which AI tools touched PHI last week and what did they access" within a few minutes, your technical controls have a visibility gap, not just a policy gap.
Telemetry and log integration with your existing SIEM or SOAR stack matters as much as the controls themselves. An audit trail that lives in a vendor's isolated dashboard, disconnected from your security operations centre, is far less useful during an incident than one that flows into the same monitoring pipeline as every other endpoint event. Interoperability between your AI governance layer and your existing security infrastructure is what turns compliance documentation into something you can actually act on in real time.
Vendor management and contracting safeguards for AI suppliers
A vendor's compliance claims are only as good as the evidence backing them, and too many organisations discover that gap after PHI has already started flowing. Treat vendor onboarding as a gate, not a formality.
The BAA comes first, always. No signed Business Associate Agreement means no PHI, full stop, regardless of how confident the sales conversation sounds. This is one of the few genuinely non-negotiable rules in healthcare AI compliance, and it's also one of the most commonly skipped under deployment pressure.
Beyond the BAA, request specific evidence rather than accepting general assurances:
- Documented security posture, including penetration testing results and any relevant certifications, not just a marketing summary.
- Traceable evidence for any regulatory claim the vendor makes, such as an FDA clearance number you can independently verify against the FDA's own database.
- Clear data flow diagrams showing exactly where PHI travels, including subprocessors and any offshore infrastructure.
- Change control commitments describing how the vendor handles model updates and whether you get advance notice before behaviour shifts.
- Audit rights written into the contract, not offered informally, so you can verify claims rather than take them on faith.
- Incident response obligations specifying notification timelines if the vendor suffers a breach involving your data.
Indemnity clauses deserve particular attention for high-risk clinical tools. If a vendor's model contributes to a clinical error, your contract should specify where liability sits rather than leaving that question for litigation to sort out after the fact.
Vendor claims about regulatory status are worth independently checking, not just filing. A vendor stating "FDA cleared" should be able to point you to the specific clearance record, and you should look it up yourself. It takes minutes and catches a meaningful number of overstated claims before they become your problem.
When does 21 CFR Part 11 and GxP apply to AI systems?
Part 11 and GxP requirements attach to specific contexts, not to AI generally, and knowing where that line sits saves you from either over-engineering compliance or missing it entirely. The clearest triggers are clinical trial data management, manufacturing quality records, and any system that's part of a regulated quality management process, laboratory information systems, batch record systems and similar infrastructure.
Where those triggers apply, adaptive AI systems need validation adapted for their nature rather than a static software validation checklist copied from legacy systems.
- Define acceptance criteria before validation starts. Specify what performance level counts as passing, and record it before you run the validation, not after you see the results.
- Run IQ/OQ/PQ appropriately for AI. Installation and operational qualification look similar to traditional systems, but performance qualification for an adaptive model needs ongoing evidence, not a single test cycle.
- Set retraining thresholds that trigger re-validation. Decide in advance what magnitude of model update counts as a Part 11-relevant change requiring documentation and re-validation, rather than deciding case by case under deadline pressure.
- Preserve audit trail integrity across model versions. Every retraining event, parameter change and deployment needs a traceable record linked to electronic signature controls where regulated records are involved.
- Document who signs off on AI-generated regulated records. Electronic signature requirements don't disappear because a model, not a person, drafted the initial output.
Getting this wrong usually looks the same way twice: teams either apply full GxP rigour to administrative tools that never needed it, wasting resources, or they let a genuinely regulated system slide because "it's just AI" felt like an exemption. Neither call holds up under audit.
A 6–12 month roadmap to healthcare AI compliance readiness
Compliance readiness for healthcare AI comes together in three phases, gating items you can't skip, mid-term controls that build the governance muscle, and longer-term priorities that keep the whole system honest over time.
- Weeks 1 to 4: complete inventory and classification. You can't govern AI tools you don't know exist, including browser copilots and unsanctioned assistants staff have quietly adopted. Classify each one against device criteria and record the rationale.
- Weeks 1 to 4: execute outstanding BAAs. Audit every vendor with any PHI access and close the gap on any missing agreement immediately, treating this as the single highest-priority gating item.
- Weeks 4 to 8: enforce minimum necessary technically. Move from policy language to operation-level ABAC enforcement for every AI agent with PHI access.
- Days 90 to 180: stand up the governance committee. Bring clinical, compliance, security and legal voices together with a defined charter and decision rights.
- Days 90 to 180: build monitoring pipelines. Set up drift detection, output logging and periodic performance review against your original validation baseline.
- Days 90 to 180: collect validation evidence systematically. Centralise documentation so it's retrievable in hours, not weeks, when an auditor asks.
- Months 6 to 12: formalise post-market surveillance. Move from ad hoc incident review to a structured programme with defined thresholds for escalation.
- Months 6 to 12: run equity monitoring across patient subgroups. Check whether performance holds steady across demographics, not just in aggregate.
- Months 6 to 12: engage regulators proactively. Early conversations with device regulators or data protection authorities tend to go better than reactive ones prompted by an incident.
| Phase | Priority action | Owner |
|---|---|---|
| Weeks 1 to 4 | Inventory and classify every AI tool | Compliance + IT security |
| Weeks 1 to 4 | Execute missing BAAs | Legal + procurement |
| Weeks 4 to 8 | Enforce minimum necessary via ABAC | Technology implementers |
| Days 90 to 180 | Stand up governance committee | Compliance leadership |
| Days 90 to 180 | Build drift monitoring pipelines | Data science + IT |
| Months 6 to 12 | Formalise post-market surveillance | Governance committee |
| Months 6 to 12 | Engage regulators proactively | Compliance + legal |
How endpoint discovery and runtime guardrails support these controls
None of the controls above work if you can't see the AI tools actually running across your fleet. That's the operational gap most healthcare organisations underestimate: staff adopt copilots, browser assistants and AI-enabled extensions long before procurement or compliance ever hears about them. An operational playbook for AI tool discovery makes the point directly: you can't govern what you can't see, and discovery is the precondition for every other control in this article, from BAAs to minimum necessary enforcement.
This is the capability class Alectura works in, discovery, inventory, monitoring and guardrail enforcement for AI running at the endpoint, including copilots, IDE assistants and Model Context Protocol integrations. Mapped against the controls covered earlier, this class of capability supports:
- Discovering unsanctioned AI tools before they become an unmanaged PHI exposure.
- Enforcing access boundaries functionally equivalent to operation-level ABAC, scoped to what each tool actually needs to touch.
- Generating tamper-evident audit trails for AI activity at the point where it happens, not after the fact.
- Feeding DLP-style detection for sensitive data heading toward an AI tool or integration.
- Supporting incident triage by giving compliance and security teams a timeline of what an AI tool accessed and when.
Endpoint-level visibility doesn't replace the governance committee, the BAA, or the classification memo. It gives all three something real to act on, instead of a policy that assumes staff behaviour matches the compliance handbook.
Where healthcare AI compliance is actually heading
Regulation is drifting away from a one-time premarket gate and toward system-level, lifecycle oversight, and that shift is going to reshape what "compliant" even means over the next few years. Premarket clearance was never designed to catch model drift eighteen months into deployment, and regulators know it. The direction of travel, from the FDA's own predetermined change control expectations to the National Commission's lifecycle recommendations, points squarely toward continuous monitoring and real-world evidence as the actual proof of ongoing safety.
Here's what I think most compliance teams still underweight: multidisciplinary governance isn't a nice-to-have structure for large hospital systems with spare headcount. It's the mechanism that catches the failure modes premarket testing structurally can't see, demographic performance gaps, workflow drift, subtle misuse. Organisations that wait for a regulator to demand this structure will be building it under duress, with an incident already on the record.
My advice is blunt: engage with regulators before you're forced to, and keep your evidence audit ready at all times, not assembled retroactively when someone asks. The gap between organisations that treat compliance as ongoing infrastructure and those treating it as a launch checklist is going to widen, not narrow.
— Nathan
A practical next step for AI visibility and control
Security and compliance teams need visibility into which AI tools are actually running across endpoints, including the copilots and browser assistants nobody formally approved. Where a traditional security stack sees a healthy device, some advanced tools provide insight into what AI is running on it, what data it can reach, and what it's connecting to through MCP and similar integrations.
That visibility feeds directly into the checklist covered above: inventory and classification, minimum necessary enforcement, tamper evident audit logging, and DLP style detection for PHI heading toward an unsanctioned tool. It's built for the security teams responsible for proving these controls exist, not just describing them in a policy document.
Vendor claims still deserve independent verification, always request the evidence, never take a compliance label at face value. If you're mapping your own AI tool landscape against the controls in this article, start with Alectura's AI security glossary to get the terminology straight, then look at the agentic endpoint security category page to see how discovery and runtime guardrails fit into your existing security stack.
Sources
- Artificial Intelligence and Machine Learning (AI/ML)-Enabled Software as a Medical Device (SaMD) Action Plan
- National Commission into the regulation of AI in healthcare: recommendations for a future regulatory framework
- HIPAA for professionals (HHS OCR)
This article is general information, not a substitute for advice from a qualified doctor. Consult a qualified healthcare professional about your own circumstances before acting on anything here.
FAQ
What is the 30% rule in AI?
There's no established "30% rule" in AI regulation or healthcare compliance frameworks. If you've seen this term used somewhere, it isn't a recognised regulatory standard from the FDA, EU AI Act or HIPAA, treat it with caution and check the primary source.
What are the compliance requirements for AI use in healthcare?
Requirements are layered: device classification under FDA or equivalent national regulators where applicable, data protection under HIPAA or GDPR wherever patient data is processed, and organisational governance covering risk assessment, monitoring and audit trails throughout the tool's lifecycle.
Is any AI HIPAA compliant?
No AI tool is inherently HIPAA compliant on its own. Compliance depends on how the vendor and the deploying organisation implement safeguards, access controls and a signed Business Associate Agreement, not on the underlying model itself.
Is it safe to use AI in healthcare?
Safety depends on governance, not on the technology alone. Tools with documented classification, clinical validation and continuous post-market monitoring carry materially lower risk than tools deployed without lifecycle oversight, which is the core argument behind lifecycle-based regulatory recommendations.
How do organisations know which AI tools even need governance?
Discovery comes first. Endpoint-level visibility into AI tools in active use, including unsanctioned copilots and browser assistants, is what allows compliance teams to classify, contract and monitor tools that policy alone would never surface.