Clinical decision support, medical devices, diagnostic AI.
Definition & scope
The cross-jurisdiction picture below shows how each of 45 tracked instruments treats this topic. The patterns vary substantially — and 38 regimes are silent, leaving gaps that future policy work could address.
Regulatory approaches
The instruments that reach clinical AI do so through structurally different modalities rather than a single shared mechanism. The EU AI Act operates two parallel high-risk gateways for this topic. Under Article 6(1), an AI system that is a medical device, or a safety component of one, and that already requires third-party conformity assessment under the Medical Device Regulation (Reg. (EU) 2017/745) or IVDR (Reg. (EU) 2017/746) is high-risk by operation of law — typically MDR Class IIa and above, or IVDR Class B and above, where a notified body is involved (EU-AIA-2024 Art. 6(1); MDCG 2025-6, 19 June 2025). Separately, Article 6(2)/Annex III §5(a) captures AI used to evaluate eligibility for essential public services including healthcare. The MDCG/AI Board joint guidance MDCG 2025-6 confirms these AI Act duties (data governance, logging, human oversight, transparency) are to be discharged within the existing MDR/IVDR conformity-assessment procedure rather than through a parallel certification (MDCG 2025-6). The United States, by contrast, governs through the device-authorization pathways the FDA already administers — predominantly 510(k) substantial-equivalence, with De Novo and PMA for higher-risk or novel devices (21 U.S.C. §360c) — supplemented by sub-regulatory guidance; comparative work mapping 222 US- and 240 EU-approved AI/ML devices shows these regimes diverge enough that, of 124 devices approved in both, 80 reached Europe first 1. Scholars further argue that regulating such adaptive software product-by-product is itself a limitation, urging a "system view" that covers human-AI interaction and organizational context 2. The UK delegates to the MHRA's software-as-a-medical-device regime under its principles-based white-paper approach (UK-WHITEPAPER-2023) (MHRA 2024). Principles instruments (OECD, G7 Hiroshima) and most frontier-safety frameworks remain silent on the sector entirely.
Key fault lines
Several questions remain genuinely contested across jurisdictions and the literature. First is definitional reach: whether general-purpose large language models that generate clinical advice should be regulated as medical devices. Peer-reviewed analyses show such systems "readily produced device-like decision support across a range of scenarios" and should fall under device frameworks if clinically deployed 3, with others stressing that the urgent safeguard is regulators enforcing the rules already on the books 4 — while warning existing pathways were not designed for general-purpose generative models 5; yet no major regime has squarely classified standalone medical chatbots, and a separate hazard — fabricated but fluent output — compounds the risk 6. The contrast is sharpened by SB 243 (California), which regulates companion-chatbot self-harm protocols via consumer-protection law (Cal. Bus. & Prof. Code §22602(b)) rather than device law. Second is the validation paradigm: whether one-time pre-market validation suffices for models whose performance drifts, with researchers contending external validation "does not guarantee generalizability" and proposing recurring local validation instead 7. Third is the regulatory philosophy itself — the EU's ex-ante, risk-tiered categorization versus the US sectoral, predominantly ex-post and product-authorization model versus the UK/OECD principles-delegation approach — the very fork Policy Window flags as the topic's locus of disagreement. Underlying all three is an evidence asymmetry: audits find most authorized devices were evaluated only retrospectively and rarely report demographic performance 8, a concern made concrete by a widely used US care-management algorithm found racially biased because it predicted cost rather than illness 9, leaving open whether any of these modalities demonstrably reduces patient harm.
Trajectory — what is changing
The governance picture for clinical AI is shifting quickly, and several dated developments postdate or refine the coverage cells above. On 18 January 2024 the WHO issued ethics-and-governance guidance specific to large multi-modal models in health, with over 40 recommendations including mandatory post-deployment audits and independent impact assessments (WHO 2024, Ethics and governance of AI for health: guidance on LMMs); the WHO/ITU Global Initiative on AI for Health has since set out priorities to harmonize these standards across UN bodies 10. On 3 December 2024 the FDA finalized its guidance on a Predetermined Change Control Plan, letting manufacturers pre-specify and pre-authorize future model modifications without a new marketing submission — a structural answer to the adaptive-model problem (FDA 2024, final PCCP guidance) whose underlying algorithm-change-protocol mechanism the literature had earlier mapped for continuously-learning devices 11, and which scholars argue must be paired with active post-market governance for drift 12. The authorized base has grown accordingly: FDA's public list passed roughly 1,000 AI-enabled device authorizations in late 2024 and exceeded 1,250 by mid-2025, with radiology consistently about 75% of clearances, though most still clear via 510(k) with limited clinical validation and poor transparency 13. In the EU, the MDCG/AI Board issued joint interplay guidance MDCG 2025-6 on 19 June 2025, and on 19 November 2025 the Commission's "Digital Omnibus" proposed postponing high-risk obligations for Annex I products — including medical devices — to 2 August 2028 (European Commission, Digital Omnibus proposal, 19 Nov 2025). These changes are pending or guidance-level, not yet reflected as new binding coverage cells.
Coverage across jurisdictions
Historical primacy & cross-jurisdiction tension
First addressed by UNESCO Recommendation on the Ethics of Artificial Intelligence on (governs). Subsequent regimes have either codified, diverged from, or remained silent on this baseline.
- Forum-shoppingEU AI Act↔Executive Order 14179 — Removing Barriers to American Leadership in AI
- Forum-shoppingUNESCO Recommendation on the Ethics of Artificial Intelligence↔Interim Measures for Generative AI Service Management
- Forum-shoppingItaly Law No. 132/2025 on Artificial Intelligence (Legge 23 settembre 2025, n. 132)↔G7 Hiroshima AI Process Code of Conduct
Compare jurisdictions: EU vs US · EU vs UK · EU vs CN
Enforcement & impact
Silent regimes — gap signal
Instruments that do not address AI in Healthcare — candidates for future policy work.
- Executive Order 14179 — Removing Barriers to American Leadership in AIUS
- Interim Measures for Generative AI Service ManagementCN
- G7 Hiroshima AI Process Code of ConductG7
- OECD AI Principles (Recommendation)OECD
- Council of Europe Framework Convention on AIcouncil_of_europe
- UN GA Resolution on Safe, Secure, Trustworthy AIUN
- NIST AI Risk Management FrameworkUS
- Bletchley Declaration on AI Safetyglobal
- Seoul Declaration on Safe, Innovative and Inclusive AIglobal
- NIST AI RMF Generative AI ProfileUS
- California SB-1047: Safe and Secure Innovation for Frontier AI Models ActUS
- India Digital Personal Data Protection Act + AI Advisory (MEITY)IN
- Brazil AI Bill (PL 2338/2023)BR
- ASEAN Guide on AI Governance and EthicsASEAN
- African Union Continental AI StrategyAfrican_Union
- Anthropic Responsible Scaling Policy (RSP) v2US
- OpenAI Preparedness FrameworkUS
- Google DeepMind Frontier Safety FrameworkUS
- Meta Frontier AI FrameworkUS
- UK-US AI Safety Institute Memorandum of Understandingglobal
- White House Voluntary AI CommitmentsUS
- Singapore Model AI Governance Framework for Generative AISG
- Japan METI AI Guidelines for BusinessJP
- General Data Protection Regulation (GDPR)EU
- EU General-Purpose AI Code of PracticeEU
- GSA Generative AI and Specialized Computing Infrastructure Acquisition Resource GuideUS
- DoD Responsible AI Strategy and Implementation PathwayUS
- FedRAMP AI Cloud Procurement GuidanceUS
- DFARS Subpart 252.204 (Safeguarding Covered Defense Information and Cyber Incident Reporting)US
- California SB-53: Transparency in Frontier Artificial Intelligence Act (TFAIA)US
- California SB 942: AI Transparency ActUS
- Revised Product Liability Directive (Directive (EU) 2024/2853)EU
- Directive (EU) 2024/2831 on improving working conditions in platform workEU
- Provisions on the Administration of Deep Synthesis of Internet Information ServicesCN
- New York RAISE Act: Responsible AI Safety and Education ActUS
- TAKE IT DOWN Act (Tools to Address Known Exploitation by Immobilizing Technological Deepfakes on Websites and Networks Act)US
- Japan AI Promotion Act (Act on the Promotion of Research, Development and Utilization of AI-Related Technologies)JP
- UN Global Digital CompactUN
See also
Further reading
13 academic & grey-literature sources bearing on this topic — catalogued metadata with a primary link; one-line findings are ✦ AI-generated summaries, labeled as such (charter §7.9). Browse the full literature index.
- Current state of Food and Drug Administration-approved artificial intelligence/machine learning medical devices: pathways, transparency, and evidence gaps Peer-reviewed✦ AIDocuments that most FDA AI/ML devices clear via the 510(k) pathway with limited clinical validation and poor transparency, exposing regulatory evidence gaps.
- Unregulated large language models produce medical device-like output Peer-reviewed✦ AIFinds general-purpose LLMs 'readily produced device-like decision support across a range of scenarios,' implying they should fall under medical-device regulation if clinically deployed.
- A general framework for governing marketed AI/ML medical devices Peer-reviewed✦ AIProposes a post-market governance framework for AI/ML medical devices addressing performance drift and ongoing monitoring beyond initial approval.
- Global Initiative on AI for Health (GI-AI4H): strategic priorities advancing governance across the United Nations Peer-reviewed✦ AISets out the WHO/ITU Global Initiative on AI for Health's strategic priorities to harmonize international regulatory and governance standards for health AI.
- A future role for health applications of large language models depends on regulators enforcing safety standards Peer-reviewed✦ AIArgues medical LLMs are likely device-like clinical decision support and that 'the urgent need to enforce existing regulations' is the key safeguard against unsafe deployment.
- External validation of AI models in health should be replaced with recurring local validation Peer-reviewed✦ AIContends external validation 'does not guarantee generalizability' and proposes recurring local validation as the safer regulatory paradigm for clinical AI.
- The imperative for regulatory oversight of large language models (or generative AI) in healthcare Peer-reviewed✦ AICalls for a new regulatory category/oversight for medical LLMs, warning existing device frameworks were not designed for general-purpose generative models.
- How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals Peer-reviewed✦ AIAudit of 130 FDA-approved medical AI devices finds evaluation gaps — mostly retrospective, scant multi-site testing — "that can mask vulnerabilities of devices when they are deployed on patients".
- Approval of artificial intelligence and machine learning-based medical devices in the USA and Europe (2015–20): a comparative analysis Peer-reviewed✦ AIMaps 222 US- and 240 EU-approved AI/ML medical devices (2015–20); of 124 approved in both regions, 80 were first approved in Europe — grounding pathway-stringency debates.
- Algorithm Change Protocols in the Regulation of Adaptive Machine Learning-Based Medical Devices Peer-reviewed✦ AIAnalyzes the SaMD prespecification and algorithm change protocol mechanism (FDA predetermined change control) for governing continuously-learning medical-device algorithms.
- The need for a system view to regulate artificial intelligence/machine learning-based software as medical device Peer-reviewed✦ AIArgues regulators of adaptive AI/ML medical software must shift from a product-centric approach to "a system view" covering human-AI interaction and organizational context.
- Dissecting racial bias in an algorithm used to manage the health of populations Peer-reviewed✦ AIA widely used US care-management algorithm is racially biased — "at a given risk score, Black patients are considerably sicker" — because it predicts costs, not illness.
- Hallucination PreprintJi, Z., et al. (2023), 'Survey of Hallucination in Natural Language Generation,' ACM Computing Surveys 55(12): 1-38.
References
Sources cited inline in the analysis (linked from the superscript markers), then the primary instrument sources behind the classifications.
- Muehlematter, Daniore & Vokinger (2021) Approval of artificial intelligence and machine learning-based medical devices in the USA and Europe (2015–20): a comparative analysis, The Lancet Digital Health. 10.1016/S2589-7500(20)30292-2 — Maps 222 US- and 240 EU-approved AI/ML medical devices (2015–20); of 124 approved in both regions, 80 were first approved in Europe — grounding pathway-stringency debates. ↩
- Gerke, Babic, Evgeniou & Cohen (2020) The need for a system view to regulate artificial intelligence/machine learning-based software as medical device, npj Digital Medicine. 10.1038/s41746-020-0262-2 — Argues regulators of adaptive AI/ML medical software must shift from a product-centric approach to "a system view" covering human-AI interaction and organizational context. ↩
- Gary E. Weissman, Toni Mankowitz, Genevieve P. Kanter (2025) Unregulated large language models produce medical device-like output, npj Digital Medicine. 10.1038/s41746-025-01544-y — Finds general-purpose LLMs 'readily produced device-like decision support across a range of scenarios,' implying they should fall under medical-device regulation if clinically deployed. ↩
- Oscar Freyer, Isabella Catharina Wiest, Jakob Nikolas Kather, Stephen Gilbert (2024) A future role for health applications of large language models depends on regulators enforcing safety standards, The Lancet Digital Health. 10.1016/S2589-7500(24)00124-9 — Argues medical LLMs are likely device-like clinical decision support and that 'the urgent need to enforce existing regulations' is the key safeguard against unsafe deployment. ↩
- Bertalan Meskó, Eric J. Topol (2023) The imperative for regulatory oversight of large language models (or generative AI) in healthcare, npj Digital Medicine. 10.1038/s41746-023-00873-0 — Calls for a new regulatory category/oversight for medical LLMs, warning existing device frameworks were not designed for general-purpose generative models. ↩
- Ji, Z., et al. (2023), 'Survey of Hallucination in Natural Language Generation,' ACM Computing Surveys 55(12): 1-38. Hallucination. arXiv:2202.03629 — Ji, Z., et al. (2023), 'Survey of Hallucination in Natural Language Generation,' ACM Computing Surveys 55(12): 1-38. ↩
- Alexey Youssef, Michael Pencina, Anshul Thakur, Tingting Zhu, David Clifton, Nigam H. Shah (2023) External validation of AI models in health should be replaced with recurring local validation, Nature Medicine. 10.1038/s41591-023-02540-z — Contends external validation 'does not guarantee generalizability' and proposes recurring local validation as the safer regulatory paradigm for clinical AI. ↩
- Wu, Wu, Daneshjou, Ouyang, Ho & Zou (2021) How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals, Nature Medicine. 10.1038/s41591-021-01312-x — Audit of 130 FDA-approved medical AI devices finds evaluation gaps — mostly retrospective, scant multi-site testing — "that can mask vulnerabilities of devices when they are deployed on patients". ↩
- Obermeyer, Powers, Vogeli & Mullainathan (2019) Dissecting racial bias in an algorithm used to manage the health of populations, Science. 10.1126/science.aax2342 — A widely used US care-management algorithm is racially biased — "at a given risk score, Black patients are considerably sicker" — because it predicts costs, not illness. ↩
- Vijaytha Muralidharan, Madelena Y. Ng, Shada AlSalamah, Sameer Pujari, et al. (WHO/ITU GI-AI4H) (2025) Global Initiative on AI for Health (GI-AI4H): strategic priorities advancing governance across the United Nations, npj Digital Medicine. 10.1038/s41746-025-01618-x — Sets out the WHO/ITU Global Initiative on AI for Health's strategic priorities to harmonize international regulatory and governance standards for health AI. ↩
- Stephen Gilbert, Matthew Fenech, Martin Hirsch, Shubhanan Upadhyay, Andrea Biasiucci, Johannes Starlinger (2021) Algorithm Change Protocols in the Regulation of Adaptive Machine Learning-Based Medical Devices, Journal of Medical Internet Research. 10.2196/30545 — Analyzes the SaMD prespecification and algorithm change protocol mechanism (FDA predetermined change control) for governing continuously-learning medical-device algorithms. ↩
- Boris Babic, I. Glenn Cohen, Ariel Dora Stern, Yiwen Li, Melissa Ouellet (2025) A general framework for governing marketed AI/ML medical devices, npj Digital Medicine. 10.1038/s41746-025-01717-9 — Proposes a post-market governance framework for AI/ML medical devices addressing performance drift and ongoing monitoring beyond initial approval. ↩
- Aditya Loganathan, Michael Friedman, Tayab Waseem, et al. (Andrew C. Meltzer, senior author) (2026) Current state of Food and Drug Administration-approved artificial intelligence/machine learning medical devices: pathways, transparency, and evidence gaps, Journal of Medical Artificial Intelligence. 10.21037/jmai-2025-196 — Documents that most FDA AI/ML devices clear via the 510(k) pathway with limited clinical validation and poor transparency, exposing regulatory evidence gaps. ↩
- EU-AIA-2024: Annex III §5(a) (high-risk: essential services) + MDR overlap
- US-EO-14110: §8 + HHS strategy
- UK-WHITEPAPER-2023: MHRA software-as-medical-device
- OMB-M-24-10: Attachment 1 examples include healthcare access decisions as rights-impacting; minimum practices apply
- CA-SB-243: Cal. Bus. & Prof. Code § 22602(b) (added by SB 243) — operator must maintain a protocol preventing production of suicidal-ideation/self-harm content and referring the user to crisis-service providers, published on its website; § 22603 reports referral data to the Office of Suicide Prevention
- UNESCO-AI-ETHICS-2021: Policy Area 'Health and Social Well-being', para 121 — employ effective AI for health and the right to life
- IT-AILAW-2025: Art. 7 — AI must not condition access to healthcare on discriminatory criteria (¶2); patient right to be informed of AI use (¶3); the therapeutic decision is always reserved to the physician (¶5). Arts. 8–10 add research, data-processing and electronic-health-record provisions.
Cite this article 8 formats · BibTeX, RIS, APA, Chicago, … · 1-click copy
Persistent identifier: https://policywindow.org/wiki/healthcare — committed-stable URL with content-versioning via ?asOf= (rollout pending per methodology §7). DOIs via Zenodo are on the roadmap.
Article tools — track changes, suggest an edit
View history — every captured revision of this article · What links here
7 instruments tracked.
Does governance work? — the social-science evidence
What the peer-reviewed social science shows: whether the harm this topic addresses is empirically real, and whether governance of it works. The badge is the epistemic status of the evidence(not the policy debate) — “thin” or “absent” efficacy evidence is itself a finding (the “second silence”). Each epistemic-status label is Policy Window's editorial assessment of the cited evidence base (a structured classification), not a verdict any single source issues.
Both the benefit and the harm of clinical AI are empirically real and well-documented, but outcomes are highly deployment-dependent. Rigorous prospective studies show genuine clinical value in narrow tasks — the MASAI RCT (>100,000 women) found AI-supported mammography detected ~20% more cancers (6.1 vs 5.1 per 1000 screened) at comparable recall rates (Lang et al. 2023, Lancet Oncology), and IDx-DR's pivotal trial achieved 87.2% sensitivity / 90.7% specificity for diabetic retinopathy (Abramoff et al. 2018, npj Digital Medicine) — yet widely deployed models can fail or harm: the Epic Sepsis Model, live at hundreds of US hospitals, scored AUC 0.63 with 33% sensitivity on external validation (Wong et al. 2021, JAMA Internal Medicine), and a population-health algorithm covering ~200M people understated Black patients' illness because it predicted cost not need (Obermeyer et al. 2019, Science). Honest caveat: there is no single 'AI in healthcare' effect — performance ranges from life-saving to dangerous depending on task, calibration, and whether the model was prospectively validated.
Sources: Lang K, Josefsson V, Larsson A-M, et al. 2023 (Lancet Oncology 24(8):936-944, MASAI trial clinical safety analysis; AI-supported screening detected 6.1 vs 5.1 cancers per 1000, ~20% higher, similar recall rates); Abramoff MD, Lavin PT, Birch M, Shah N, Folk JC. 2018 (npj Digital Medicine 1:39, IDx-DR pivotal trial; 87.2% sensitivity / 90.7% specificity); Wong A, Otles E, Donnelly JP, et al. 2021 (JAMA Internal Medicine 181(8):1065-1070, Epic Sepsis Model external validation; AUC 0.63, 33% sensitivity); Obermeyer Z, Powers B, Vogeli C, Mullainathan S. 2019 (Science 366(6464):447-453, racial bias from cost-as-proxy)
There is essentially no impact-evaluation evidence that the prevailing governance regime for medical AI — FDA authorization, predominantly via the 510(k) substantial-equivalence pathway — measurably reduces patient harm or improves outcomes. Analyses of authorized AI devices find that clinical validation is frequently absent or non-prospective (of 521 FDA-authorized AI devices, ~43% had no published clinical-validation data and only ~28% were prospectively validated; Chouffani El Fassi & Henderson et al. 2024) and that demographic performance is almost never reported (race/ethnicity in 3.6%, and only 9.0% of 692 510(k)/cleared AI devices carried a prospective post-market-surveillance study; Muralidharan et al. 2024). Earlier analysis of 130 cleared devices likewise found 97% were evaluated only retrospectively (Wu et al. 2021). The closest analogue evidence on the pathway itself is discouraging: the Institute of Medicine (2011) concluded the 510(k) process was not designed to assess safety and effectiveness — i.e., no direct study establishes that the rule, as written, prevents the harms it targets. Caveat: this is an absence of impact evaluation plus reporting-gap and design-critique evidence, not a study showing the regime fails to reduce harm.
Sources: Chouffani El Fassi S, Abdullah A, Fang Y, ... Henderson GE, et al. 2024 (Nature Medicine, 'Not all AI health tools with regulatory authorization are clinically validated', s41591-024-03203-3; 521 devices, ~43% no clinical validation, ~28% prospectively validated); Muralidharan V, Adewale BA, Huang CJ, et al. 2024 (npj Digital Medicine 7:273, scoping review of reporting gaps in 692 FDA-approved AI medical devices; race/ethnicity 3.6%, prospective post-market surveillance 9.0%); Wu E, Wu K, Daneshjou R, Ouyang D, Ho DE, Zou J. 2021 (Nature Medicine 27:582-584, analysis of 130 FDA approvals; 97% retrospective-only evaluation); Institute of Medicine 2011 (Medical Devices and the Public's Health: The FDA 510(k) Clearance Process at 35 Years)