Irresponsible Use of Artificial Intelligence by Medical Professionals in Clinical Diagnostics: Patient Safety Risks, Systemic Failures, and the Imperative for Regulatory Reform

Introduction

Artificial intelligence is reshaping clinical medicine at unprecedented speed. The global AI-in-healthcare market, valued at approximately USD 22.4 billion in 2023, is projected to exceed USD 188 billion by 2030, driven substantially by diagnostic applications (Grand View Research, 2024). Yet the pace of adoption has dramatically outstripped the development of the evidentiary standards, governance frameworks, and professional competencies required to ensure these tools are used safely and equitably.

The result is a documented pattern of irresponsible AI use in clinical diagnostics: the uncritical acceptance of AI-generated outputs by medical professionals without adequate validation, clinical contextualisation, or independent professional scrutiny. This is evidenced by fabricated diagnoses entering NHS patient records, AI blood test tools missing common haematological conditions, wound assessment systems identifying the correct primary diagnosis in fewer than one in three cases, and systematic over-trust of AI in imaging producing preventable misdiagnoses (Fortune, 2025; WellnessPulse, 2025; NCBI PMC12615213, 2025).

The problem is not AI per se. Appropriately validated, transparently governed AI tools have demonstrated genuine diagnostic utility. The problem is the conditions of adoption: without mandatory pre-market validation standards, without professional training in AI critical appraisal, without clear medico-legal accountability, and without the post-market surveillance infrastructure to identify and remediate failures. This article presents a comprehensive, evidence-based analysis of the patient safety risks, professional conduct failures, structural contributing factors, and governance deficiencies arising from this irresponsible adoption, and advances a framework of reform recommendations.

Background and Conceptual Framework

The Promise and the Reality Gap

In controlled benchmarks, AI diagnostic tools have demonstrated impressive performance, expert-level diabetic retinopathy detection (Gulshan et al., 2016) and dermatologist-comparable skin cancer classification (Esteva et al., 2017). However, transfer of these findings to real-world clinical deployment consistently reveals a performance reality gap: a phenomenon known as distributional or domain shift, whereby models trained on homogenous benchmark datasets fail to generalise to the heterogeneous, diverse populations encountered in practice (Finlayson et al., 2021). This gap is not a technical footnote, it is the central safety problem with current AI diagnostic deployment.

Automation bias, the systematic tendency to defer to automated outputs irrespective of their accuracy, was first rigorously described in aviation contexts (Mosier & Skitka, 1996) and has since been extensively documented in clinical medicine. Goddard et al. (2012) demonstrated that AI-generated diagnostic flags significantly altered clinician interpretation behaviour regardless of AI accuracy. More recently, controlled studies found that clinicians take 41% longer to identify errors in AI-generated outputs than equivalent human-generated errors (NCBI PMC12321131, 2025), a temporal cost with direct implications for time-critical diagnoses. Automation bias is sustained by time pressures, the framing of AI as an authoritative source, liability concerns, and the near-total absence of AI critical appraisal training in medical education.

Evidence of Diagnostic AI Failure

Blood Test and Laboratory Interpretation

Blood test interpretation is foundational to clinical medicine and requires contextual sensitivity to detect not only pathological values but also pre-analytical errors such as haemolysis or sample contamination. Evaluations of large language model (LLM)-based tools, including ChatGPT derivatives, found systematic failure to identify iron-deficiency anaemia, hypercholesterolaemia, and laboratory processing errors in presented clinical scenarios. Clinical reliability scores averaged 2–4 out of 10 across multiple assessors (WellnessPulse, 2025). Tools additionally failed to generate appropriate specialist referral recommendations, a particularly dangerous omission in primary care settings where blood investigations drive onward pathways. These findings are consistent with the broader literature documenting that general-purpose LLMs cannot replicate the contextual clinical reasoning required for safe laboratory interpretation (Omiye et al., 2023).

Diagnostic Imaging and Demographic Inequity

AI imaging tools have demonstrated performance disparities that disproportionately harm underrepresented populations. Pneumonia detection AI trained predominantly on data from urban, high-resource institutions, produced false-negative rates 23% above baseline for rural cohorts (NCBI PMC12615213, 2025). In dermatology, multiple published studies confirmed systematically elevated melanoma false-negative rates for patients with darker skin tones, a direct consequence of training dataset underrepresentation (Daneshjou et al., 2022; Adamson & Smith, 2018). These are not marginal statistical differences: they represent clinically significant missed diagnoses in populations with the fewest alternative diagnostic pathways and the greatest historical barriers to specialist care. Obermeyer et al. (2019) quantified an analogous mechanism racially biased risk stratification algorithms systematically denying Black patients access to care management establishing the measurable reality of AI-mediated health inequity.

Wound Assessment and Clinical Decision Support

AI-enabled wound assessment tools, including implementations using Microsoft Copilot, correctly ranked the primary diagnosis first in only 30% of evaluated clinical cases. In wound management, first-ranked differentials drive immediate treatment decisions, antibiotic selection, surgical planning, offloading strategy. A 70% probability of the primary diagnosis not being listed first represents an unacceptable error burden with direct consequences for antimicrobial stewardship and surgical outcomes.

AI-Generated Clinical Documentation

Ambient AI scribes, deployed at scale across U.S.health systems to generate clinical notes from recorded encounters, have raisedserious accuracy and governance concerns. Documented deficiencies includemisattribution of clinical statements, omission of clinically significantinformation, and introduction of factual errors into records that subsequentlypropagate through multi-provider care systems. Many systems were deployedwithout full regulatory pre-market review under FDA Clinical Decision Supportexemptions, and without systematic HIPAA compliance evaluation for audiocapture practices (Bipartisan Policy Center, 2025; Salvi Law, 2025).

Documented Clinical Incidents

‍

Contributing Structural Factors

‍

Patient Safety and Health Equity Implications

Direct Patient Harm

Irresponsible AI diagnostic use harms patients through three principal pathways. First, direct misdiagnosis: false-negative outputs leave conditions undetected and untreated; false-positive outputs generate unnecessary investigations and procedures with iatrogenic risk profiles. In oncology, cardiovascular disease, and infectious disease, the temporal consequences of missed diagnosis are measured in disease progression and mortality. Second, cascade effects: AI-generated diagnoses entering clinical records without human verification propagate through multi-provider systems, informing prescribing, referral, and surgical decisions made by clinicians who trust the documented diagnosis without knowledge of its AI provenance, as illustrated by the NHS Annie incident. Third, unnecessary procedural risk: false-positive AI findings order invasive investigations with independent complication profiles.

Health Equity: Amplification of Diagnostic Disparities

AI diagnostic failures are not distributed randomly across patient populations, they fall disproportionately on those already facing the greatest diagnostic inequities. Training data underrepresentation produces systematically worse AI performance for rural, elderly, lower-income, and ethnically diverse patients. Obermeyer et al. (2019) demonstrated with quantitative precision that a widely deployed health risk algorithm produced racially biased outputs that denied Black patients’ equivalent access to care management. Without mandatory demographic performance disaggregation in AI validation and post-market monitoring, clinical AI will reproduce and amplify historical diagnostic inequities at scale, affecting the communities least equipped to access alternative diagnostic pathways.

Legal and Institutional Exposure

The legal responsibility for clinical diagnosis rests unambiguously with the qualified practitioner, not the AI tool (DJ Holt Law, 2025). This creates a corresponding professional duty to independently verify AI outputs, a duty that, where demonstrably unmet and harm results, supports professional negligence claims of growing viability (Salvi Law, 2025). Institutions deploying AI tools without documented governance, validation, and training protocols face parallel organisational liability exposure as regulatory frameworks tighten and enforcement precedent develops.

Governance & Reform Recommendations

‍

Discussion

Three overarching themes warrant emphasis. First, the accountability diffusion problem: when AI-related diagnostic errors occur, responsibility is dispersed across developers, regulators, institutions, and clinicians, a diffusion that enables systemic risk without systemic accountability. Clearer legislative frameworks assigning specific, non-diffusable responsibilities to each actor category are essential to creating the incentive structures that sustainable AI safety requires.

Second, the innovation-safety tension is real but resolvable. Stringent pre-market requirements will not halt beneficial AI deployment; they will ensure that what is deployed has been validated for the populations in which it will be used. The evidentiary standard proposed here is not novel, it is the standard already applied to pharmacological interventions and Class III medical devices. No principled basis exists for applying a lower standard to AI diagnostic tools whose errors carry equivalent patient harm potential.

Third, the medical profession has an independent ethical obligation, grounded in beneficence, non-maleficence, and professional accountability, to resist the institutional and time pressures that drive uncritical AI adoption. The progressive erosion of independent clinical diagnostic reasoning, documented as a consequence of AI over-reliance in training environments (NCBI PMC12321131, 2025), represents a long-term resilience risk that the profession must actively counter by preserving unassisted diagnostic reasoning as a core and assessed competency.

Conclusion

The irresponsible use of artificial intelligence in clinical diagnostics is an established, documented, present-day patient safety threat, not a future risk. Fabricated NHS diagnoses, AI blood test tools with reliability scores of 2–4/10, wound assessment systems correct only 30% of the time, and imaging AI that systematically under-diagnoses in underrepresented populations are not anomalies. They are the foreseeable consequences of deploying AI without the governance infrastructure that its risk profile demands.

The medical profession, regulatory agencies, healthcare institutions, and AI developers share a collective responsibility to establish the conditions under which AI can be used in clinical diagnostics safely, transparently, equitably, and accountably. The governance standards detailed in this article, bias auditing, explainability requirements, mandatory validation protocols, liability clarity, and AI literacy training, are technically and institutionally achievable. The deficit is not knowledge; it is urgency.

‍

References

1. Adamson, A. S., & Smith, A. (2018). Machine learning and health care disparities in dermatology. JAMA Dermatology, 154(11), 1247–1248. https://doi.org/10.1001/jamadermatol.2018.2348
2. American Medical Association. (2023). Augmented intelligence in medicine policy H-480.939. AMA Policy Repository. https://www.ama-assn.org/
3. Bipartisan Policy Center. (2025, December 9). FDA oversight: Understanding the regulation of health AI tools. https://bipartisanpolicy.org/issue-brief/fda-oversight-understanding-the-regulation-of-health-ai-tools/
4. Daneshjou, R., Vodrahalli, K., Novoa, R. A., et al. (2022). Disparities in dermatology AI performance on a diverse, curated clinical image set. Science Advances, 8(31), eabq6147. https://doi.org/10.1126/sciadv.abq6147
5. DJ Holt Law. (2025, July 4). Can AI diagnose patients? What healthcare providers need to know. https://djholtlaw.com/ai-legal-diagnosis-healthcare-liability/
6. Esteva, A., Kuprel, B., Novoa, R. A., et al. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639), 115–118. https://doi.org/10.1038/nature21056
7. European Parliament and Council. (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (EU AI Act). Official Journal of the European Union.
8. Finlayson, S. G., Subbaswamy, A., Singh, K., et al. (2021). The clinician and dataset shift in artificial intelligence. New England Journal of Medicine, 385(3), 283–286. https://doi.org/10.1056/NEJMc2104626
9. Fortune. (2025, July 20). UK health service AI tool generated a set of false diagnoses for a patient screening. https://fortune.com/2025/07/20/uk-health-service-ai-tool-false-diagnoses-patient-screening-nhs-anima-health-annie/
10. Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121–127. https://doi.org/10.1136/amiajnl-2011-000089
11. Grand View Research. (2024). Artificial intelligence in healthcare market size, share and trends analysis report, 2024–2030. https://www.grandviewresearch.com/industry-analysis/artificial-intelligence-ai-healthcare-market
12. Gulshan, V., Peng, L., Coram, M., et al. (2016). Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA, 316(22), 2402–2410. https://doi.org/10.1001/jama.2016.17216
13. IDR Medical. (2025, October 23). The risks of using AI for patient diagnosis. https://info.idrmedical.com/blog/why-ai-should-not-be-used-for-diagnosing-patients
14. Keragon. (n.d.). 5 major disadvantages of AI in healthcare. https://www.keragon.com/blog/disadvantages-of-ai-in-healthcare
15. Mosier, K. L., & Skitka, L. J. (1996). Human decision makers and automated decision aids: Made for each other? In R. Parasuraman & M. Mouloua (Eds.), Automation and human performance (pp. 201–220). Lawrence Erlbaum Associates.
16. National Center for Biotechnology Information. (2025, August 3). AI in clinical diagnostics: Is overreliance eroding clinical expertise? PMC12321131. https://pmc.ncbi.nlm.nih.gov/articles/PMC12321131/
17. National Center for Biotechnology Information. (2025, October 30). Reducing misdiagnosis in AI-driven medical diagnostics. PMC12615213. https://pmc.ncbi.nlm.nih.gov/articles/PMC12615213/
18. National Institute for Health and Care Excellence. (2022). Evidence standards framework for digital health technologies (3rd ed.). NICE. https://www.nice.org.uk/
19. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342
20. Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V., & Daneshjou, R. (2023). Large language models propagate race-based medicine. NPJ Digital Medicine, 6(1), 195. https://doi.org/10.1038/s41746-023-00939-z
21. Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), 206–215. https://doi.org/10.1038/s42256-019-0048-x
22. Salvi, Schostok & Pritchard P.C. (2025, October 29). AI misdiagnosis: Risks in healthcare. https://www.salvilaw.com/blog/ai-misdiagnosis/
23. Tolsgaard, M. G., Pusic, M. V., Sebok-Syer, S. S., et al. (2023). The contribution of artificial intelligence to reducing the educational and clinical workforce burden. Medical Education, 57(1), 43–52. https://doi.org/10.1111/medu.14921
24. U.S. Food and Drug Administration. (2021). Artificial intelligence/machine learning (AI/ML)-based software as a medical device (SaMD) action plan. FDA. https://www.fda.gov/
25. WellnessPulse. (2025, May 20). AI is highly unreliable in interpreting blood work. https://wellnesspulse.com/research/artificial-intelligence-interpret-blood-test-results/
26. World Health Organization. (2021). Ethics and governance of artificial intelligence for health: WHO guidance. WHO Press. https://www.who.int/publications/i/item/9789240029200