A Guide to AI Governance in Healthcare
The use of artificial intelligence (AI) in healthcare is growing. The American Medical Association reports that over 80% of doctors use AI in their work, and three-quarters said AI helps them care for their patients, which is up from 65% in 2023.
With their expanded use of AI healthcare applications for tasks like clinical decision-making, medical imaging, patient monitoring, and administrative work, healthcare organizations face questions about how these tools should be selected, evaluated, implemented, and monitored.
AI governance in healthcare refers to the set of policies that healthcare organizations enact that encompass processes and oversight frameworks for the responsible use of AI. AI governance is intended to ensure that the AI systems these organizations use are safe, effective, ethical, secure, and appropriate for their intended use throughout the AI’s life cycle, from initial consideration through implementation, monitoring, and eventual retirement.
As professionals who interact directly with patients, nurses have an important role in helping their organizations determine how AI is used. They should know what AI governance in healthcare is, and how to help build programs that maximize the benefits to patients’ health.
How Is AI Used in Healthcare?
AI comprises various technologies and approaches, including machine learning, deep learning, neural networks, predictive models, and generative AI. These technologies can process massive amounts of data, recognize patterns, and predict or produce content.
AI has wide-ranging clinical applications in areas such as diagnosis, treatment, patient monitoring, and patient engagement. It can also be used by researchers to analyze datasets that are difficult to assess manually. This feature enables them to examine disease development, discover biomarkers, predict disease risk, evaluate potential drugs, and identify patients who might respond to particular treatments.
AI governance policies provide hospital and healthcare organization leaders with a way to track where AI is being used and whether the systems are producing the intended results.
Examples of the growing role of AI in healthcare include the following:
Improve Diagnosis
Physicians and nurses understand that early diagnosis of disease can have a big impact on the long-term health of patients. For example, according to the Cancer Research Institute, early detection can save lives, with a greater than 99% survival rate for patients who are given a stage 1 breast cancer diagnosis.
AI can help healthcare providers analyze X-rays, MRIs, CT scans, ultrasounds, and other medical images to recognize patterns that could signal early signs of disease. It can also assist providers with clinical decision-making by analyzing data from patients’ electronic health records (EHRs), lab results, and other sources to help identify risks, consider potential diagnoses, or evaluate treatment options.
Track Changes in Patients’ Health
The Cleveland Clinic reports that health monitoring technology plays a significant role in the lives of Americans, and about 3 in 5 think AI will enhance healthcare.
AI systems can track patients’ health information over time and spot changes that need attention. Some systems can alert providers to changes in a patient’s condition. Others can send patients reminders or information intended to help them adhere to their medication schedule, provide follow-up care, or assist with other parts of their treatment plan.
Streamline Administrative Tasks
Healthcare organizations also use AI for nonclinical work, such as clinical documentation, scheduling, coding, claims processing, prior authorization, and medical records management. With generative AI tools, providers can more easily create clinical notes or summarize information from their visits with patients.
According to the Journal of Medical Systems, using AI for administrative tasks such as patient admissions, transfers, and discharges can help nurses save 37% to 46% of their time. This improvement can lead to improved clinical and administrative efficiency, as well as better patient care overall.
What Does AI Governance in Healthcare Entail?
AI governance in healthcare is focused on ensuring all AI systems are used and functioning as intended. It considers, for example, how poor quality or unrepresentative data can contribute to inaccurate or biased results. It also considers how the AI systems may change over time or perform differently when used with populations or in settings that differ from those in which they were developed.
Creating a healthcare organization’s AI governance policies requires developing structured processes for managing AI systems and defining who is responsible for AI decisions. Designing AI governance frameworks is a collaborative process. Individuals who can provide insights in areas such as clinical quality, patient safety, information technology, cybersecurity, compliance, and data governance often consider the following:
Tool Selection: Determining whether an AI system addresses a defined clinical, administrative, or operational need
Risk Assessment: Identifying possible effects on patient safety, privacy, security, equity, and clinical decision-making
Validation: Testing whether a system performs as expected in the population and setting where it will be used
Implementation: Establishing workflows, responsibilities, training, and safeguards before deployment
Documentation: Recording how a system was evaluated, approved, configured, and used
Monitoring: Tracking performance, errors, bias, unexpected outcomes, and changes over time
AI Issues and Challenges in Healthcare
Using AI can introduce risks in healthcare organizations. Some risks come from the technology itself, while others result from how the AI is implemented or used. There are also health equity and ethical challenges that can arise.
Examples of AI challenges and risks in healthcare include the following:
Errors and Patient Safety
AI systems can produce incorrect predictions, recommendations, or content. Errors may result from poor quality data, algorithm shortcomings, inappropriate use, or differences among the types of data used to develop a system.
In clinical settings, these errors can contribute to missed diagnoses, incorrect treatment recommendations, or other patient safety concerns. Ensuring that humans review all AI outputs and provide clearly defined procedures for responding to questionable AI outputs are important parts of AI governance in healthcare.
Bias and Equity
AI models learn from data, and if the healthcare data they learn from reflects biases in access, diagnoses, or treatments then the AI’s outputs will not adequately represent the population where the system will be used. The AI model may also perform differently across different patient groups.
To address these challenges, healthcare organizations should evaluate the data used to develop their AI systems and test the AI’s performance across their relevant patient populations.
Transparency
Some AI systems provide results without clearly showing how they reached them. This is sometimes described as the “black box” problem.
Limited transparency can make it difficult for providers to determine why an AI system produced a particular recommendation or to identify the source of an error. AI governance policies can establish requirements for documentation, explainability, and human review based on the level of risk involved to help track the source of issues should they arise.
Data Privacy and Security
AI systems can require access to large amounts of patient health information. This creates questions about how patient data is collected, stored, shared, and used.
Healthcare organizations must consider their existing privacy and security requirements as well as uses that patients may not expect, such as using healthcare data to train commercial AI models. Security breaches or inappropriate data use can expose patients’ sensitive information and reduce their trust in the healthcare organization.
Training and Human Oversight
Rapid changes in AI can make it difficult for healthcare workers to understand the capabilities and limitations of every system they use. Without adequate training, healthcare providers and staff may rely solely on AI outputs when human review of the outputs is necessary or use a tool for purposes for which it was not designed.
Healthcare organizations should define when human review is required and provide training to all users that is appropriate to each AI system and its risks.
Workflow Integration
Even AI systems that have been validated or cleared by regulators can create problems if they do not fit a healthcare organization’s existing clinical workflows. For example, new systems may change how information is collected, reviewed, or communicated.
Healthcare organizations should examine their current workflows before introducing any AI systems and determine how the new technology will affect their providers, staff, and patients.
Best Practices for AI in Healthcare
AI governance policies should cover the full life cycle of an AI system rather than treat its approval as the only concern. Organizations can prioritize using AI best practices to maximize the success of their AI initiatives, including the following:
Establish Clear Oversight
Healthcare organizations should define who has the authority to review, approve, monitor, and discontinue AI systems. There are many stakeholders in AI governance in healthcare processes, including healthcare providers, data scientists, information technology (IT) staff, cybersecurity professionals, AI specialists, legal and compliance experts, ethicists, and representatives of affected patient populations.
Different systems may require different levels of review. An administrative scheduling tool, for example, may not require the same oversight as an AI system that influences patients’ diagnoses or treatments.
Start With a Defined Need
Before selecting an AI system, healthcare organizations should identify the problem it is intended to address and examine the existing standard of care or workflow connected to that problem. This provides a basis for determining whether the AI is necessary, which of its outcomes should be measured, and whether a particular system fits the needs of the organization and its patient population.
Evaluate Data and Bias
Healthcare organizations should determine what data was used to develop and validate an AI system and whether that data represents the population in which the system will operate. The amount and quality of the data used, relevant patient characteristics, data-processing methods, and known limitations should be documented. The AI’s performance should also be evaluated across relevant patient groups when possible.
Establish Documentation Requirements
Healthcare organizations should maintain records covering the AI systems they use throughout their life cycle. Documentation can include the system’s intended use, data sources, validation results, known limitations, approval decisions, responsible parties, training requirements, and monitoring results. Maintaining these records makes it easier for organizations to investigate problems, demonstrate compliance, and determine whether the system remains appropriate over time.
Monitor AI After Deployment
Monitoring is particularly important because patient populations, clinical practices, data sources, and AI models can change. A system that performed appropriately when it was introduced may not continue to produce the same results.
Healthcare organizations should monitor AI systems for errors, changes in accuracy, biases, unexpected outcomes, and other performance issues. They should also establish procedures for reporting any problems and investigating any adverse events.
Prepare for Changing Regulations
Healthcare AI is governed by a developing combination of federal and state laws, regulatory requirements, professional standards, and organizational policies. Requirements may differ depending on whether an AI application qualifies as a medical device or performs an administrative, operational, or clinical function.
Healthcare organizations therefore need internal oversight that can adapt as regulations and standards change.
AI in Healthcare Governance Resources
AI governance in healthcare continues to evolve. As AI capabilities and use cases expand in healthcare, so will the need for professionals to stay attuned to the changes. The following resources provide additional insights into AI governance. They can be useful for individuals involved in, or planning to be involved in, building AI governance policies within their organizations.
American Medical Association, “Governance for Augmented Intelligence”: A learning tool primarily focused on the importance of augmented intelligence governance to ensure that the implementation of AI technologies in healthcare settings is safe, ethical, and responsible
Congress.gov, “Artificial Intelligence (AI) in Health Care”: A report on the challenges and questions that Congress may wish to consider regarding ongoing AI in healthcare oversight and regulatory efforts
JAMA, “AI, Health, and Health Care Today and Tomorrow: The JAMA Summit Report on Artificial Intelligence”: Recommendations on how health and healthcare AI should be developed, evaluated, regulated, disseminated, and monitored
NPJ Digital Medicine, “Advancing Healthcare AI Governance Through a Comprehensive Maturity Model Based on Systematic Review”: A framework that includes adaptive governance strategies for AI implementation that delivers tangible benefits
Safety Science, “Artificial Intelligence Adoption Challenges From Healthcare Providers’ Perspectives: A Comprehensive Review and Future Directions”: An overview, including a framework and road map, of how to overcome challenges and advance the safe and effective adoption of AI in healthcare
American Medical Association, “8 Steps to Position Your Health System for AI Success”: Strategies regarding accountability, oversight, policies, and structure to help ensure the effective use of healthcare AI
HealthCatalyst, “Safeguarding the Ethics of AI in Healthcare: Three Best Practices”: Insights on the role of data in using AI in an ethical manner to improve healthcare delivery, and best practices to guard against bias, ensure patients’ privacy, and maximize efficiencies while assisting humanity
AAMC, “Principles for the Responsible Use of Artificial Intelligence in and for Medical Education”: Guidance on the integration of AI in medicine, primarily in medical education
Staying Ahead in a Changing Field
AI governance in healthcare is not just of concern to healthcare organizations; AI policies are being developed by the federal government as well.
The MedTech Association reports that the Food and Drug Administration (FDA) has been evaluating and authorizing AI-enabled medical devices for over 25 years. More than 1,000 products have already been approved that have helped reshape patient care and led to better outcomes, diagnostic accuracy, and treatment pathways. However, ensuring that these technologies are effective, safe, responsible, and ethical remains a pressing concern for the FDA. For example, it is currently seeking public feedback to inform its regulatory approach to generative AI-enabled medical devices.
At every level of the healthcare industry, medical professionals who continue to learn and adapt can play a key role in developing regulatory approaches that keep up with the rapid innovations in digital health technologies.
