AI Ethical Minefields in Clinical Decision-Making

Hospices are exploring the multiple ways they can use artificial intelligence (AI) by carefully considering the technology’s potentially unknown and unintended impacts.

Some providers have used AI tools to reduce errors in clinical and billing documentation. Others have used AI tools to improve operational efficiency and clinical decision making.

These quickly evolving AI technologies will play a role in hospices’ sustainability as demand rises and workforce resources dwindle. Hospices need innovative care models, which involves leveraging AI in different ways, said Dr. Kimberly Curseen, the new board of directors president at the American Academy of Hospice and Palliative Medicine (AAHPM). Curseen is also director of supportive and palliative care of outpatient services for Emory Healthcare and an associate professor at Emory University’s Palliative Medicine Program.

“It’s using technology in a smart way to lessen the human burden of a task that is ‘the work behind the work,’” Curseen told Hospice News. “That work that takes you away from the patient’s bedside and away from critical thinking. There’s a way to embrace AI that doesn’t diminish our connections with patients, but can actually elevate it if we can move some of those tasks. But it’s the rules and regulations we need to consider to accomplish this goal.”

Ensuring accurate data and safeguarding patient privacy and autonomy can lead companies to run into certain AI ethical “danger zones.” Hospices need to understand the limitations of AI, particularly when considering the use of these technologies to augment clinical decision making, according to Dr. Javier Zaglul, hospice and palliative care fellow at Florida Atlantic University.

Human oversight of AI technologies is essential to ensure ethical consideration, Zaglul indicated during the Annual Assembly of Hospice & Palliative Care, an event by AAHPM and the Hospice and Palliative Nurses Association (HPNA). He presented a co-authored study that examined nine simulated patient scenarios across four AI language models. Each model analyzed potential outcomes among mock patients with varying terminal illnesses and demographics, and provided ethical reasoning and relevance to palliative care principles.

“While each [AI system’s} individual response appears ethical, appears to have a good reason behind it, the lack of consistency or any structured framework emphasizes that there really needs to be human oversight whenever evaluating treatment decisions,” Zagul said during the assembly.

The FAU study compared information across the different AI platforms to analyze the technology’s potential to better capture a patients’ overall condition and assist in more informed, standardized ethical considerations in clinical decision making. The varying designs of each AI system resulted in “very different” ethical reasoning and patient outcomes, Zaglul said.

AI technologies are limited in the capabilities of ethical understanding, he stated. Each of the four AI platforms included in the study had different ethical frameworks for decision-making, according to Zagul. However, a common theme was that all of the AI tools optimized the likelihood of patient survival, he stated.

Some of the AI platforms in the study would not attempt to make an ethical clinical decision due to awareness of the system’s limited understanding. Gemini, for instance, cited that the “value of human life was beyond the scope of its capabilities,” Zagul said. The study’s results illustrate the potential ethically-related challenges associated with AI-assisted decision making, he stated.

“In attempts to create something standardized, you wind up creating approaches that are susceptible to bias, to variability, and regardless, still create an emotional burden,” he said during the assembly. “In our hopes of finding a consistent output from AI, something that would be able to be replicable, instead we saw divergent ideas presenting themselves. These AI chat tools are powerful, impressive, but they are not specialized triage tools trained on data for making medical decisions.”