ShieldForce Colored Variance Logo
AI and HIPAA in Palliative Care: Protecting Goals-of-Care Conversations From Shadow AI
palliative care

AI and HIPAA in Palliative Care: Protecting Goals-of-Care Conversations From Shadow AI

6 min read
SF
Obi Ibeto

Not with a personal account. Free and paid personal ChatGPT accounts are not covered by a HIPAA business associate agreement. When a palliative care clinician, social worker or chaplain pastes...

Not with a personal account. Free and paid personal ChatGPT accounts are not covered by a HIPAA business associate agreement. When a palliative care clinician, social worker or chaplain pastes patient details into one, that is an impermissible disclosure the program must investigate as a possible breach. A written AI policy, an approved tool, training and monitoring prevent it.

Palliative care teams hold some of the most personal information in healthcare. That includes goals-of-care conversations, advance directive wishes, prognosis discussions, family conflict, and psychosocial and spiritual notes. Summarizing those conversations clearly is hard, careful work. It's exactly the kind of writing a tired clinician might hand to an AI tool to tidy up.

That is already happening elsewhere. In November 2026, HIPAA Journal reported two cases of patient information entered into AI tools without authorization:

  • At Lake Burien Transitional Care Facility in Washington State, a staff member used a personal ChatGPT account to summarize client case notes containing names, diagnoses and medications. The state found several earlier uses after spotting the first and wrote to 12 families.

  • At Weill Cornell Medicine in New York, radiology research data, including diagnoses and medications, was entered into AI tools against the organization's policy.

Neither case involved a hacker. Both involved people trying to save time. This guide explains where the HIPAA line is, what to do if it has been crossed, and how palliative care programs can use AI without putting patient trust at risk.

Can palliative care teams use ChatGPT with patient information?

Only through a tool your organization has a signed business associate agreement for. Without one, patient information should never go into an AI tool, however well known the brand.

A business associate agreement (BAA) is the contract in which a vendor commits to protect health information the way HIPAA requires. The U.S. Department of Health and Human Services (HHS) Guidance on HIPAA and Cloud Computing says a cloud service that stores or processes electronic health information is a business associate. It also says using one without a BAA violates the HIPAA Rules. Online AI tools work the same way: what staff type in is processed on the vendor's servers.

OpenAI's help center lists only specific business offerings, such as ChatGPT for Healthcare and ChatGPT Enterprise with a Regulated Workspace, as available with a BAA. Personal free and paid accounts are not on that list. Apply the same test to Gemini, Copilot, Claude, AI transcription apps and note-taking tools.

A word on recording apps. Some AI tools listen to a visit and write the note for you. In palliative care, that can mean recording a family meeting about end-of-life wishes. Use these only when the vendor has signed a BAA and your organization has approved the tool. Follow your consent practices and state recording laws, and confirm the specifics with counsel.

What happens after a slip

Under the HIPAA Breach Notification Rule, an impermissible disclosure is presumed to be a breach. The exception is when a documented risk assessment shows a low probability that the information was compromised. The assessment weighs what was disclosed, who received it, whether it was actually viewed, and how far the risk was reduced. Goals-of-care and psychosocial details make that bar harder to clear.

If you can't show a low probability, notify affected patients, or, if a patient has died, their next of kin or personal representative where you have an address, within 60 days of discovery. Report to HHS, and notify the media if more than 500 residents of one state are affected. Many states add their own rules.

In the first 48 hours: stop the use without blame; find out which patients and details went in, and how many times; have the conversations deleted and record when; document every decision; and run the risk assessment with your privacy officer and legal counsel.

How can palliative care programs stop shadow AI without banning it?

Give clinicians a safe way to get documentation help, then close the unsafe ways. A ban alone moves the habit onto personal phones between home visits, where no one can see it.

Shadow AI means AI tools staff use for work without the organization's approval. A Wolters Kluwer Health survey, cited by HIPAA Journal, found 40% of hospital workers knew colleagues using unauthorized AI tools, and 20% admitted using them. Palliative care teams are often small, work across hospital, clinic and home settings, and include clinicians, social workers and chaplains who each document differently.

1. Find out what's already in use

Ask the whole interdisciplinary team, without blame, which AI tools they use: note cleanup, family letters, advance-care-planning summaries, transcription apps. Add AI tools to your annual risk analysis.

HIPAA tie-in: risk analysis, 45 CFR 164.308(a)(1)(ii)(A).

2. Offer an approved tool, with one simple rule

The rule: no patient or family information goes into any tool unless the organization has a signed BAA for it. Then offer an approved option for the writing help clinicians want, such as an AI feature inside your EHR or a business AI service with a BAA.

Need Audit-Ready HIPAA Compliance for Your Field Team?

Protect remote clinical staff, field laptops, mobile EVV apps, and EHR systems starting at $35/user/month.

HIPAA tie-in: business associate contracts, 164.308(b)(1) and 164.502(e).

3. Train with palliative care examples

Use scenarios your team will recognize: a social worker asking a chatbot to soften the wording of a goals-of-care summary, or a chaplain drafting a bereavement letter with family names in it. Explain that once submitted, the text is on someone else's servers. Back the training with a written sanctions policy.

HIPAA tie-in: security awareness and training, 164.308(a)(5); sanctions policy, 164.308(a)(1)(ii)(C).

4. Limit who sees the most sensitive notes, and watch for leaks

Role-based access matters more when notes cover prognosis and family dynamics. Give each person access only to what their role needs. On agency-managed devices, restrict consumer AI sites and apps, and use data loss prevention (DLP) to catch patient details before they leave the device.

HIPAA tie-in: access control, 164.312(a); audit controls, 164.312(b); information system activity review, 164.308(a)(1)(ii)(D).

5. Treat an AI slip as a security incident

Add the AI scenario to your incident response plan, including who speaks with families if notification is needed. Make reporting safe, so staff come forward early.

HIPAA tie-in: security incident procedures, 164.308(a)(6).

What should a palliative care AI policy include?

One or two pages your whole interdisciplinary team can follow. Cover these points:

Policy element

What to write down

Approved tools

The exact tools allowed with patient or family information, each with a signed BAA on file.

The bright line

No patient or family information in any other tool, including personal phones and accounts.

Recording and transcription

Which ambient or transcription tools are approved, when they may be used, and how consent is handled.

Allowed without PHI

Tasks fine with any tool when no patient details are included, such as a general family education handout.

Human review

The responsible clinician reviews and owns anything AI drafts for the record or for a family, including tone.

Requesting a new tool

Who to ask, and how security and a BAA are checked first.

Reporting mistakes

Who to tell, how fast, and an assurance that early reporting is encouraged.

Sanctions, training, review

Consistent sanctions, sign-off at onboarding and annually, and a set review date.

You can fold this into your acceptable use policy; our acceptable use policy guide shows the structure. For virtual visits, see Telehealth Security for Palliative Care.

The bottom line

Patients and families share things with palliative care teams they share with no one else. AI can help clinicians document those conversations well, but only inside tools your organization has an agreement with and can see into. Set one rule, offer an approved option, train the whole team, and make reporting safe.

ShieldForce works alongside the EHR you already run, adding monitoring, data loss prevention, training and incident response at a flat $35 per user per month. Start with the Palliative Care Compliance Audit or our Palliative Care Cybersecurity Guide.

Sources

This article is general information, not legal advice. Consult qualified HIPAA counsel for guidance specific to your program.

Share this post

Need Audit-Ready HIPAA Compliance for Your Field Team?

Protect remote clinical staff, field laptops, mobile EVV apps, and EHR systems starting at $35/user/month.

Topics

#palliative care#AI Security#Shadow AI#HIPAA compliance
Free Security Assessment

Ready to Secure Your Business?

Don't let cyber threats put your business at risk. Discover how ShieldForce protects organizations like yours - 24/7.