Not with a personal account. Free and paid personal ChatGPT accounts are not covered by a HIPAA business associate agreement. When a palliative care clinician, social worker or chaplain pastes patient details into one, that is an impermissible disclosure the program must investigate as a possible breach. A written AI policy, an approved tool, training and monitoring prevent it.
Palliative care teams hold some of the most personal information in healthcare. That includes goals-of-care conversations, advance directive wishes, prognosis discussions, family conflict, and psychosocial and spiritual notes. Summarizing those conversations clearly is hard, careful work. It's exactly the kind of writing a tired clinician might hand to an AI tool to tidy up.
That is already happening elsewhere. In November 2026, HIPAA Journal reported two cases of patient information entered into AI tools without authorization:
At Lake Burien Transitional Care Facility in Washington State, a staff member used a personal ChatGPT account to summarize client case notes containing names, diagnoses and medications. The state found several earlier uses after spotting the first and wrote to 12 families.
At Weill Cornell Medicine in New York, radiology research data, including diagnoses and medications, was entered into AI tools against the organization's policy.
Neither case involved a hacker. Both involved people trying to save time. This guide explains where the HIPAA line is, what to do if it has been crossed, and how palliative care programs can use AI without putting patient trust at risk.
Can palliative care teams use ChatGPT with patient information?
Only through a tool your organization has a signed business associate agreement for. Without one, patient information should never go into an AI tool, however well known the brand.
A business associate agreement (BAA) is the contract in which a vendor commits to protect health information the way HIPAA requires. The U.S. Department of Health and Human Services (HHS) Guidance on HIPAA and Cloud Computing says a cloud service that stores or processes electronic health information is a business associate. It also says using one without a BAA violates the HIPAA Rules. Online AI tools work the same way: what staff type in is processed on the vendor's servers.
OpenAI's help center lists only specific business offerings, such as ChatGPT for Healthcare and ChatGPT Enterprise with a Regulated Workspace, as available with a BAA. Personal free and paid accounts are not on that list. Apply the same test to Gemini, Copilot, Claude, AI transcription apps and note-taking tools.
A word on recording apps. Some AI tools listen to a visit and write the note for you. In palliative care, that can mean recording a family meeting about end-of-life wishes. Use these only when the vendor has signed a BAA and your organization has approved the tool. Follow your consent practices and state recording laws, and confirm the specifics with counsel.
What happens after a slip
Under the HIPAA Breach Notification Rule, an impermissible disclosure is presumed to be a breach. The exception is when a documented risk assessment shows a low probability that the information was compromised. The assessment weighs what was disclosed, who received it, whether it was actually viewed, and how far the risk was reduced. Goals-of-care and psychosocial details make that bar harder to clear.
If you can't show a low probability, notify affected patients, or, if a patient has died, their next of kin or personal representative where you have an address, within 60 days of discovery. Report to HHS, and notify the media if more than 500 residents of one state are affected. Many states add their own rules.
In the first 48 hours: stop the use without blame; find out which patients and details went in, and how many times; have the conversations deleted and record when; document every decision; and run the risk assessment with your privacy officer and legal counsel.
How can palliative care programs stop shadow AI without banning it?
Give clinicians a safe way to get documentation help, then close the unsafe ways. A ban alone moves the habit onto personal phones between home visits, where no one can see it.
Shadow AI means AI tools staff use for work without the organization's approval. A Wolters Kluwer Health survey, cited by HIPAA Journal, found 40% of hospital workers knew colleagues using unauthorized AI tools, and 20% admitted using them. Palliative care teams are often small, work across hospital, clinic and home settings, and include clinicians, social workers and chaplains who each document differently.
1. Find out what's already in use
Ask the whole interdisciplinary team, without blame, which AI tools they use: note cleanup, family letters, advance-care-planning summaries, transcription apps. Add AI tools to your annual risk analysis.
HIPAA tie-in: risk analysis, 45 CFR 164.308(a)(1)(ii)(A).
2. Offer an approved tool, with one simple rule
The rule: no patient or family information goes into any tool unless the organization has a signed BAA for it. Then offer an approved option for the writing help clinicians want, such as an AI feature inside your EHR or a business AI service with a BAA.

