A few days ago I sent out an email on possible criteria for using AI to help write progress notes. To my way of thinking it boils down to three criteria:
- Security: Sending client data to an AI in the cloud is obviously not okay. Questionable even with a BAA agreement (or too expensive like Mentalyc – the first service I have seen for this purpose). It could work if the AI is downloaded to your computer and does not communicate with the Internet on its own.
- Creativity: AIs tend to add their own information beyond what you specify to whatever they write. This does not work in a progress note.
- Worth It: The effort of providing the AI with the information to write a progress note needs to be less than the effort of just writing it yourself.
So… I’ve been experimenting…
Obviously – I’m using fake client data. Vacuna is an AI that can be downloaded to your own laptop to run. I’m testing it online to see if that is worth doing.
Also – I’m not trying to write great progress notes right now – I’m just testing to see if a) the AI (Vacuna) is capable of not adding creative elements to what I want it to write, and b) can write a note without me spending more effort on it than just writing it myself.
ROUND #1: Trying to get Vacuna to understand not to create fake items in the progress note.**
PROMPT 1: Please write a paragraph with the following facts. Do not add additional facts or get creative. Marie is 32 years old. Has depression. Has anxiety. We practiced breathing exercises, CBT, and guided visualization.
VICUNA 1: Marie is a 32-year-old woman who has been struggling with depression and anxiety. As part of her treatment, we have been practicing various techniques, including breathing exercises, cognitive-behavioral therapy (CBT), and guided visualization. These techniques have helped Marie to better manage her symptoms and improve her overall well-being. By learning how to breathe more effectively and calm herself down when she feels anxious, Marie has been able to reduce her reliance on medication and feel more in control of her emotions. Through CBT, Marie has been able to identify negative thought patterns and challenge them, leading to a more balanced outlook on life. And through guided visualization, Marie has been able to visualize positive outcomes and increase her motivation and hope for the future. These techniques have played an important role in Marie's recovery and will continue to be useful tools for her as she continues her journey towards wellness.
PROMPT 2: Stick to only information provided in the prompt and try again please.
VICUNA 2: Marie is a 32-year-old woman who has depression and anxiety. We have been practicing breathing exercises, CBT, and guided visualization with her. These techniques have helped her manage her symptoms and improve her well-being. By practicing breathing exercises, Marie has been able to reduce her reliance on medication and feel more in control of her emotions. Through CBT, Marie has been able to identify negative thought patterns and challenge them, leading to a more balanced outlook on life. And through guided visualization, Marie has been able to visualize positive outcomes and increase her motivation and hope for the future. These techniques have played an important role in Marie's recovery.
PROMPT 3: Marie is not on medication and her symptoms remain stable. Please try again thanks.
VICUNA 3: Marie is a 32-year-old woman who has depression and anxiety. We have been practicing breathing exercises, CBT, and guided visualization with her to help manage her symptoms and improve her well-being. These techniques have helped Marie to better control her anxiety and depression symptoms without the need for medication. By practicing breathing exercises, Marie has been able to calm herself down when she feels anxious and reduce her stress levels. Through CBT, Marie has been able to identify negative thought patterns and challenge them, leading to a more balanced outlook on life. And through guided visualization, Marie has been able to visualize positive outcomes and increase her motivation and hope for the future. These techniques have played a significant role in Marie's recovery and will continue to be useful tools for her as she maintains her well-being.
PROMPT 4: Marie has not yet experienced improvement. Please try again thanks.
VICUNA 4: Marie is a 32-year-old woman who has been struggling with depression and anxiety. We have been practicing breathing exercises, CBT, and guided visualization with her in an effort to help manage her symptoms and improve her well-being. While these techniques have potential benefits for many people, unfortunately, Marie has not yet experienced any improvement. Despite our best efforts, we have not been able to help her reduce her symptoms or improve her overall well-being. We will continue to work with Marie and explore other treatment options to help her manage her symptoms and move towards recovery.
So, yeah, not working so far at all... [I do get it to work further down in the note.]
ROUND 2: Getting Vacuna to just rearrange what I write.**
While the main point of psychotherapy is for the client to get BETTER, sometimes they don't for awhile. Some clients with severe conditions need ongoing supportive therapy to simply maintain their highest level of functioning without backsliding.
When this happens, the clinician has to find new ways to write the same thing again, and again, and again...
Why? Because insurance auditors frown on cutting and pasting the same exact old text into multiple notes.
In this unfortunate situation, Vicuna actually does fairly well in this sample:
PROMPT: Please reword the following paragraph changing some words and sentence order, but using the exact same information facts and adding nothing: Pete is a 45 year old electrician reporting continuing depression. Symptoms remain unchanged from last session. Today we discussed his workplace frustrations at length. He continues to blame workplace stressors for his mood and problems with his family.
VICUNA: Pete, who is 45 years old and a electrician, reported continued symptoms of depression during his previous session. The symptoms have remained unchanged since the last time he visited. During this session, Pete and his therapist discussed in detail his workplace frustrations. Despite this, Pete continues to attribute his mood and problems with his family to workplace stressors.
Success!
ROUND #3: Retrying Round #1 (Trying to get Vacuna to understand not to create fake items in the progress note.)
Okay -- so now I may have the right wording to keep Vicuna from adding elements to the note! To make this worth it, I would need to be able to shorten/bullet-point my prompt. Maybe have the AI take data from a point and click tablet form I fill out.
Why not just use the point and click form? Some auditors require written text -- not just checkbox forms. Also -- this may just be best practice for notes to capture more of value.
Note how random the right "magic formula" is for getting an AI to understand. In Round #1 I clearly told Vicuna not to invent information not given to it (no luck). Below, I have stumbled upon the right word combination:
PROMPT 1: Please reword the following changing some words and order, but using the exact same information facts and adding nothing: Tom is 34 years old, a plumber, reports continuing depression, symptoms remain unchanged, we discussed his workplace frustrations, he blames workplace stressors for his mood and problems with his family.
VICUNA 1: Tom, a 34-year-old plumber, reported ongoing symptoms of depression during his latest session. The symptoms have remained unchanged since his last session. During the session, Tom and his therapist discussed his workplace frustrations in detail. Tom continues to attribute his mood and family problems to workplace stressors.
STARTING OVER TO SEE IF I CAN JUST LIST ELEMENTS WITHOUT WRITING MUCH:
PROMPT 2: Please reword the following changing some words and order, but using the exact same information facts and adding nothing: Tom, 34 years old, plumber, continuing depression, symptoms unchanged, discussed his workplace frustrations, he blames workplace stressors for mood and family problems.
[Note how I just listed data with commas between after the initial sentence.]
VICUNA 2: Tom, who is 34 years old and works as a plumber, has been experiencing ongoing depression with unchanged symptoms. He discussed his workplace frustrations, citing workplace stressors as the cause of his mood and family problems.
MOSTLY SUCCESSFUL!!
So now I have the AI not inventing garbage to add to the progress note. Would need to run this AI from a laptop computer making sure it sends nothing back out to Internet for medical data security HIPAA reasons. This might be useful for notes that are 100% written, and for point and click forms with a bit of text added.
It would be better if the AI could take information from a point and click form. It does seem to take 90% of the effort of just writing the note myself!
This was a quick demo, so I was not trying to write a beautiful note -- just test if the darn thing could write sentences from lists and NOT invent information not provided.
