How to use Gen AI to read images and process notes

Summary

In this video, the presenter explores a unique use case of generative AI in the healthcare setting. They demonstrate how large language models and image recognition tools can transform handwritten notes from busy physicians into structured clinical records within an Electronic Medical Record (EMR) system. By scanning a napkin with shorthand medical notes, the AI processes the information into a draft clinical note. Although the AI is not perfect and requires human editing, this innovative approach significantly reduces the time spent documenting patient encounters, showcasing the potential of AI in healthcare workflows.

Takeaways

• 😀 The speaker alternates between education, use case, and safety/ethics topics in their series.
• 😀 Today’s focus is on a use case involving generative AI in healthcare, specifically in a busy emergency department (ED).
• 😀 A physician shared an example of handwritten notes on a napkin, which led to the idea of converting these notes into structured clinical records using AI.
• 😀 The napkin notes contained abbreviations and shorthand, commonly used by physicians to quickly record key information during patient care.
• 😀 The script involves analyzing a recreated napkin image to understand its content and test if AI can interpret and convert it into a clinical note.
• 😀 The AI model used in this experiment was capable of turning handwritten notes from the napkin into a structured medical record in an EMR system.
• 😀 The AI system processed the napkin image and generated a text output, which was then formatted into a clinical note using another AI model.
• 😀 The result of the AI process was impressive, converting shorthand notes into full sentences, though it still required human review and editing for accuracy.
• 😀 The AI correctly identified and processed critical medical information such as patient history, medications, vital signs, and assessments.
• 😀 This experiment demonstrated the potential of using two different AI models (vision and language) in healthcare to save time and assist clinicians in documentation.
• 😀 While this technology is still experimental and subject to regulatory approval, it showcases the promise of AI in improving efficiency in healthcare documentation.

Q & A

What is the main focus of the video?

The video focuses on exploring a use case where generative AI is used to convert handwritten medical notes, often written on napkins by physicians, into structured clinical notes within an electronic medical record (EMR) system.

What challenge does the video address in healthcare settings?

The video addresses the challenge of physicians writing quick, shorthand notes during busy clinical work, which can be difficult to transcribe into a formal medical record. The challenge is to convert these unstructured, often handwritten notes into accurate, structured clinical documentation using AI.

How does the AI system process the handwritten notes?

The AI system uses a two-step process. First, a vision model scans and converts the handwritten notes (like those on a napkin) into raw text. Then, a language model refines the text into a structured medical note that can be added to the EMR.

What is the role of the vision model in this process?

The vision model's role is to extract the text from the image of the handwritten notes, identifying medical acronyms, numbers, and shorthand used by the physician, and turning them into raw, legible text.

What does the language model do with the raw text?

The language model takes the raw text extracted by the vision model and processes it to generate a structured medical note, properly formatted for use in the EMR system, ready for the physician to review and edit.

What was the outcome when the AI system processed the handwritten notes?

The AI system successfully converted the handwritten notes into a medical note, capturing key information like the patient's medical history, medications, vital signs, and assessment. However, some errors occurred, such as misinterpreting 'left upper extremity' as 'left lower extremity,' highlighting the need for human review.

What kind of errors were observed in the AI-generated note?

One error was the misinterpretation of 'LUE' (left upper extremity) as 'LLE' (left lower extremity), demonstrating that while the AI performs well, it's not flawless and requires human intervention to correct inaccuracies.

What was the significance of the 'Cardiology consult' and 'two lab tests' in the output?

These elements represent the AI's ability to correctly identify certain aspects of the physician's shorthand notes, such as the recommendation for a cardiology consult and lab tests, and translate them into formal recommendations in the structured clinical note.

Why is the AI process not fully automated, according to the video?

The AI process is not fully automated because the technology is still experimental and the accuracy of the generated notes is not perfect. A human physician is required to review and make final edits to ensure that the note is accurate and appropriate for patient care.

What future improvements are expected for the AI system discussed in the video?

Future improvements include enhancing the accuracy of the AI models through better versions, more refined prompts, and incorporating human feedback to reduce errors. The system's efficiency in converting handwritten notes into medical documentation is expected to improve as the technology evolves.