QS305 - The Use of Large Language Models to Extract Operative Standards from Narrative Operative Reports: A Proof-of-Concept Study

Purpose/Background: While ascertaining critical steps to surgical procedures is essential for quality improvement and research, the narrative format of operative reports makes automation of this highly challenging. Synoptic operative notes are promising, but their adoption has been slow, and manual extraction of narrative reports is prohibitively expensive. Large language models (LLMs) like ChatGPT can perform a range of functions including text summarization, text analysis, and data extraction. We hypothesized that a HIPAA-compliant LLM could effectively extract synoptic information from operative notes with accuracy similar to expert chart review.

Methods/Interventions: This is a single-institution retrospective study of operative reports (01/2022 – 12/2022) for low anterior resections performed for rectal cancer. A HIPPA-compliant platform “Versa” was used with two OpenAI models: gpt-35-turbo and gpt-4. Prompt engineering (iterative prompt refinement) was performed to ensure consistent LLM responses in at least 5 repeated prompt-response cycles on separate data (Figure 1). Accuracy of LLMs was compared to consensus manual review. Secondary endpoints included evaluation of LLMs for sensitivity and specificity for free-text data, comparison of LLMs and resident against attending review, and assessment of time and costs. Two independent surgeon reviewers evaluated operative reports for (1) adherence to rectal cancer surgery standards set forth by the American College of Surgeons Operative Standards for Cancer Surgery and (2) unique patient- or procedure-related factors. Four standards were assessed: vascular pedicle ligation, total mesorectal excision, lymphadenectomy, and distal margin. Consensus between the manual reviewers was obtained with regard to standards adherence for each note by discussion with adjudication by a third surgeon.

Results/Outcomes: Forty-five operative reports across 4 colorectal surgeons were evaluated (24 robotic, 16 laparoscopic, and 5 open LARs) performed for locally advanced (n = 38) or metastatic (n = 7) rectal cancer. Cohen’s kappa was calculated to determine the interrater reliability of the two manual reviewers for each operative standard, with Standard 1 being in perfect agreement, Standard 3 and 4 in substantial agreement (kappa 0.77 and 0.68, respectively), and Standard 2 with slight agreement (kappa 0.04). Compared to the consensus review, GPT-4 was slightly more accurate than GPT-3.5 (Table 1). Compared to attending review, the accuracy of GPT-4 was not significantly different from a PGY-3 surgical resident (paired t-test, p = 0.95). When asked to extract “unusual circumstances”, GPT-3.5 had a sensitivity of 48.3% and specificity of 93.8% while GPT-4 had a sensitivity of 51.7% and specificity of 81.3%. The time for attending and resident review were 122 min and 153 min, respectively. LLM processing elapsed 3 min 20 sec (GPT-3.5) and 10 min 2 seconds (GPT-4). The total processing costs were $0.21 (GPT-3.5) and $5.01 (GPT-4).

Conclusion/Discussion: Narrative operative reports are highly technical and difficult to interpret even for other physicians. LLMs can extract information and evaluate adherence to operative standards as well as a mid-level surgical resident, with increasing accuracy for measures that have strong agreement in manual review, at a rapid speed and affordable cost. LLMs have the potential to change the landscape of surgical research and quality improvement by expanding access to data that is currently cost-prohibitive to obtain.