Introduction

Clinicians in a number of disciplines work in an overburdened healthcare system that leads to difficult working environments and an epidemic of physician burnout [1]. AI-related technologies have the potential for improving efficiency on repetitive tasks, therefore increasing both patient throughput and decreasing physician burnout. For example, physicians often spend as much time doing paperwork as with patients [2]. Automatic Speech Recognition (ASR) could be leveraged to reduce the amount of the paperwork. However, the adoption of speech technology in the medical community has been slow [3].
The most common metric for measuring ASR performance, Word Error Rate (WER), has significant practical drawbacks. First, all mistakes are treated equally. In clinical settings, however, medical words are more important (e.g. "had complete resection" -> "had complete c-section" is a worse mistake than -> "has complete resection", but both have equal WER). Second, some mistakes affect the overall intelligibility more than others (e.g. "was no perforation" -> "was no puffer age" vs "was not any perforation"). Although researchers have proposed alternatives to the WER, no metric combines medical domain knowledge with recent AI advances in language understanding.
In this work, we present a new metric for evaluating the usability of medical transcription and show that it outperforms existing metrics on clinically relevant sentences.

Aims & Methods

Our proposed metric, the Clinical BERTScore (CBERTScore), combines the BERTScore [5] and the medical subset of the Knowledge Graph in a novel way. We inject medical information into this metric in two ways. First, we compute a weighted score on a subset of words involving medical terms, as determined by the Knowledge Graph [4]. Second, we tune the weight of the clinical term penalty to best match a clinician transcript dataset (CTP) that we collected.
The CTP is a collection of 18 clinician preferences on 150 medical phrases. The phrases were derived from what gastroenterologists said when dictating their procedures (ex. "We see a sessile 3mm hyperplastic polyp."). Clinicians were shown a ground truth phrase and two "transcripts," both with different mistakes. In this initial study, two clinicians were each asked which transcript was more useful, or whether they were equally useful. We measure the performance of CBERTScore and other metrics by how well they agree with clinician preferences.

Results

CBERTScore agreed with clinician preferences the most (75.4% agreement), as compared to BERTScore (66.7%), METEOR (53.4%), BLEU (24.6), and WER (10.5%). Notably, WER is the worst at determining which transcript clinicians think will be more useful. In addition, CBERTScore was not worse on non-medical text, indicating that it is useful for evaluating transcripts in a general setting. We make the preference dataset publicly available.

Conclusion

CBERTScore, a novel metric that combines medical domain knowledge and recent advances in AI, may more closely indicate clinically relevant ASR accuracy than other existing metrics. This metric warrants further validation towards becoming the primary metric to train and evaluate speech features in clinical settings.

References

[1] V. J. Dzau, et al., “To care is human—collectively confronting the clinician-burnout crisis,” N Engl J Med, vol. 378, no. 4, pp. 312–314, 2018.
[2] M. Tai-Seale, et al., “Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine,” Health affairs, vol. 36, no. 4, pp. 655–662, 2017.
[3] S. Latif, et al., “Speech technology for healthcare: Opportunities, challenges, and state of the art,” IEEE Reviews in Biomedical Engineering, vol. 14, pp. 342–356, 2021.
[4] I. Shafran, et al., “The medical scribe: Corpus development and model performance analyses,” in Proceedings of the Twelfth Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association, May 2020, pp. 2036–2044.
[5] T. Zhang, et al., “BERTScore: Evaluating Text Generation with BERT,” arXiv eprints, p. arXiv:1904.09675, Apr. 2019.
[6] H. Soltau, M. et al., “Understanding medical conversations: Rich transcription, confidence scores & information extraction,” in Interspeech, 2021.
[7] D. Goldhahn, T. Eckart, and U. Quasthoff, “Building large monolingual dictionaries at the Leipzig corpora collection: From 100 to 200 languages,” in Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12).

Disclosure

Authors work at Verily Life Sciences Israel, Verily Life Sciences USA, MIT, and Google.

Introduction

Aims & Methods

We developed 120 clinical cases based on real patient data, distributed equally across four topics:

1. Emergency esophagogastroduodenoscopy (EGD)
2. Emergency endoscopic retrograde cholangiopancreatography (ERCP) for cholangitis
3. Polypectomy
4. Percutaneous endoscopic gastrostomy (PEG) placement

Each case was assessed by three artificial intelligence models: Google GEMINI®, ChatGPT® (GPT-4o mini version), and Microsoft Copilot. These models, acting as virtual gastroenterology experts, were evaluated based on their recommendations for each scenario:

• Did they suggest urgent or delayed EGD?
• For cholangitis cases, did they recommend ERCP within 12 hours, within 48–72 hours, or in a delayed ("cold") setting?
• What management strategy did they propose for polyps based on their endoscopic description?
• Did they indicate the need for or against PEG placement?

The AI-generated recommendations were recorded in an Excel file and compared with expert endoscopist evaluations (based on ESGE guidelines) for each topic. Statistical analysis was performed using Jamovi® software (version 2.3). The level of agreement between the AI models and the human evaluator was measured using Cohen’s kappa coefficient (κ).

Results

Conclusion

Introduction

The adherence of doctors and patients to published colorectal post-polypectomy surveillance guidelines varies greatly, and patient follow-up is critical but time-consuming.

Aims & Methods

Aims: We aimed to develop an automatic surveillance (AS) system to accurately identify post-polypectomy patients, assign surveillance intervals for different risks of patients and proactively follow up with patients on time.
Methods: The endoscopic and pathological reports from 47,544 patients from three hospitals were used to develop AS system based on natural language processing. The performance of AS system was fully evaluated in internal and external tests according to five different guidelines worldwide and compared with that of doctors. Furthermore, a multi-reader-multi-case (MRMC) trial was conducted to evaluate the effect of AS system on improving doctors’ guideline adherence and efficiency, and a prospective trial was performed to evaluate the success rate for automatically informing patients and its effect on reducing the burden of nurses.

Results

In internal, external 1, and external 2 tests, the system reached an overall accuracy of 99.73%, 99.54%, and 99.77% for identifying different types of patients, respectively; and achieved an overall accuracy of ≥99.45%, ≥98.89%, and ≥98.56% for stratifying patients' risk levels and assigning surveillance intervals according to five guidelines. The system greatly increased the accuracy of doctors (98.67% vs. 78.10%, p=0.043) in MRMC trial. In a prospective trial, AS system successfully informed 93·18% of patients, and significantly reduced the follow-up time cost (127.4s vs. 14.02s per patient).

Conclusion

This system has great potential to improve doctors' adherence to guidelines while relieving the workload of doctors and nurses.

Disclosure

The authors declare no conflict of interest.

Introduction

The development of single-use endoscopes was driven by hygiene aspects. However, an even more interesting aspect is that these endoscopes are ready to use and can be immediately disposed after the examination. Therefore, these aspect favors its use outside regular working hours.

Aims & Methods

This study aimed to provide the first experiences of the Ambu single-use gastroscopes with different working channels in daily routine in an emergency setting outside regular working hours in a tertiary center.I
n a time period between Oktober 2023 and Oktober 2024 single use endoscope with 2.8 or 4.2 mm working channel were used in 63 patients. Due to quality assessment different aspects of the single-use endoscope were reported using Lickert scale. All examinations were performed outside regular working hours. 83% of the examinations were done in emergency room or intensive care unit.

Results

Successful completion of endoscopy using only the single-use gastroscope was achieved in 50 out of 63 cases (79%, suspicion of bleeding (n=44), extraction of a foreign body (n=7), bolus obstruction (n=4), pain or suspicion of perforation (n=4), reflux or gastroparesis (n=2), dysphagia (n=1), intoxication (n=1), and megacolon (n=1).Overall, 43 out of 63 patients (68%) required an intervention during the endoscopy In 13 out of 63 cases the endoscopy failed. In 8 out of 13 cases the examination could be finished successfully after a crossover to a reusable endoscope. A re-intervention was indicated in 7 out of these 13 patients. In a subgroup analyses the success rate was comparable in patients who were treated with single use gastroscope with a 2.8 mm (75%) and 4.2 mm (84%) working channel (p=0.535). The gastroscope with the 4.2 mm channel was significantly better regrading image quality (p=0.006, wheel functionality (p=0.025) and suction function (p=0.032).

Conclusion

This study demonstrates that single-use endoscopes can be used in different emergency settings, however without further technical improvement its usage especially in acute bleeding situations is limited.

Introduction

High-quality endoscopy in IBD is associated with better clinical outcomes, while quality measurement supports effective quality control. The European Society of Gastrointestinal Endoscopy recommends using nine performance measures, depending on the indication for endoscopic examination in IBD.

Aims & Methods

Our study aimed to compare colonoscopy quality between tertiary and secondary referral centers for IBD patients, based on endoscopic reports. In this cross-sectional study, colonoscopy reports of IBD patients were analyzed. At our tertiary center, we consecutively enrolled patients over 18 years old with established IBD between July and October 2024, using their available colonoscopy reports. The secondary care group included patients referred to our tertiary center, with the last existing report from secondary care used for analysis, while the tertiary care group consisted of our patients whose most recent report was similarly used. Nine key performance measures (indication, bowel preparation, photo documentation, ileal intubation, biopsy, endoscopic activity score, high-definition and chromoendoscopy use, and neoplasia detection) were evaluated based on colonoscopy reports. Patients were stratified by procedure indication (diagnostic, activity assessment, or surveillance) as well. Descriptive statistics, Welch’s, and Fisher’s exact tests were used to compare groups, with p-values <0.05 considered significant.

Results

Overall, 207 colonoscopy reports were assessed, of which 68/207 were performed at secondary care level. A total of 78/207 patients were diagnosed with Crohn’s disease, while unclassified diagnoses were noted only at secondary centers. Activity control (88% vs. 38%, p<0.001) and surveillance (8% vs. 0%, p=0.02) examinations were more common at the tertiary level. Bowel preparation was better at tertiary centers (Boston Bowel Preparation Scale [BBPS] 8.3±2.3 vs. 7.2±1.2; p=0.003), and use of BBPS was more frequent at higher levels of care (p<0.001). Disease activity scores were more commonly used at the tertiary level during activity control examinations (51.2% vs. 19.2%, p=0.004), while ileal intubation was also more frequent at the tertiary level (53% vs. 28%; p<0.001). Procedure and withdrawal times did not differ between levels in diagnostic or activity control examinations. Neoplasia detection was low, and biopsy rates were similar.

Conclusion

Quality indicator targets were met in neither secondary nor tertiary centers; however, wider investigation is needed to verify these results. Broad education can help to improve the quality of colonoscopy in IBD care.