Introduction
Clinicians in a number of disciplines work in an overburdened healthcare system that leads to difficult working environments and an epidemic of physician burnout [1]. AI-related technologies have the potential for improving efficiency on repetitive tasks, therefore increasing both patient throughput and decreasing physician burnout. For example, physicians often spend as much time doing paperwork as with patients [2]. Automatic Speech Recognition (ASR) could be leveraged to reduce the amount of the paperwork. However, the adoption of speech technology in the medical community has been slow [3].
The most common metric for measuring ASR performance, Word Error Rate (WER), has significant practical drawbacks. First, all mistakes are treated equally. In clinical settings, however, medical words are more important (e.g. "had complete resection" -> "had complete c-section" is a worse mistake than -> "has complete resection", but both have equal WER). Second, some mistakes affect the overall intelligibility more than others (e.g. "was no perforation" -> "was no puffer age" vs "was not any perforation"). Although researchers have proposed alternatives to the WER, no metric combines medical domain knowledge with recent AI advances in language understanding.
In this work, we present a new metric for evaluating the usability of medical transcription and show that it outperforms existing metrics on clinically relevant sentences.
Aims & Methods
Our proposed metric, the Clinical BERTScore (CBERTScore), combines the BERTScore [5] and the medical subset of the Knowledge Graph in a novel way. We inject medical information into this metric in two ways. First, we compute a weighted score on a subset of words involving medical terms, as determined by the Knowledge Graph [4]. Second, we tune the weight of the clinical term penalty to best match a clinician transcript dataset (CTP) that we collected.
The CTP is a collection of 18 clinician preferences on 150 medical phrases. The phrases were derived from what gastroenterologists said when dictating their procedures (ex. "We see a sessile 3mm hyperplastic polyp."). Clinicians were shown a ground truth phrase and two "transcripts," both with different mistakes. In this initial study, two clinicians were each asked which transcript was more useful, or whether they were equally useful. We measure the performance of CBERTScore and other metrics by how well they agree with clinician preferences.
Results
CBERTScore agreed with clinician preferences the most (75.4% agreement), as compared to BERTScore (66.7%), METEOR (53.4%), BLEU (24.6), and WER (10.5%). Notably, WER is the worst at determining which transcript clinicians think will be more useful. In addition, CBERTScore was not worse on non-medical text, indicating that it is useful for evaluating transcripts in a general setting. We make the preference dataset publicly available.
Conclusion
CBERTScore, a novel metric that combines medical domain knowledge and recent advances in AI, may more closely indicate clinically relevant ASR accuracy than other existing metrics. This metric warrants further validation towards becoming the primary metric to train and evaluate speech features in clinical settings.
References
[1] V. J. Dzau, et al., “To care is human—collectively confronting the clinician-burnout crisis,” N Engl J Med, vol. 378, no. 4, pp. 312–314, 2018.
[2] M. Tai-Seale, et al., “Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine,” Health affairs, vol. 36, no. 4, pp. 655–662, 2017.
[3] S. Latif, et al., “Speech technology for healthcare: Opportunities, challenges, and state of the art,” IEEE Reviews in Biomedical Engineering, vol. 14, pp. 342–356, 2021.
[4] I. Shafran, et al., “The medical scribe: Corpus development and model performance analyses,” in Proceedings of the Twelfth Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association, May 2020, pp. 2036–2044.
[5] T. Zhang, et al., “BERTScore: Evaluating Text Generation with BERT,” arXiv eprints, p. arXiv:1904.09675, Apr. 2019.
[6] H. Soltau, M. et al., “Understanding medical conversations: Rich transcription, confidence scores & information extraction,” in Interspeech, 2021.
[7] D. Goldhahn, T. Eckart, and U. Quasthoff, “Building large monolingual dictionaries at the Leipzig corpora collection: From 100 to 200 languages,” in Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12).
Disclosure
Authors work at Verily Life Sciences Israel, Verily Life Sciences USA, MIT, and Google.