Introduction

People can die from unwarranted variation in colonoscopy quality; endoscopists with low polyp detection rate have higher post-colonoscopy colorectal cancer incidence and mortality. The UK's National Endoscopy Database (NED) automatically captures real-time patient data and provides endoscopy key performance indicators (KPIs).

Aims & Methods

The primary objective of this study was to assess if a theory-informed and evidence-based behaviour change intervention, involving automated feedback of endoscopist case-mix adjusted Mean Number of Polyps (aMNP), improves detection performance.
The NED Automated Performance Reports to Improve Quality outcomes Trial (APRIQOT) is a national multicentre, prospective, cluster-randomised control trial. UK NHS endoscopy centres were randomised to intervention or control arms. Intervention-arm endoscopist were emailed personalised and tailored monthly reports automatically generated within NED. These contained their aMNP and an action plan, developed using evidence from qualitative interviews and informed by behaviour change theory, and access to a bespoke website. The primary outcome was endoscopists’ aMNP during the 9-month intervention. Secondary outcomes included traditional KPI. Analysis plans and a full trial protocol were published.(1)

Results

During November 2020-July 2021, 541 endoscopists across 36 centres (19 intervention; 17 control) performed 54,770 procedures during the intervention period, and 15,960 procedures during the 3-months post-intervention. Comparing intervention-arm to control-arm endoscopists during the intervention period: aMNP was 7% higher although not statistically significance (95% confidence interval (CI) -1% to 14%; p=0·08). Unadjusted MNP (10%, 95%CI 1-20%, p=0.04) and polyp detection rate (PDR) (10%, 95%CI 4-16%, p=0.002) were significantly higher. Intervention participants were significantly more likely than controls to be above the aMNP minimum standard (OR 1.77, 95%CI 1.02-3.10, p=0.04). Differences were not maintained in a 3-month post-intervention period. The intervention effect on aMNP was modified by centre workload (aMNP in low-workload centres: intervention 108.85 vs. control 92.75; high-workload centres: intervention 100.48 vs. control 102.5; p for interaction=0.01).
Within the intervention-arm, endoscopists accessing the website had higher aMNP than those who did not (118 vs 102 aMNP, p=0.03). Intervention arm endoscopists prescribed hyoscine butylbromide in a larger proportion compared to the control group (p<0.001). There was no difference in withdrawal times between groups.

Conclusion

Our evidence-based and theory-informed automated feedback intervention impacted behaviour and significantly improved MNP and PDR during the intervention period. aMNP was non-significantly higher, but more intervention endoscopists achieved minimum standards. Differences were not maintained post-intervention, suggesting feedback should be ongoing.
We demonstrate a highly efficient, scalable and entirely automated process for endoscopy feedback; and the positive impact it can have on performance. This methodology holds great potential to improve clinical practice and patient outcomes in a wide range of settings.
Intervention effects were greater in lower workload centres, this may reflect the increased effectiveness of feedback on lower baseline performance. Endoscopists engaging in the intervention benefited most. Low-performing endoscopists have the most to gain from feedback intervention, and future research should focus on engagement in feedback.

References

1. Catlow J, Sharp L, Kasim A, et al. The National Endoscopy Database (NED) Automated Performance Reports to Improve Quality Outcomes Trial (APRIQOT) randomized controlled trial design. Endosc Int Open 2020;08(11):E1545–52.

Introduction

Adenoma detection rate (ADR) has been criticised as a colonoscopy key performance indicator (KPI), for excluding significant serrated polyps, being dependent on histological data and fostering a “one-and-done” attitude. A case-mix adjusted Mean Number of Polyp (aMNP) was selected as preferred KPI in the NED-APRIQOT study Delphi process.(1,2) We hypothesised that aMNP would address the criticisms of ADR, providing a more tailored and comprehensive assessment of quality and across the whole colon. UK endoscopy centre quality indicators include post-colonoscopy colorectal cancer (PCCRC) rates and a quality Global Rating Scale assessed by the UK Joint Advisory Group for endoscopy (JAG) when accrediting endoscopy units.

Aims & Methods

We aimed to develop an aMNP using the National Endoscopy Database (NED) and assess its relationship with traditional KPI, post-colonoscopy colorectal cancer (PCCRC) and broader markers of endoscopy quality using the National Endoscopy Database (NED).
We extracted colonoscopy procedure data from NED for 1/1/2019-4/4/2019. Multiple negative binomial regression was undertaken to estimate the effects of patient age, sex and procedure indication on MNP, and generate an aMNP using the formulae:
Expected number of polyp for each procedure = e^{(constant +∑incidence ratio rates age, sex and indication)}
Expected MNP = (∑expected number of polyps per procedure) ÷ number of colonoscopies
Case-mix adjusted MNP per 100 procedures (aMNP) = Observed MNP ÷ Expected MNP x National MNP x 100
Associations between aMNP and polyp detection rate (PDR), proximal polypectomy rate (PPR), Trust PCCRC rate and endoscopy unit accreditation score were explored.

Results

92,892 complete colonoscopies were analysed. Patient age, sex and procedure indication were all significantly associated with MNP. At endoscopist level, aMNP strongly correlated with PDR (Spearman rho=0.834, p<0.001) and PPR (rho=0.709, p<0.001).

Table Distribution of aMNP, PDR and PPR per endoscopist.

Conclusion

We have demonstrated a method to compute a new case-mix adjusted KPI, aMNP, which is significantly associated with other detection KPI of PDR, PPR, PCCRC and JAG endoscopy unit quality standards. Subsequently the NED-APRIQOT study has demonstrated endoscopy unit aMNP also correlates with ADR (Spearman r=0.65, 95%CI 0.61 to 0.70)(Paper under review). aMNP addresses some limitations of traditional detection KPI, allowing consideration of warranted variation in detection, and therefore improving the face validity of detection data for endoscopists, potentially addressing engagement in audit and feedback processes. We propose it should be considered a candidate gold standard KPI for reporting endoscopy quality.

References

1. Catlow J, Sharp L, Kasim A, et al. The National Endoscopy Database (NED) Automated Performance Reports to Improve Quality Outcomes Trial (APRIQOT) randomized controlled trial design. Endosc Int Open 2020;08:E1545–52. doi:10.1055/a-1261-3151
2. Catlow J, Sharp L, Rutter M. ACCEPTABILITY OF KEY PERFORMANCE INDICATORS (KPI) IN THE NATIONAL ENDOSCOPY DATABASE (NED) AUTOMATED PERFORMANCE REPORTS TO IMPROVE QUALITY OUTCOMES TRIAL (APRIQOT), A DELPHI PROCESS. In: ESGE Days. © Georg Thieme Verlag KG 2020. ePP228. doi:10.1055/s-0040-1704620

Introduction

Gastrointestinal endoscopy training relies increasingly on simulation tools, especially in foundation years, to teach skills like scope handling and tip control. However, access to simulators offering accurate, real-time feedback is limited. Existing simulators also lack a performance score aligned with UK National Endoscopy Database (NED) Key Performance Indicators (KPIs) such as Caecal intubation rate (CIR), Polyp detection rate (PDR), and Colonoscopy comfort score (CCS). The MIKOTO colonoscopy training simulator aims to replicate diagnostic colonoscopy conditions and provide real-time feedback based on a simulator score derived primarily from patient comfort during the procedure.

Aims & Methods

This study aimed to validate the comparability of real-time performance scores from the MIKOTO (colonoscopy training simulator) with endoscopists’ NED KPIs. Conducted in a major UK Joint Advisory Group (JAG) accredited centre, the non-inferiority study involved 20 endoscopists of varying experience levels, categorized into Novice, Less Competent, Competent, and Expert groups based on their lifetime colonoscopy numbers. Endoscopists provided their NED KPIs before using the MIKOTO simulator. They then performed three full colonoscopies on the simulator, with caecal intubation time and MIKOTO scores collected after each attempt. Statistical analyses were performed to compare the results.

Results

The mean lifetime colonoscopies of the novices, less competent, competent and expert endoscopists were 31, 255, 593 and 2890 respectively. The median CIR for novices, less competent, competent and expert endoscopists were 31%, 92%, 97% and 99% respectively while the PDR across the groups were 7%,20% 30% and 45% respectively. In addition, the CCS for the various groups were 37.5%, 10%, 4% and 0.8%. The MIKOTO scores for novices, less competent, competent and expert endoscopists were 47, 69, 84 and 81 respectively. There is statistically significant correlations were observed between CCS, CIR, and MIKOTO scores across all groups (Table 1). No difference was found between Competent and Expert endoscopists in MIKOTO scores and CIR. Overall, higher MIKOTO scores were associated with better NED KPIs (Fig 1).

Table 1: Comparison between NED-KPIs and MIKOTO simulator scores for different competency levels.

Conclusion

The MIKOTO simulator score closely aligns with NED KPIs for colonoscopy, potentially serving as a real-time indicator of endoscopic competence, and a valuable tool for early endoscopy training. Larger, multi-institutional studies are needed to assess the integration of the MIKOTO simulator into UK endoscopy training for skill development and assessment.

Disclosure

No Conflict of interest

Introduction

Adenoma detection rate (ADR) and serrated polyp detection rate (SDR) are internationally recognised quality indicators of colonoscopy. However, determining ADR and SDR requires the matching of endoscopy and histology reports, which can be challenging in the presence of separate electronic medical record systems. The use of coding analytics may expedite this process, enabling rapid and efficient data collection and extraction from reports. This study aimed to validate an innovative coding analytic in measuring ADR and SDR.

Aims & Methods

A coding analytic algorithm was developed to semi-automatically extract patients’ names, polyp type, caecal intubation rate, and bowel preparation quality, from colonoscopy and histology reports. These reports were then manually reviewed to verify the concordance of patient names, ADR and SDR between the two methods. This process was applied in the initial phase, repeated following coding analytic change, and again with a validation cohort.

Results

The initial phase, which included 5911 colonoscopies performed by 53 endoscopists from January 2021 to June 2022, revealed that the coding analytic program extracted all patient names with 99.9% concordance and had a 98.9% accuracy in ADR and SDR. The discrepancies were mainly limited by spelling variation, different histology report dates, missing histology reports from the hospital’s pathology database, and overlapping terms such as tubular adenoma and tubulovillous adenoma, or villous changes that were not specific to polyps. Coding analytic search terms were subsequently modified to take into consideration these limitations. Following modification, using data from the same cohort, accuracy of the coding analytics improved to 100%, excluding 4 colonoscopies that had missing histology reports.
A total of 2022 colonoscopies performed by 40 endoscopists at the same hospital network, over July to December 2022, were included in a validation cohort. The updated coding analytic program accurately extracted all patient names with 100% concordance, and a 99.9% accuracy in ADR and SDR. The sensitivities of ADR and SDR were 99.9% and 99.6% respectively, with specificities of 100% for both ADR and SDR (Table 1). The discrepancies in the validation phase arose from 2 histology mismatches, that were a result of missing histology reports from the hospital’s histology database program. Excluding this software error renders the coding analytic program to be 100% accurate.

Table 1: Performance of coding analytics before and after modification of coding search terms

Conclusion

This study demonstrated and validated an automated extraction coding algorithm to have an accuracy of almost 100% in determining ADR and SDR for endoscopists in a large tertiary healthcare colonoscopy service. Wider adoption of these coding analytics may enable significant improvements in auditing of colonoscopy quality that is more time-efficient.