Applying the “positive predictive value–recall diagram” to monitor performance and provide recommendations for screening radiologists

Abstract

Objectives: To evaluate the suitability of “positive predictive value–recall” (PPV-recall) diagrams for monitoring performance and providing recommendations for groups of radiologists (RUs or reading units) in breast cancer screening. Materials and methods: This retrospective study used datasets from triennial quality assurance audits within the Dutch screening programme. The recall rate (RR), cancer detection rate (CDR), and PPV between 2010 and 2019 were plotted in PPV-recall diagrams separately for initial and subsequent screening. Using PPV-recall diagrams per year we compared variations in performance of the RUs within the screening programme. Each group’s screening behaviour characteristics were evaluated over time with RU-specific PPV-recall diagrams and related audit recommendations. Results: The dataset comprised the aggregated results of 779,887 initial and 6,021,598 subsequent screenings read by 12 RUs between 2010 and 2019. The PPV-recall diagrams showed substantial variations in the individual RU performance over time, with PPVs ranging between 4.9 and 23.7% for initial and 21.2–54.3% for subsequent screening. Target values were less often met for initial (2010: 0 RUs; 2019: 5 RUs) than for subsequent screening (2010: 8 RUs; 2019: 10 RUs), resulting in more recommendations regarding initial screening (24 versus 13). All recommendations focused on adjusting RR, which often (17 out of 24) changed in the recommended direction, though not always sufficient to meet target values. Conclusion: PPV-recall diagrams offer valuable insights into variations and interrelationships between screening outcomes, helping the audit team in providing recommendations for improvement. However, feedback based on these diagrams alone may not always be sufficient for individual radiologists to achieve these improvements. Key Points: Question Can positive predictive value (PPV)–recall diagrams help audit teams provide recommendations to radiologists to enhance their reading performance in a breast cancer screening programme? Findings PPV-recall diagrams help audit teams identify screening outcome variation. Recall rates often changed in the desired direction after recommendations, but did not always meet targets. Clinical relevance Incorporating PPV-recall diagrams into quality assurance audits in breast cancer screening can support audit teams to provide recommendations to radiologists to maximise cancer detection and minimise false positives. Radiologists may need additional individual feedback to optimally achieve these improvements.