External validation of the RSNA 2020 pulmonary embolism detection challenge winning deep learning algorithm

Abstract

Purpose: To evaluate the diagnostic performance and generalizability of the winning DL algorithm of the RSNA 2020 PE detection challenge to a local population using CTPA data from two hospitals. Materials and Methods: Consecutive CTPA images from patients referred for suspected PE were retrospectively analysed. The winning RSNA 2020 DL algorithm was retrained on the RSNA-STR Pulmonary Embolism CT (RSPECT) dataset. The algorithm was tested in hospital A on multidetector CT (MDCT) images of 238 patients and in hospital B on spectral detector CT (SDCT) and virtual monochromatic images (VMI) of 114 patients. The output of the DL algorithm was compared with a reference standard, which included a consensus reading by at least two experienced cardiothoracic radiologists for both hospitals. Areas under the receiver operating characteristic curve (AUCs) were calculated. Sensitivity and specificity were determined using the maximum Youden index. Results: According to the reference standard, PE was present in 73 patients (30.7%) in hospital A and 33 patients (29.0%) in hospital B. For the DL algorithm the AUC was 0.96 (95% CI 0.92–0.98) in hospital A, 0.89 (95% CI 0.81–0.94) for conventional reconstruction in hospital B and 0.87 (95% CI 0.80–0.93) for VMI. Conclusion: The RSNA 2020 pulmonary embolism detection on CTPA challenge winning DL algorithm, retrained on the RSPECT dataset, showed high diagnostic accuracy on MDCT images. A somewhat lower performance was observed on SDCT images, which suggest additional training on novel CT technology may improve generalizability of this DL algorithm.