Purpose: To assess the diagnostic performance of a commercial artificial intelligence (AI) software in detecting pulmonary embolism (PE) in emergency settings, compared with on-call radiology residents.

Methods: All consecutive emergency CT pulmonary angiographies (CTPA) performed over a 3-month period in the emergency department of a university hospital in patients with suspected PE, initially interpreted by the radiology residents during the emergency workflow and subsequently verified and approved by a board-certified radiologist, were concomitantly analyzed by an AI software for the presence of PE. The AI results were sent in a separate PACS partition and were not available for the preliminary and for the final reporting. Diagnostic performance of AI and residents for PE detection was assessed against the final report of the radiologist attending, which served as the reference standard.

Results: Among 594 CTPA examinations, PE was present in 82 patients (13.8% prevalence), including 41 (50%) proximally located emboli. Overall, AI achieved a sensitivity and a specificity of 89% and 99%, respectively, compared with 97.6% sensitivity and 99.2% specificity for residents (p > 0.05). For proximal PE, sensitivity was 97.6% for AI and 100% for residents (p = 1), whereas for peripheral PE, AI sensitivity was 80.5% versus 95.1% for residents (p = 0.08).

Conclusion: Radiology residents showed an overall better performance in detecting PE compared to the AI software. The AI software achieved good diagnostic performance for PE detection, particularly for proximal located PE.