Feasibility and Validation of an Artificial Intelligence Based Software to Mimic a Clinically Relevant Approach for Anatomical Assessment of 2D Fetal Neurosonogram Video Loops - A Retrospective Multi-Reader Study

Objective
To clinically validate Origin Health Examination Assistant (OHEA), an artificial intelligence (AI) based system that mimics a clinically relevant approach to systematically and comprehensively assess fetal anatomy from 2D ultrasound video loops.

Methods
The Origin Health Examination Assistant (OHEA) comprises more than 10 different artificial intelligence (AI) based algorithms working together to perform a thorough and comprehensive anatomical assessment of the mid-trimester (18-24 weeks) fetal brain (axial and sagittal views) systematically. In each 2D video loop, the examination quality is first assessed prior to detecting standard diagnostic views. From the standard diagnostic views, relevant fetal anatomical landmarks are detected and assessed for their visibility. The AI algorithms were trained and clinically validated on a large and expert annotated dataset of 39,420 (2,249 patients) and 11,220 (311 patients) 2D ultrasound images of the fetal brain axial and sagittal views, respectively, obtained from a single tertiary fetal care centre. We used an empirical approach to measure the confidence of each AI algorithm at every stage to ensure reliability and quality control to identify incomplete examinations.

The OHEA assessed an entire 2D video loop (average: 170 frames) in under 25 seconds (Nvidia T4 16GB GPU). On an unseen external test set of 93 axial and 27 sagittal mid-trimester examinations (2D video loops) acquired from a single tertiary fetal care centre between July 2019 and February 2022, a reader panel of 7 clinicians (6 OBGYNs and 1 Radiologist) trained in fetal medicine benchmarked the performance of OHEA. Every clinician in the reader panel was allowed to accept or modify the findings provided by the OHEA. The sequence followed consisted of assessing examination quality (minimum magnification of 50%), detecting 3 standard diagnostic views (transcerebellar [TC], transventricular [TV], mid-sagittal [MS]), detecting and verifying the visibility of 10 key fetal anatomical structures (cranium, cavum septum pellucidum (CSP), midline falx, cerebellar lobes, cisterna magna, nuchal fold, choroid plexus, lateral cerebral ventricle, corpus callosum, and vermis).

To demonstrate the robustness of the OHEA, the external test set included both normal (95 cases) and abnormal cases (14 cases; 1 enlarged cisterna magna, 3 increased nuchal fold thickness, 6 choroid plexus cyst, 1 agenesis of the corpus callosum, 3 partial agenesis of the corpus callosum, and 1 vermian hypoplasia). A majority consensus from the reader panel was used as the gold standard to benchmark the performance of the OHEA. We used accuracy to benchmark the performance and inter-rater reliability Cohen’s Kappa “𝞳” to quantify the reader panel's agreement.

Results
When benchmarked against the reader panel for both the normal and abnormal cases, we observed a high accuracy and an excellent agreement (𝞳 value; interpretation: 0.84-1.0 - excellent agreement) of 97.3 % (𝞳=0.91) in the assessment of examination quality, 97.5% (𝞳=0.93) in the detection of standard diagnostic views (98.9% [𝞳=0.95] for TV, 100 % [𝞳=1.0] for TC, and 93.7% [𝞳=0.84)] for MS views), and 98.2 % (𝞳=0.95) in the detection and verification of the anatomical structures (100% [𝞳=1.0] for Cranium, 97.8% [𝞳=0.94] for CSP, 100% [𝞳=1.0] for midline falx, 100% [𝞳=1.0] for cerebellar lobes, 100% [𝞳=1.0] for cisterna magna, 98.9% [𝞳=1.0] for nuchal fold, 98.9% [𝞳=0.95] for choroid plexus, 98.9% [𝞳=0.96] for lateral ventricle, 93.7% [𝞳=0.88] for corpus callosum, and 93.7% [𝞳=0.87] for vermis).

Conclusion
We have demonstrated the feasibility of developing and validating an AI system for the clinically relevant and systematic approach to assessing fetal anatomy. We believe such assistive technologies could be highly critical in low-resource and remote settings that lack well-trained clinicians and operators, enabling timely referrals for suspected abnormalities and high-risk pregnancies. In high volume clinical practices, assistive technologies that can mimic a clinically-relevant approach to automated assessment can help faster examination and reading times and reduce operator fatigue and burnout.