Performance Assessment of an Artificial Intelligence System Used for Case Detection in a Large Community-based Diabetic Retinopathy Screening Program: A Retrospective Diagnostic Accuracy Validation Study

Background. In the Philippines, Artificial Intelligence (AI) systems used for diabetic retinopathy (DR) screening are still an emerging technology. There are several AI and Deep Learning Systems (DLS) developed and licensed abroad that are currently undergoing FDA approval in the Philippines. The challenge is to validate these systems on Filipino diabetic cohorts within the uncontrolled environment of a community-based setting before it could be adapted and integrated into routine DR screening programs locally.

Objective. This study aimed to determine the performance of an AI system as a population-based mass screening tool to detect referable cases of DR.

Methods. This study was designed as a retrospective diagnostic accuracy validation study using fundus images obtained from a community-based screening program by the Ilocos Training & Regional Medical Center Eye Mobile Team in the province of La Union. The diagnostic accuracy of the iCare Illume System (Thirona RetCAD + iCare DRS plus) in detecting any stage DR (ADR) and referable DR (RDR) was validated against a reference standard of two-field fundus photographs read by expert human graders (retina specialists). The diagnostic performance was evaluated using various metrics, including Accuracy, Sensitivity (SN), Specificity (SP), Negative Predictive Value (NPV), Positive Predictive Value (PPV), Negative Likelihood Ratio (LR-), and Positive Likelihood Ratios (LR+).

Results. A total of 284 eyes (142 diabetic patients, 717 images) were included in the study. Human expert graders classified 187 eyes (65.85%) as no apparent DR while 77 eyes (27.11%) as ADR. 63 eyes (22.18%) were considered RDR. The iCare Illume AI system achieved an accuracy of 83.21 % and 92.37% for ADR and RDR, respectively. The SN, SP, NPV, PPV, LR-, and LR+ for ADR were 92.11%, 79.57%, 96.1%, 64.81%, 0.10, and 4.50, respectively. The SN, SP, NPV, PPV, LR-, and LR+ for RDR were 79.03%, 96.50%, 93.69%, 87.50%, 0.22, and 22.58, respectively. Technical failure rate was 0.70% (99.3% Imageability).

Conclusion. The iCare Illume system exhibits strong diagnostic accuracy in screening for referable DR in our cohort of Filipino Diabetics. Although its sensitivity is marginally below the UK NICE guideline, the system demonstrates high specificity, negative predictive value, and positive predictive value. This combination highlights its potential as a valuable tool for diabetic retinopathy screening in the Philippines. Future research should focus on refining local AI validation protocols, optimizing data sets of commercially available AI systems, assessing the impact on patient outcomes and disease burden, and evaluating cost-effectiveness.