Artificial Intelligence in Ophthalmologic Imaging: Joint Learning of Repeatable and Reliable Detectors and Descriptors for Inter-Device Optical Coherence Tomography Image Registration

Abstract

Optical Coherence Tomography (OCT) is a cornerstone in ophthalmologic imaging, offering visualization of important eye anatomy and notably the retina. Discrepancies between devices due to method of acquisition or differences in resolution and noise pose render automated registration highly challenging. This thesis explores the feasibility and effectiveness of a deep-learning based approach by utilizing the existing framework of the Repeatable and Reliable Detector and Descriptor (R2D2). Images of the same retina were captured using two distinct modalities. The first was a high end but expensive and inaccessible device that produced clear resolution images of the retinal layers and the second was a portable and affordable modality but produced images that provided less spatial information. Three different datasets were utilized to produce three models that jointly learned repeatable and reliable detectors and descriptors. The first consisted of the existing images after application of random transformations and pairing of the original with the newly derived augmented images. The second utilized roughly aligned images between two different modalities by expert annotation and considered them as equal. This resulted in the creation of paired images of different modalities. The third dataset was a combination of the first two. The models gave the output of dense descriptors for every pixel, repeatability and reliability heatmaps, both of which were used to extract keypoints for registration. A quantitative and qualitative evaluation of the keypoints derived by the training of the preexisting model on the original was performed. The three models derived by the corresponding dataset Crafted (C), Threepoint (3P) and Omni (O) demonstrated strengths and disadvantages in different aspects. 3P performed the best quantitatively while C showed the best repeatability maps in the high-quality OCT dataset and O managed to capture keypoints in the portable OCT dataset in a repeatable and reliable manner. However, each model on its own was not able to produce a satisfying registration result based on the traditional approach of Euclidean distance based mutual descriptor matching. A novel fusion model with a keypoint matching approach demonstrated the best results in multimodal image registration. This thesis provides a demonstration of the ability of unsupervised or semi-supervised keypoint based deep learning framework for inter-device OCT image registration. While current results are promising, challenges remain for the pipeline to be applicable in the clinical setting. Future work in novel matching strategies, automated masking techniques or other image preprocessing steps is required to bridge the gap between deep learning research and translational applications in ophthalmologic imaging.