Generating Realistic and Sparse Medical Image Counterfactuals using StyleGAN

Abstract

Deep Learning (DL) models have demonstrated great applicability in the field of medical imaging, significantly ameliorating diagnostic capabilities. However, their inherent black-box nature poses substantial ethical and practical challenges, raising concerns about their adoption. Counterfactual explanations, which provide human-interpretable insights by suggesting minimal yet meaningful modifications to input data to alter model predictions, have emerged as a promising approach to illuminate these opaque models.However,current methods for generating medical image counterfactuals exhibit critical limitations, including insufficient realism and diagnostic detail,the requirement of some form of supervision during training ,the necessity to retrain explainers for each classifier independently, and lack of sparsity in generated edits.

In this thesis, we address these challenges,by creating SPRUCE (SParse Realistic and Uncoupled Counterfactual Explanations), a novel framework specifically designed for generating sparse and realistic medical image counterfactuals. SPRUCE introduces a Generative Adversarial Network (GAN)-based approach which utilizes a specialized loss function, explicitly crafted to maintain the diagnostic relevance and visual fidelity of generated images while enforcing sparsity in modifications. The framework's core advantage lies in its ability to decouple the explainer's training from the classifier's training, allowing for the independent training of a generative model that can subsequently be employed across various classifiers within the same medical imaging domain.

In our extensive experimental evaluation, we employed state-of-the-art GAN architectures, particularly StyleGAN2-ADA, coupled with advanced GAN inversion methods, including encoder-based inversion and pivotal tuning, to ensure high-quality, editable latent representations. We validated SPRUCE using multiple medical imaging modalities, such as chest X-rays, OCT scans, and brain MRIs, demonstrating superior performance in terms of Fréchet Inception Distance (FID), Conditional Maximum Mean Discrepancy (CMMD), and other metrics. Furthermore,through our qualitative and quantitative analysis we find that the semantic coherence of counterfactual edits is tied to classifier robustness, positioning SPRUCE not only as an explanatory tool but also as a diagnostic mechanism to assess and improve model robustness prior to clinical deployment.