Exploring Text Counterfactual Explanations: A Multi-Metric Evaluation Approach for Counterfactual Editors

Abstract

Amidst the exponential growth and breakthroughs in machine learning (ML) and its profound impact on critical domains, the need for interpretability of the models is paramount. A bridge for this model-human gap is provided by Explainable AI (XAI), which has seen rapid progress in recent years, adding transparency to machine learning processes. In this work, we focus on counterfactual explanations, a method that provides insights into the decision-making process of machine learning models by exploring alternative scenarios and hypothetical transformations. Specifically, we are concerned with the generation of text counterfactual explanations and the evaluation of counterfactual editors, which leverage natural language processing (NLP) models and tasks to generate perturbations of text sentences. Our approach involves experimenting with multiple counterfactual editors from the recent literature, models, and generation methods in order to understand their inner mechanisms and make their decisions comprehensive. In order to achieve this, we present a counterfactual editing system where we generate counterfactual, contrastive edits combining counterfactual editors with a predictor and then selecting the most minimal edit that flips the predictor’s original prediction. Moreover, we utilize methods of counterfactual generation used in current academic publications and introduce a novel method of generating counterfactual edits using part-of-speech tags to constrain the generation. We also explore multiple evaluation techniques and metrics that allow us to extract valuable conclusions that cover numerous aspects of counterfactual generation. In summary, our experiments have yielded valuable conclusions and insights. We manage to unveil hidden characteristics and patterns of counterfactual editors, explain their results, and explore various aspects of counterfactual generation. Our experiments showcase performance enhancements in counterfactual generation methods through a systematic exploration of their structural components and methodologies. Therefore, the contributions of this thesis including the utilization and introduction of novel methods in the field of counterfactual generation and a comprehensive analysis on the evaluation of counterfactual editors prove to be a promising avenue for future research.