Scene Graph Retrieval Using Contrastive Learning in Graph Neural Networks

Abstract

Graph Neural Networks (GNNs) have emerged as a transformative paradigm in various domains due to their remarkable ability to model complex relationships inherent in graph-structured data. The representation power of GNNs extends across diverse fields such as social network analysis, bioinformatics, recommendation systems and molecular sciences among others. Traditionally, in order to tackle the well known graph similarity problem, algorithms approximating Graph Edit Distance (GED) as well as Graph Kernels have been widely used. Recently, the advancement of deep learning techniques for graph-structured data has given rise to graph based neural approaches for the graph similarity problem. In this context, GNNs have been proven to be particularly potent, demonstrating the capability to capture intricate structural patterns and semantic relationships within graphs.

This diploma thesis delves into the representation power of Graph Neural Networks (GNNs) trained within the Contrastive Learning Framework for scene graph retrieval, a task pivotal for comprehensive scene understanding. Leveraging the capabilities of GNNs in capturing complex relationships, the study employs well-established unsupervised contrastive learning techniques to produce high quality and distance preserving graph embeddings. Additionally, a rank aware weak supervised contrastive learning loss is introduced to further enhance the retrieval metrics of these models. Ground truth for evaluation is established using approximate Graph Edit Distance (GED) algorithms, with a focus on the bipartite matching algorithm. The experimental results showcase the superior performance of the proposed contrastive learning models in approximating the GED ground truth compared to well known Graph Kernels, validating the effectiveness of Contrastive GNNs in capturing both subtle relationships and the semantic contents of scene graphs. Given their superiority in producing high-quality embeddings, GNNs can be then used to provide Counterfactual Explanations by leveraging their adeptness in graph retrieval tasks. These models enable the extraction of the most similar scene graph from another class in response to a query scene graph. This capability serves as a powerful tool for semantically explaining the differential classification of the underlying pair of images from which the scene graphs have been generated. By uncovering and highlighting the subtle structural nuances within the graphs that contribute to dissimilar classifications, GNN-based counterfactual explanations offer valuable insights into the decision-making processes of the model, promoting a deeper understanding of the semantic disparities between images and enhancing interpretability in machine learning systems.