Please use this identifier to cite or link to this item: http://artemis.cslab.ece.ntua.gr:8080/jspui/handle/123456789/18056
Title: Cognitive methods for image captioning
Authors: Σωτηρίου, Δημήτριος
Ποταμιάνος Αλέξανδρος
Keywords: μηχανική μάθηση
machine learning
βαθιά μάθηση
deep learning
νευρωνικά δίκτυα
neural networks
μετασχηματιστές
transformers
γνωσιακή νευροεπιστήμη
cognitive neuroscience
λειτουργική μαγνητική τομογραφία
functional MRI
δημιουργία λεζάντας εικόνας
image captioning
Issue Date: 26-Jul-2021
Abstract: Even though image captioning is a difficult task for computers, humans can easily describe images through inherent capabilities of their brains with little effort. Recent research has shown that brain activations encode semantic information about what people see and think. In the domain of neuroscience, several studies have attempted to extract this information from brain activations. In this work, we propose several techniques of incorporating fMRI brain activations to an image captioning model that is based on the transformer encoder-decoder architecture. Specifically, we consider fusion at the encoder, attention conditioning on the decoder and other techniques with a separate transformer encoder for the brain activations. In addition, more adaptive variants of the aforementioned fusion techniques are explored in order to enforce the usage of the weak modality of brain activations or to enable the usage of the brain activations only when they are likely to contribute significant information to the model. Due to the fact that fMRI data are limited, a “lexical expansion” step is performed in various different ways, where brain activations are predicted for novel visual stimuli, that were not used in the fMRI experiment. Our results indicate that the quality of the “lexical expansion” is not guaranteed by the main evaluation process proposed in the literature, as other evaluation procedures indicate that this mapping is not very robust, potentially introducing additional noise to the predicted activations. Therefore, the scope for improvement of the model via brain activations seems to be quite limited and only minor deviations from the baseline are observed in all our experiments, suggesting that the model fails to extract meaningful information from the weak modality of brain activations. Finally, we conclude that additional research is needed in order to establish the usefulness of brain activations in complex computational tasks such as image captioning.
URI: http://artemis.cslab.ece.ntua.gr:8080/jspui/handle/123456789/18056
Appears in Collections:Διπλωματικές Εργασίες - Theses

Files in This Item:
File Description SizeFormat 
Cognitive_methods_for_image_captioning.pdf1.72 MBAdobe PDFView/Open


Items in Artemis are protected by copyright, with all rights reserved, unless otherwise indicated.