Please use this identifier to cite or link to this item: http://artemis.cslab.ece.ntua.gr:8080/jspui/handle/123456789/18056
Full metadata record
DC FieldValueLanguage
dc.contributor.authorΣωτηρίου, Δημήτριος-
dc.date.accessioned2021-07-28T18:54:10Z-
dc.date.available2021-07-28T18:54:10Z-
dc.date.issued2021-07-26-
dc.identifier.urihttp://artemis.cslab.ece.ntua.gr:8080/jspui/handle/123456789/18056-
dc.description.abstractEven though image captioning is a difficult task for computers, humans can easily describe images through inherent capabilities of their brains with little effort. Recent research has shown that brain activations encode semantic information about what people see and think. In the domain of neuroscience, several studies have attempted to extract this information from brain activations. In this work, we propose several techniques of incorporating fMRI brain activations to an image captioning model that is based on the transformer encoder-decoder architecture. Specifically, we consider fusion at the encoder, attention conditioning on the decoder and other techniques with a separate transformer encoder for the brain activations. In addition, more adaptive variants of the aforementioned fusion techniques are explored in order to enforce the usage of the weak modality of brain activations or to enable the usage of the brain activations only when they are likely to contribute significant information to the model. Due to the fact that fMRI data are limited, a “lexical expansion” step is performed in various different ways, where brain activations are predicted for novel visual stimuli, that were not used in the fMRI experiment. Our results indicate that the quality of the “lexical expansion” is not guaranteed by the main evaluation process proposed in the literature, as other evaluation procedures indicate that this mapping is not very robust, potentially introducing additional noise to the predicted activations. Therefore, the scope for improvement of the model via brain activations seems to be quite limited and only minor deviations from the baseline are observed in all our experiments, suggesting that the model fails to extract meaningful information from the weak modality of brain activations. Finally, we conclude that additional research is needed in order to establish the usefulness of brain activations in complex computational tasks such as image captioning.en_US
dc.languageenen_US
dc.subjectμηχανική μάθησηen_US
dc.subjectmachine learningen_US
dc.subjectβαθιά μάθησηen_US
dc.subjectdeep learningen_US
dc.subjectνευρωνικά δίκτυαen_US
dc.subjectneural networksen_US
dc.subjectμετασχηματιστέςen_US
dc.subjecttransformersen_US
dc.subjectγνωσιακή νευροεπιστήμηen_US
dc.subjectcognitive neuroscienceen_US
dc.subjectλειτουργική μαγνητική τομογραφίαen_US
dc.subjectfunctional MRIen_US
dc.subjectδημιουργία λεζάντας εικόναςen_US
dc.subjectimage captioningen_US
dc.titleCognitive methods for image captioningen_US
dc.description.pages114en_US
dc.contributor.supervisorΠοταμιάνος Αλέξανδροςen_US
dc.departmentΤομέας Σημάτων, Ελέγχου και Ρομποτικήςen_US
Appears in Collections:Διπλωματικές Εργασίες - Theses

Files in This Item:
File Description SizeFormat 
Cognitive_methods_for_image_captioning.pdf1.72 MBAdobe PDFView/Open


Items in Artemis are protected by copyright, with all rights reserved, unless otherwise indicated.