Παρακαλώ χρησιμοποιήστε αυτό το αναγνωριστικό για να παραπέμψετε ή να δημιουργήσετε σύνδεσμο προς αυτό το τεκμήριο:
http://artemis.cslab.ece.ntua.gr:8080/jspui/handle/123456789/18056
Τίτλος: | Cognitive methods for image captioning |
Συγγραφείς: | Σωτηρίου, Δημήτριος Ποταμιάνος Αλέξανδρος |
Λέξεις κλειδιά: | μηχανική μάθηση machine learning βαθιά μάθηση deep learning νευρωνικά δίκτυα neural networks μετασχηματιστές transformers γνωσιακή νευροεπιστήμη cognitive neuroscience λειτουργική μαγνητική τομογραφία functional MRI δημιουργία λεζάντας εικόνας image captioning |
Ημερομηνία έκδοσης: | 26-Ιου-2021 |
Περίληψη: | Even though image captioning is a difficult task for computers, humans can easily describe images through inherent capabilities of their brains with little effort. Recent research has shown that brain activations encode semantic information about what people see and think. In the domain of neuroscience, several studies have attempted to extract this information from brain activations. In this work, we propose several techniques of incorporating fMRI brain activations to an image captioning model that is based on the transformer encoder-decoder architecture. Specifically, we consider fusion at the encoder, attention conditioning on the decoder and other techniques with a separate transformer encoder for the brain activations. In addition, more adaptive variants of the aforementioned fusion techniques are explored in order to enforce the usage of the weak modality of brain activations or to enable the usage of the brain activations only when they are likely to contribute significant information to the model. Due to the fact that fMRI data are limited, a “lexical expansion” step is performed in various different ways, where brain activations are predicted for novel visual stimuli, that were not used in the fMRI experiment. Our results indicate that the quality of the “lexical expansion” is not guaranteed by the main evaluation process proposed in the literature, as other evaluation procedures indicate that this mapping is not very robust, potentially introducing additional noise to the predicted activations. Therefore, the scope for improvement of the model via brain activations seems to be quite limited and only minor deviations from the baseline are observed in all our experiments, suggesting that the model fails to extract meaningful information from the weak modality of brain activations. Finally, we conclude that additional research is needed in order to establish the usefulness of brain activations in complex computational tasks such as image captioning. |
URI: | http://artemis.cslab.ece.ntua.gr:8080/jspui/handle/123456789/18056 |
Εμφανίζεται στις συλλογές: | Διπλωματικές Εργασίες - Theses |
Αρχεία σε αυτό το τεκμήριο:
Αρχείο | Περιγραφή | Μέγεθος | Μορφότυπος | |
---|---|---|---|---|
Cognitive_methods_for_image_captioning.pdf | 1.72 MB | Adobe PDF | Εμφάνιση/Άνοιγμα |
Όλα τα τεκμήρια του δικτυακού τόπου προστατεύονται από πνευματικά δικαιώματα.