Extending self-supervised natural language models through human brain

Abstract

In this diploma thesis, we are concerned with tasks in the domains of CognitiveScience and Natural Language Processing (NLP). We investigate natural languagerepresentations in human brain and comparing them with traditional machinelearning representations. In this work, we developed a pipeline for extractingneural representations from fMRI datasets by using machine learning techniques,following literature’s guideline. Moreover, we evaluate our work on downstreamtasks and provide useful comparative tables.In this work, we first utilize a well known fMRI dataset to map traditional wordembeddings to cognitive representations. We present a neural activation model,with ridge regression directly from glove embeddings instead of an intermediatesemantic feature model proposed in the literature, that uses a set words withavailable fMRI measurements in order to find a mapping between word seman-tics and localized neural activations. Then, we compare this encoding model,in several variations, with traditional word embeddings on a similarity task andconclude that its performance is not affected overall from the semantic or glovespace.Thereafter, we investigate how cognitive embeddings could affect a languagemodel’s representations. We incorporate cognitive embeddings into languagemodels by adding them as queries in the attention layer, in order to induce thecognitive bias of these embeddings into their training process. After finding thattheir ability to predict brain recordings improves, we test the models’ performanceat NLP tasks. Our results indicate that, even the complex BERT architecture isnegatively affected by the noisy neural representations. Our experiment setup,although it’s very promising, can not fully exploit the potential of cognitive em-beddings.