Skip to Main content Skip to Navigation
Journal articles

What do you mean, BERT? Assessing BERT as a Distributional Semantics Model

Abstract : Contextualized word embeddings, i.e. vector representations for words in context, are naturally seen as an extension of previous non-contextual distributional semantic models. In this work, we focus on BERT, a deep neural network that produces contextualized embeddings and has set the state-of-the-art in several semantic tasks, and study the semantic coherence of its embedding space. While showing a tendency towards coherence, BERT does not fully live up to the natural expectations for a semantic vector space. In particular, we find that the position of the sentence in which a word occurs, while having no meaning correlates , leaves a noticeable trace on the word embeddings and disturbs similarity relationships.
Complete list of metadatas

Cited literature [70 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-02484933
Contributor : Timothee Mickus <>
Submitted on : Tuesday, February 25, 2020 - 3:34:06 PM
Last modification on : Wednesday, February 26, 2020 - 2:02:58 AM

Identifiers

Collections

Citation

Timothee Mickus, Mathieu Constant, Denis Paperno, Kees van Deemter. What do you mean, BERT? Assessing BERT as a Distributional Semantics Model. Proceedings of the Society for Computation in Linguistics, Gaja Jarosz, 2020, 3, ⟨10.7275/t778-ja71⟩. ⟨hal-02484933⟩

Share

Metrics

Record views

19

Files downloads

20