Word2vec

Word2vec is a method, published by Tomas Mikolov and colleagues at Google in 2013, for learning embeddings – lists of numbers representing words, arranged so that words used in similar contexts end up close together. It made word vectors fast to train on billions of words and popularized them across AI.

How it works

Word2vec trains a shallow neural network on a simple prediction task: guessing a word from its neighbors (“continuous bag of words”) or the neighbors from a word (“skip-gram”). The learned weights become the word vectors. The approach reflects the “distributional” idea, summed up by the linguist J. R. Firth in 1957: “you shall know a word by the company it keeps.” The vectors showed surprising regularities: subtracting the vector for “man” from “king” and adding “woman” landed close to “queen”.

Impact and criticism

Word2vec spread quickly through research and industry and inspired successors such as GloVe (Stanford, 2014) and fastText (Facebook, 2016); in 2023 its second paper received the NeurIPS Test of Time Award. In 2016 Tolga Bolukbasi and colleagues showed that word vectors trained on news text reproduced stereotypes – pairing “man” with “computer programmer” and “woman” with “homemaker” – which helped launch research on bias in embeddings (see Algorithmic Bias). Later work showed that such analogies, including the queen and homemaker examples, partly depend on a rule that bars the input words from the answer; without it, “king” itself often comes out on top. Contextual models such as BERT replaced fixed word vectors with representations that change with context.

Why it matters

Word2vec made “meaning as geometry” practical – the idea behind the embeddings that power today’s search and language models (see Latent Space).

Written with Claude, made by Anthropic; Word2vec was developed at Google, an Anthropic investor and competitor, fastText at competitor Meta (then Facebook), and the 2016 bias study was co-authored at Microsoft Research, Microsoft being an Anthropic investor.

Related

Embedding · Latent Space · BERT · Natural Language Processing · Algorithmic Bias · Semantic Search

Sources: Tomas Mikolov et al.: Efficient Estimation of Word Representations in Vector Space (arXiv) (2013) · Tomas Mikolov et al.: Distributed Representations of Words and Phrases and their Compositionality (arXiv) (2013) · Tolga Bolukbasi et al.: Man is to Computer Programmer as Woman is to Homemaker? (arXiv) (2016) · Malvina Nissim, Rik van Noord and Rob van der Goot: Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor (Computational Linguistics) (2020)

Leave a Reply

Your email address will not be published. Required fields are marked *