BERT

BERT (Bidirectional Encoder Representations from Transformers) is a language model introduced by Google researchers in October 2018 (code and models followed in November) that learns the meaning of words from the context on both sides of them. It set new records on language-understanding tests and became one of the most widely used models in AI research and industry.

How it works

Unlike GPT-style models, which predict text from left to right, BERT is an encoder (see Encoder-Decoder): during pre-training it learns to fill in randomly hidden words – “masked language modeling” – while seeing the surrounding text on both sides (see Self-Supervised Learning). It is then fine-tuned for tasks such as answering questions or classifying text, a prominent example of transfer learning in language, building on ELMo, ULMFiT and OpenAI’s GPT from earlier in 2018. The larger version had 340 million parameters.

Impact

BERT topped the GLUE benchmark and spawned many variants, including RoBERTa (Facebook, 2019), the smaller DistilBERT and multilingual versions. In October 2019 Google began using it for about one in ten English-language searches in the US, calling it “the biggest leap forward in the past five years” (see Semantic Search), and in 2020 the company said BERT was used for almost every English-language query. Research into what such models learn internally became a small field nicknamed “BERTology”.

After the chatbot boom

Generative decoder models took the spotlight from 2022 (see ChatGPT Moment), but BERT-style encoders remain workhorses for classification, search and embeddings because they are small and fast; ModernBERT (December 2024) updated the design.

Why it matters

BERT marked the moment when pre-trained language models became the default tool of language processing – the step before the generative era (see Natural Language Processing).

Written with Claude, made by Anthropic, a developer of generative language models that also researches what such models learn internally; BERT was developed at Google, an Anthropic investor and competitor, RoBERTa at competitor Meta (then Facebook) and GPT at competitor OpenAI.

Related

Transformer · Encoder-Decoder · Transfer Learning · Semantic Search · Word2vec · Natural Language Processing

Sources: Jacob Devlin et al.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv) (2018) · Google: Understanding searches better than ever before (October 25, 2019) · Anna Rogers, Olga Kovaleva and Anna Rumshisky: A Primer in BERTology (arXiv) (2020) · Benjamin Warner et al.: ModernBERT (arXiv) (2024)

Leave a Reply

Your email address will not be published. Required fields are marked *