Tokenization and Embeddings in NLP Explained

Computers cannot read words; they work with numbers. Natural language processing (NLP) therefore needs a reliable way to convert text into numbers that keep its meaning. Tokenization and embeddings do exactly that, and they sit underneath search engines, chatbots, translation and large language models.

What is tokenization?

Tokenization breaks a string of text into a sequence of tokens from a fixed vocabulary. Each token in the vocabulary has a unique ID. There are three main approaches.

Word tokenization

Split on spaces and punctuation: “The pump failed.” becomes [“The”, “pump”, “failed”, ”.”]. It is simple and readable, but the vocabulary becomes huge, and any word not seen during training (an out of vocabulary word) becomes an unknown token.

Character tokenization

Every character is a token. The vocabulary is tiny and nothing is ever unknown, but sequences become very long and individual characters carry little meaning, which makes learning harder and slower.

Subword tokenization

The standard for modern models. Common words stay whole, and rare words are split into meaningful pieces. “unbelievably” might become [“un”, “believ”, “ably”]. Main algorithms:

• Byte Pair Encoding (BPE): start from characters (or bytes) and repeatedly merge the most frequent adjacent pair into a new token until the vocabulary reaches the target size. Used by GPT models.
• WordPiece: similar, but merges pairs that most increase the likelihood of the training data. Used by BERT.
• Unigram and SentencePiece: treat text as a raw stream including spaces, which works well for languages without spaces such as Japanese and Chinese.

Tokenizers also add special tokens that mark the start and end of sequences, padding, or separators between sentences.

Download diagram (WebP)

What are embeddings?

Token IDs are only labels. ID 48291 is not “bigger” or “closer” to anything than ID 82; the numbers carry no meaning. A model needs a representation in which related words are near each other, so that what it learns about one word transfers to similar words.

The simplest numeric form of a token is a one-hot vector: a vector as long as the vocabulary, with a 1 at the token’s ID and 0 everywhere else. It is huge, sparse and carries no meaning; every pair of words is equally different.

An embedding replaces this with a short, dense vector, typically 100 to a few thousand numbers. The embedding layer is simply a matrix with one row per token; looking up a token ID returns its row. The values are learned during training so that the geometry of the vectors reflects meaning: “pump” and “compressor” end up close together, far from “banana”.

Static embeddings: Word2Vec and GloVe

Word2Vec (Mikolov and colleagues, 2013) learns vectors by predicting a word from its neighbours (CBOW) or neighbours from a word (skip-gram). The idea, from linguistics, is that words used in similar contexts have similar meanings. GloVe learns vectors from global word co-occurrence counts. These embeddings show famous patterns, such as the vector “king” − “man” + “woman” landing near “queen”.

Their limitation is one vector per word, whatever the context.

Contextual embeddings

Transformer models such as BERT and GPT start from a token embedding, add position information, then pass the sequence through attention layers. The output vector for each token depends on all the surrounding tokens, so “bank” in “river bank” and “bank account” gets different vectors. These contextual embeddings are why modern NLP understands meaning so much better than earlier methods.

Measuring similarity

The closeness of two embedding vectors a and b is usually measured by cosine similarity:

$$\cos\theta = \frac{\mathbf{a}\cdot\mathbf{b}}{\lVert\mathbf{a}\rVert\,\lVert\mathbf{b}\rVert}$$

A value near 1 means very similar direction (similar meaning), near 0 means unrelated, and negative means opposite.

Where tokenization and embeddings are used

• Semantic search: embed documents and queries, then return the documents with the highest cosine similarity, even when they share no exact words.
• Retrieval augmented generation (RAG): fetch relevant passages by embedding similarity and give them to a language model as context.
• Text classification: sentiment, ticket routing, fault report categorisation.
• Clustering: group similar maintenance logs or customer feedback, an unsupervised learning task.
• Language models: every neural network for language starts with an embedding layer.

Practical tips

• Always use the same tokenizer the model was trained with; a mismatch produces garbage.
• Count tokens, not words, when working with model context limits and API costs.
• Domain jargon, part numbers and tag names often split into many tokens; check how your text tokenizes.
• Watch for multilingual text: languages other than English often need more tokens per word, which affects cost and context length.
• Normalise text consistently (case, Unicode, whitespace) before tokenizing for classical models.
• For search, choose an embedding model trained for retrieval, and store vectors in a vector database or index.

Tokenization and embeddings are where language becomes mathematics. Understanding them makes the behaviour of modern language models, including their limits, much easier to reason about.