Tokens & Embeddings

Models do not read characters or words directly. Text is first broken into tokens — chunks that are often a few characters long. The word "explainer" might become two or three tokens. This is why model limits are measured in tokens rather than words.

Each token is mapped to an embedding: a long list of numbers that places the token in a high-dimensional space. Tokens with related meanings end up near each other, which is what lets the model reason about similarity and analogy.

Understanding tokens is practical, not just academic. Pricing, context windows, and truncation all happen at the token level, so a rough sense of how text tokenizes helps you budget your prompts.