Generative AI Engineering · 20 min · 140 XP

Embeddings explained

Turn text into vectors, compare them by meaning, and know when not to trust the nearest match.

An embedding model turns a piece of text into a list of numbers, a vector, so that texts with similar meanings end up close together. "My parcel hasn't arrived" and "Where is my delivery?" share almost no words, but their vectors point in nearly the same direction. That's what lets you search by meaning rather than by keywords.

Closeness is usually measured with cosine similarity, the cosine of the angle between two vectors. 1 means the same direction, 0 means unrelated, and negative means opposite. It ignores length, so a long document and a short question can still match.

python
import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

cosine([1, 0, 0], [0.9, 0.1, 0])   # 0.994: nearly the same direction
cosine([1, 0, 0], [0, 1, 0])       # 0.0: unrelated

Real embeddings have hundreds or thousands of dimensions. The examples here use three so you can check them by hand, and the rules are the same at any size.

Three rules that save hours. First, embed queries and documents with the same model, because vectors from different models aren't comparable. Second, similar isn't the same as relevant: the closest text to a returns question might be your returns policy, or a blog post that mentions returns once. Third, there's always a nearest neighbour, even when nothing relevant exists, so set a minimum score below which you treat a match as no match.

Scenario
q = [1, 0, 0]
A = [0.9, 0.1, 0]
B = [0, 1, 0]
C = [-1, 0, 0]

Loading your workspace…