Models, dimensions, and neighborhoods

An embedding model is the function that maps an input to a vector. If model A turns “orange” into 768 numbers and model B also returns 768 numbers, the outputs are not thereby compatible. Each model learned its own coordinate system from its architecture, data, and objective.

Think of two maps: one reports latitude and longitude; another uses distances east and north of a local landmark. Both return pairs of numbers, but [40, -74] means different things under each system. Embedding spaces are even less directly interpretable. Store the model name and version beside every vector collection, and re-embed the collection when changing to an incompatible model.

The input format matters too. Some retrieval models are trained with different prefixes or modes for queries and passages. A provider may instruct you to encode “query: battery life” differently from “passage: this laptop lasts 14 hours.” Ignoring that contract can reduce retrieval quality even though the API returns valid vectors.

A vector with 768 values has 768 dimensions. Mathematically, each dimension is an axis, and the complete vector names one point in 768-dimensional space. More dimensions provide room for a model to arrange many relationships, but a larger number is not automatically a better representation.

Real embeddings often exhibit distributed representation. Consider color in digital images: red, green, and blue channels have explicit meanings because humans designed them that way. Learned embedding dimensions are more like internal coordinates after a complicated rotation of many useful features. One axis might contribute to distinctions involving topic, syntax, tone, and specificity at once.

Researchers can sometimes probe a space and find directions correlated with an attribute. That can help analysis, but correlation is not a universal dictionary for individual coordinates. A direction may depend on the sampled data, and rotating the entire space could preserve pairwise geometry while changing every coordinate. Search results could stay the same even though “dimension 17” now contains different values. Geometry, not a label on each axis, carries the utility.

Suppose one model outputs 384 values and another outputs 1,536. The second vector consumes roughly four times as much raw storage when both use the same numeric type. It also increases transfer size and may increase comparison cost. It might perform better on your task—or not.

Quality depends on training examples, objective, language coverage, input limits, and domain fit as well as dimensionality. A carefully trained 384-dimensional retrieval model can outperform a larger general-purpose model on a specific benchmark. Conversely, compressing a vector too aggressively can erase distinctions the application needs.

Some services allow shorter output dimensions from one model. Treat shortening as a deployment choice to test, not a free optimization. Measure whether smaller vectors preserve recall for difficult queries, not only average latency or a generic leaderboard score.

Nearby items are often similar in more than one possible sense. “Jaguar speed” could relate to an animal, a car, or a sports team. The surrounding words help the model choose a context, but short and ambiguous inputs remain difficult. A long sentence can supply intent; a one-word query may occupy a compromise neighborhood.

Granularity changes neighborhoods as well. An embedding of an entire annual report blends many subjects: revenue, risks, staffing, and sustainability. A paragraph embedding can represent one claim more sharply. Neither is always correct. Document-level vectors can support broad discovery, while chunk-level vectors often support precise question answering.

Nearness reflects what the training process rewarded. A model trained to match questions with answers may place differently worded answers near a query. A model trained mainly on general sentence similarity may prioritize paraphrase. A multilingual model may align equivalent sentences across languages. Read the model card and test behavior that matches your use case.

Many systems normalize vectors so their length equals one. For a toy vector [3, 4], the length is 5, so its normalized form is [0.6, 0.8]. Normalization preserves direction while discarding magnitude. This makes cosine similarity and dot product equivalent for those normalized vectors.

Do not normalize blindly. Follow the model provider’s recommendation and ensure indexing and query code use the same convention. If magnitude carries information for a particular model, normalization removes it. If an index expects normalized input and receives raw vectors, rankings may differ from evaluation.

Keep metadata outside the vector: source ID, title, language, date, permissions, product, and model version. Metadata enables exact filtering before or during search. An embedding might put two policy documents close together, but a filter can ensure a user retrieves only the policy for their country and access level.

To understand a model, assemble diagnostic examples. Include paraphrases, negations, numbers, abbreviations, multilingual pairs, domain terms, and intentionally confusing near-matches. Embed them and inspect nearest neighbors or retrieval rankings. This reveals operational behavior more honestly than staring at coordinate values.

For example, compare “coverage begins after 30 days,” “coverage ends after 30 days,” and “coverage begins after 60 days.” A model may group all three because they discuss the same topic. If the application must distinguish them, preserve exact text and add lexical search, reranking, or structured filters.

Visualization tools can project high-dimensional vectors into two or three dimensions, but projection distorts some distances. A colorful cluster plot is exploratory evidence, not proof that production search will work. Evaluate in the original retrieval setup with representative queries and relevance judgments.