How do embedding models represent relationships between words?

How do embedding models represent relationships between words?

Embedding models represent word relationships through patterns in numerical vectors. During training, the model adjusts those vectors to perform an objective, such as predicting surrounding words or identifying related text. The resulting geometry can capture some useful relationships without storing a separate hand-written rule for every pair.

Training creates useful regularities

Words appearing in related contexts may receive representations with shared structure. For instance, “violin” and “cello” often occur in discussions of orchestras, instruments, and music. A suitable model may therefore represent them as related, even though they are different objects.

Classic word-vector research demonstrated that learned representations can support evaluations of syntactic and semantic relationships. See the original word2vec research paper for that historical work.

Distance depends on the representation

Cosine similarity compares vector direction, while other measures use different notions of closeness. The appropriate measure depends on how the embeddings were trained and normalized. Two vectors having the same dimension is not enough to establish that comparing them is meaningful.

Relatedness also differs from synonymy. “Hot” and “cold” may occur in similar contexts, so a model can place them near each other despite their opposite meanings. Topic similarity should not be mistaken for agreement.

Context changes the question

A fixed word embedding combines evidence across many uses of a word. A contextual encoder can instead create different representations for “apple pie” and “Apple laptop.” For practical search or classification, test the exact relationships that matter rather than assuming that all kinds of linguistic meaning are preserved equally well.