Why do transformers need positional encodings, and how do sinusoidal and RoPE approaches differ?

Self-attention is permutation-invariant: it treats input as a set, so without position information 'dog bites man' and 'man bites dog' look identical. Positional encodings inject order. The original transformer added fixed sinusoidal vectors (sines/cosines of different frequencies) to token embeddings, letting the model infer relative distances and extrapolate somewhat to unseen lengths. Learned absolute embeddings (BERT/GPT-2) instead train a position vector per index, but don't extrapolate past training length. RoPE (Rotary Position Embedding, used in Llama, etc.) rotates the query and key vectors by an angle proportional to position, so the dot product naturally encodes relative position. RoPE is now standard because it preserves relative-distance information, works well with long contexts, and supports interpolation/scaling tricks to extend context length.