The Transformer architecture, introduced in the 2017 paper “Attention Is All You Need” by Vaswani et al., fundamentally changed the landscape of artificial intelligence. It replaced recurrence with attention and enabled the parallel processing that powers today’s large language models.
Why Transformers Replaced RNNs and LSTMs
Before Transformers, sequence modeling was dominated by Recurrent Neural Networks (RNNs) and Long Short-Term Memory networks (LSTMs). These architectures processed tokens one at a time, sequentially — making them inherently slow and limited in their ability to capture long-range dependencies. Transformers solve both problems with a single innovation: the self-attention mechanism.
The Self-Attention Mechanism
Self-attention allows each token in a sequence to attend to every other token simultaneously. For each token, the model computes three vectors:
- Query (Q): What am I looking for?
- Key (K): What do I contain?
- Value (V): What information do I carry?
The attention scores are computed as: Attention(Q, K, V) = softmax(QK^T / √d_k) × V. The division by √d_k prevents the dot products from growing too large, which would push the softmax into regions of extremely small gradients.
Multi-Head Attention
Instead of computing a single attention pattern, Transformers use multi-head attention — running multiple attention operations in parallel with different learned projections. This allows the model to attend to different aspects of the input simultaneously:
- One head might focus on syntactic relationships
- Another on semantic similarity
- A third on positional relationships
Encoder-Decoder Architecture
The original Transformer had two main components:
- Encoder: Processes the input sequence. Each encoder layer has a multi-head self-attention sub-layer followed by a position-wise feed-forward network, with residual connections and layer normalization around each
- Decoder: Generates the output sequence autoregressively. It has three sub-layers: masked self-attention (preventing attention to future tokens), cross-attention to the encoder output, and a feed-forward network
Positional Encoding
Since Transformers process all tokens in parallel, they have no inherent sense of order. Positional encoding injects information about token positions using sinusoidal functions or learned embeddings. The original paper used sine and cosine functions of different frequencies:
PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
Key Variants
- BERT (Encoder-only): Bidirectional context, ideal for understanding tasks like classification and NER
- GPT (Decoder-only): Autoregressive, ideal for text generation — the architecture behind ChatGPT
- T5 (Encoder-Decoder): Text-to-text framework, casting all NLP tasks as text generation
Scalability and Impact
The Transformer’s parallelizable design enabled the scaling seen today. Training on GPUs and TPUs with massive batch sizes, models grew from BERT’s 340M parameters to GPT-4’s estimated trillion-plus. The architecture’s flexibility has extended beyond text to images (Vision Transformer), audio, video, and even protein folding.