Skip to content

Chapter 15

Chapter 15 sections from Deep Learning with PyTorch.

deep-learning pytorch

Sections

  1. 15.1 Attention Mechanisms
  2. 15.2 Self-Attention
  3. 15.3 Multi-Head Attention
  4. 15.4 Positional Encoding
  5. 15.5 Transformer Encoders
  6. 15.6 Transformer Decoders
  7. 15.7 Efficient Attention Methods
Attention MechanismsAttention is a method for letting a model choose which parts of an input are most relevant when producing an output.
11 min
Self-AttentionSelf-attention is attention applied within a single sequence. The same input supplies the queries, keys, and values. Each position builds a new representation by reading from other positions in the same sequence.
8 min
Multi-Head AttentionMulti-head attention runs several attention operations in parallel.
8 min
Positional EncodingSelf-attention compares tokens by content. By itself, it has no built-in notion of token order.
10 min
Transformer EncodersA transformer encoder is a stack of layers that maps an input sequence to a contextual sequence representation.
7 min
Transformer DecodersA transformer decoder maps a partial output sequence to predictions for the next token or next output step.
7 min
Efficient Attention MethodsStandard self-attention compares every token with every other token. For a sequence of length $T$, this produces a $T \times T$ attention matrix. The cost grows quadratically with sequence length.
10 min