Language Modeling

Language modeling is the task of predicting text sequences. A language model assigns probabilities to sequences of tokens and learns the statistical structure of language.

Given a token sequence:

a language model estimates:

Modern language models are the foundation of many NLP systems, including text generation, dialogue systems, translation systems, summarizers, code assistants, and retrieval-augmented systems.

Autoregressive Factorization

A sequence probability can be decomposed using the chain rule:

where:

The model predicts the next token conditioned on all previous tokens.

Example:

the cat sat on the

The model predicts the next token distribution:

A good language model assigns high probability to plausible continuations.

Vocabulary and Tokens

Language models operate on token sequences rather than raw text.

A tokenizer converts text into token IDs:

"The cat sleeps."

may become:

[314, 892, 12011, 13]

The vocabulary size is:

Each token corresponds to one row in the embedding matrix:

The input token IDs have shape:

[B, T]

After embedding:

[B, T, D]

where:

Next-Token Prediction

The central training objective of autoregressive language models is next-token prediction.

Suppose the token sequence is:

the cat sat

The model receives:

The model predicts one token at each position.

If:

logits: [B, T, V]

then the target tensor is:

targets: [B, T]

The loss compares predicted logits with the next-token targets.

Causal Masking

Autoregressive models must not see future tokens during training.

For the sequence:

the cat sat

the prediction for cat must not depend on sat.

Transformers enforce this using a causal attention mask.

For sequence length $T=4$:

1 0 0 0
1 1 0 0
1 1 1 0
1 1 1 1

Position $t$ may attend only to positions:

In PyTorch:

import torch

T = 4

mask = torch.tril(torch.ones(T, T))
print(mask)

Output:

tensor([[1., 0., 0., 0.],
        [1., 1., 0., 0.],
        [1., 1., 1., 0.],
        [1., 1., 1., 1.]])

Without causal masking, the model could trivially copy future tokens during training.

Cross-Entropy Training Objective

Autoregressive language models usually use cross-entropy loss.

Suppose:

logits: [B, T, V]
targets: [B, T]

We flatten the tensors:

import torch.nn as nn

loss_fn = nn.CrossEntropyLoss()

B, T, V = logits.shape

loss = loss_fn(
    logits.reshape(B * T, V),
    targets.reshape(B * T),
)

The target at each position is the next token.

The model minimizes:

A lower loss means the model assigns higher probability to the correct next token.

Perplexity

Perplexity is a common evaluation metric for language models.

If the average negative log-likelihood per token is:

then perplexity is:

Perplexity measures how uncertain the model is.

Interpretation:

If a model has perplexity 10, it behaves roughly as if it chooses among 10 equally likely options per step.

Lower perplexity usually indicates better language modeling performance, though it does not perfectly correlate with downstream usefulness or factual accuracy.

Recurrent Language Models

Before transformers, many language models used recurrent neural networks.

An RNN language model processes tokens sequentially:

The hidden state summarizes previous tokens.

An LSTM language model:

import torch.nn as nn

class LSTMLanguageModel(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim):
        super().__init__()

        self.embedding = nn.Embedding(vocab_size, embedding_dim)

        self.lstm = nn.LSTM(
            embedding_dim,
            hidden_dim,
            batch_first=True,
        )

        self.output = nn.Linear(hidden_dim, vocab_size)

    def forward(self, input_ids):
        x = self.embedding(input_ids)
        h, _ = self.lstm(x)
        logits = self.output(h)
        return logits

RNN models struggle with long-range dependencies and parallelization. Transformers largely replaced them for large-scale language modeling.

Transformer Language Models

Transformer language models use self-attention instead of recurrence.

Advantages:

A decoder-only transformer computes:

input_ids: [B, T]
-> embeddings
-> transformer blocks
-> hidden_states: [B, T, D]
-> output projection
-> logits: [B, T, V]

Each position predicts the next token.

Modern large language models such as GPT-style systems use this architecture.

Weight Tying

Many language models tie input embeddings and output projection weights.

The embedding matrix:

is reused for output logits:

Advantages:

Weight tying is now common in transformer language models.

Positional Encoding

Transformers do not inherently know token order.

Example:

dog bites man
man bites dog

contain the same tokens but different meanings.

Positional information must therefore be added.

A positional encoding provides a vector:

for each position $t$.

The transformer input becomes:

where:

Modern models use several positional methods:

Position encoding strongly affects long-context behavior.

Context Length

A transformer attends over a finite context window.

If the maximum context length is:

then tokens beyond $L$ positions cannot be attended to directly.

Longer context windows improve:

However, self-attention cost grows approximately as:

where $T$ is sequence length.

This motivates research into sparse attention, memory systems, state-space models, and linear attention methods.

Training Data

Language models are trained on large corpora.

Common data sources:

Training quality depends heavily on data quality.

Problems include:

Data filtering and deduplication are important parts of large-scale language model training.

Scaling Laws

Large language models exhibit scaling behavior.

Performance improves predictably as:

Empirical scaling laws show approximate power-law relationships between loss and compute scale.

However, scaling eventually encounters constraints:

Scaling alone does not guarantee reasoning ability, factuality, or safety.

Inference and KV Caching

Autoregressive generation repeatedly predicts one token at a time.

Naively recomputing all attention states each step is expensive.

Transformers therefore cache previous key and value tensors.

At generation step $t$:

where:

KV caching reduces generation complexity from recomputing the entire sequence at every step.

Sampling from Language Models

The model outputs logits:

A decoding algorithm converts logits into tokens.

Common methods:

Generation quality depends strongly on decoding configuration.

Low temperature:

High temperature:

Emergent Behaviors

Large language models sometimes exhibit capabilities not obvious in smaller models.

Examples:

The exact causes remain an active research topic.

Some behaviors appear gradually with scale. Others appear more abruptly.

Failure Modes

Language models have important limitations.

Language models optimize token prediction, not truth, reasoning correctness, or safety.

This distinction is critical when deploying systems in high-stakes settings.

Pretraining and Fine-Tuning

Most modern systems use two stages:

Pretraining uses large-scale next-token prediction.

Fine-tuning adapts the model for:

Instruction tuning and RLHF further shape model behavior.

PyTorch Training Example

A simplified transformer language model training step:

def training_step(model, batch, optimizer):
    input_ids = batch["input_ids"]
    targets = batch["targets"]

    logits = model(input_ids)
    # logits: [B, T, V]

    B, T, V = logits.shape

    loss_fn = nn.CrossEntropyLoss()

    loss = loss_fn(
        logits.reshape(B * T, V),
        targets.reshape(B * T),
    )

    optimizer.zero_grad(set_to_none=True)
    loss.backward()
    optimizer.step()

    return loss.item()

The targets are usually shifted by one token relative to the inputs.

Summary

Language modeling predicts token sequences autoregressively. Modern language models use transformer architectures with causal masking and next-token prediction objectives.

Key components include tokenization, embeddings, positional encoding, self-attention, output projections, and decoding algorithms. Training uses cross-entropy loss over large text corpora. Evaluation often uses perplexity.

Large language models extend basic language modeling into dialogue, reasoning, retrieval augmentation, tool use, and multimodal systems, but they still inherit core limitations from probabilistic next-token prediction.