Instruction Tuning

Pretraining teaches a language model to predict text. It does not directly teach the model to follow user instructions, answer safely, maintain dialogue structure, or format outputs in a useful way.

A pretrained model may continue text well but still behave poorly in interactive settings. For example, it may ignore instructions, generate irrelevant continuations, produce unsafe content, or imitate undesirable patterns from the training corpus.

Instruction tuning adapts a pretrained language model into a system that responds to tasks expressed in natural language.

The core idea is simple: instead of training on generic text continuation, we train the model on pairs of instructions and desired responses.

A typical example looks like:

Instruction	Response
“Translate this sentence into French.”	Correct French translation
“Summarize the following article.”	Summary
“Write a Python function for binary search.”	Python code
“Explain gradient descent.”	Educational explanation

Instruction tuning changes the model’s behavior from generic next-token continuation toward task-oriented response generation.

From Language Modeling to Task Following

A pretrained autoregressive model learns

p_\theta(x_t \mid x_{<t}).

The model predicts the next token from previous tokens. During pretraining, the corpus may contain instructions, answers, conversations, code, essays, and many unrelated text types mixed together.

Instruction tuning reorganizes the training distribution. Instead of arbitrary web text, the model receives structured examples:

(\text{instruction}, \text{response}).

The model then learns the conditional distribution

p_\theta(\text{response} \mid \text{instruction}).

This appears superficially similar to pretraining, since the model still predicts tokens autoregressively. The difference is the structure of the data distribution.

Pretraining teaches language structure broadly. Instruction tuning teaches cooperative task behavior.

Supervised Fine-Tuning

Instruction tuning is usually implemented as supervised fine-tuning, often abbreviated SFT.

The dataset contains demonstrations written by humans, synthetic systems, or mixtures of both. Each example includes:

Field	Purpose
System prompt	Defines global behavior
User prompt	Contains the instruction
Assistant response	Desired output

A training sample may look like:

<system>
You are a helpful assistant.

<user>
Explain backpropagation in simple terms.

<assistant>
Backpropagation computes gradients by applying the chain rule ...

The model is trained to predict the assistant tokens conditioned on all previous tokens.

The supervised loss is standard cross-entropy:

\mathcal{L} = -\sum_{t} \log p_\theta(y_t \mid x, y_{<t}),

where:

Symbol	Meaning
$x$	Prompt or instruction
$y_t$	Target response token
$y_{<t}$	Previous response tokens

Only assistant tokens usually contribute to the loss. User and system tokens provide conditioning context but are not prediction targets.

Why Instruction Tuning Works

Instruction tuning works because pretrained models already contain broad latent capabilities. Pretraining exposes the model to many tasks indirectly through text. The model may already contain useful representations for translation, reasoning, summarization, coding, and dialogue.

Instruction tuning teaches the model when and how to use those capabilities.

This is often described as eliciting latent knowledge rather than creating entirely new knowledge.

The model learns patterns such as:

Behavior	Example
Obeying instructions	Following formatting requests
Maintaining dialogue roles	Responding as assistant rather than continuing user text
Producing concise answers	Avoiding irrelevant continuation
Refusing unsafe requests	Safety alignment
Using chain-of-thought style reasoning	Stepwise solutions
Formatting outputs	Markdown, JSON, code blocks

A relatively small instruction dataset can significantly change model behavior because the pretrained model already contains strong language representations.

Prompt Formatting and Chat Templates

Modern instruction-tuned models usually rely on structured prompt templates.

A dialogue is converted into a token sequence with role markers:

<system>
You are a concise assistant.

<user>
What is overfitting?

<assistant>

The model generates the assistant continuation.

Different model families use different formatting conventions:

Model family	Example format
ChatML-style	`<system>`, `<user>`, `<assistant>`
Instruction-style	`### Instruction:`
Llama-style chat	`[INST] ... [/INST]`
XML-style	`<instruction>` tags
JSON-style	Structured objects

The formatting matters because the model learns statistical associations between role markers and behavior.

Changing the template can affect performance substantially.

Multi-Task Instruction Tuning

Instruction datasets often combine many tasks:

Task type	Example
Question answering	Factual responses
Summarization	Compress documents
Translation	Convert languages
Coding	Generate programs
Classification	Assign labels
Dialogue	Multi-turn interaction
Reasoning	Solve structured problems
Tool use	Call APIs or functions

The model learns a unified interface: natural language instructions.

Instead of separate models for each task, one instruction-tuned model learns many conditional behaviors.

This unification is one reason large language models are flexible. The instruction itself acts as part of the program specification.

Zero-Shot and Few-Shot Generalization

Instruction tuning improves zero-shot generalization. A zero-shot task is one where the model receives only the instruction, without examples.

Example:

Classify this review as positive or negative:
"The battery life is excellent."

The model may perform the task correctly even without task-specific training examples in the prompt.

Few-shot prompting provides demonstrations inside the prompt itself:

Input: "Amazing product."
Label: Positive

Input: "Very disappointing."
Label: Negative

Input: "Battery life is excellent."
Label:

Instruction tuning improves the model’s ability to interpret such prompts consistently.

Pretraining alone may give weak task following. Instruction tuning calibrates the model toward cooperative interaction.

Chain-of-Thought Supervision

Some instruction datasets include intermediate reasoning steps rather than only final answers.

Example:

Question: If a train travels 60 km in 2 hours, what is its average speed?

Reasoning:
Speed = distance / time
= 60 / 2
= 30 km/h

Answer: 30 km/h

Training on reasoning traces can improve performance on multi-step reasoning tasks.

The model learns statistical patterns associated with decomposition, intermediate computation, verification, and explanation.

This is called chain-of-thought supervision.

However, chain-of-thought introduces several concerns:

Concern	Description
Verbosity	Longer outputs increase cost
Faithfulness	Reasoning text may not reflect internal computation
Data contamination	Public reasoning datasets may leak benchmarks
Safety	Hidden reasoning may expose unsafe internal content

Some systems therefore separate visible reasoning from internal latent reasoning.

Instruction Diversity

An instruction-tuned model must generalize across many instruction styles.

If the dataset is too narrow, the model may overfit to specific phrasing. High-quality instruction tuning datasets therefore vary:

Variation	Example
Wording	“Summarize” versus “Give a short overview”
Tone	Formal versus conversational
Format	JSON, markdown, prose
Difficulty	Simple and complex tasks
Domain	Science, law, code, dialogue
Language	Multilingual prompts

Diversity improves robustness.

The model learns abstract task semantics rather than memorizing exact templates.

Synthetic Instruction Data

Human-written instruction datasets are expensive. Many modern systems therefore generate synthetic instruction data.

A strong model can generate:

Synthetic component	Example
Instructions	“Write a SQL query for…”
Responses	High-quality completions
Reasoning traces	Stepwise derivations
Critiques	Error analysis
Preference labels	Ranking candidate answers

Synthetic data generation creates a recursive training loop:

Train a strong model.
Use the model to generate instruction data.
Filter or rank the outputs.
Train a new model on the expanded dataset.

This process scales data generation beyond purely human annotation.

However, synthetic data can amplify errors, stylistic artifacts, and model biases. Filtering and evaluation become increasingly important.

Catastrophic Forgetting

Instruction tuning changes the model distribution. If done poorly, it can damage capabilities learned during pretraining.

This is called catastrophic forgetting.

Possible symptoms include:

Problem	Example
Reduced factual recall	Worse knowledge retrieval
Lower language diversity	Repetitive responses
Reduced multilingual ability	Strong English bias
Style collapse	Overly uniform outputs
Short-answer bias	Failure on long reasoning tasks

Instruction tuning datasets are much smaller than pretraining corpora. Aggressive fine-tuning can therefore distort the pretrained representation space.

Several techniques reduce forgetting:

Technique	Purpose
Small learning rates	Preserve pretrained features
Mixed training data	Blend instruction and pretraining text
Parameter-efficient tuning	Update fewer parameters
Regularization	Prevent large parameter drift
Replay buffers	Reintroduce older data

Balancing specialization and preservation is a major practical challenge.

Parameter-Efficient Instruction Tuning

Full fine-tuning updates all parameters. For large models, this is expensive.

Parameter-efficient fine-tuning updates only small subsets of parameters.

Common approaches include:

Method	Idea
LoRA	Low-rank weight updates
Adapters	Small trainable modules inserted into layers
Prefix tuning	Train virtual prompt vectors
Prompt tuning	Learn soft prompts
BitFit	Train only bias terms

For example, LoRA approximates weight updates using low-rank matrices:

\Delta W = AB,

where $A$ and $B$ have much smaller rank than $W$ .

This greatly reduces memory and compute requirements while preserving much of the model’s performance.

Parameter-efficient tuning is widely used for domain adaptation and open-source fine-tuning.

Instruction Tuning and Alignment

Instruction tuning improves usability, but it does not fully solve alignment.

A model may still:

Failure mode	Example
Hallucinate	Invent facts
Follow harmful requests	Unsafe outputs
Over-refuse	Reject harmless queries
Manipulate users	Social persuasion
Leak training data	Memorized content
Produce biased responses	Social stereotypes

Instruction tuning mainly teaches behavioral imitation from demonstrations.

More advanced alignment methods, such as reinforcement learning from human feedback, constitutional training, and preference optimization, further shape the model’s behavior.

PyTorch View of Supervised Fine-Tuning

Suppose a tokenized batch has shape:

[B, T]

where:

Symbol	Meaning
`B`	Batch size
`T`	Sequence length

The model produces logits:

[B, T, V]

where $V$ is the vocabulary size.

Instruction tuning usually masks non-assistant tokens from the loss.

Example:

import torch
import torch.nn.functional as F

# input_ids: [B, T]
# labels: assistant tokens kept, others set to -100

logits = model(input_ids)

loss = F.cross_entropy(
    logits.view(-1, logits.size(-1)),
    labels.view(-1),
    ignore_index=-100
)

The label tensor may look like:

Input tokens:   [SYSTEM USER USER ASSISTANT ASSISTANT]
Loss mask:      [  -100  -100  -100     y1        y2 ]

Only assistant outputs contribute gradients.

Data Mixture and Curriculum

Instruction tuning datasets are often mixtures of many sources:

Source	Example
Human annotation	Expert-written prompts
Public QA datasets	Reading comprehension
Code datasets	Programming tasks
Synthetic conversations	Generated dialogues
Tool traces	API interaction examples
Reasoning datasets	Math and logic problems

The mixture ratio matters.

Too much conversational data may weaken reasoning. Too much code data may distort natural language style. Too much synthetic data may create repetitive outputs.

Some training pipelines also use curricula:

Easier tasks first.
More complex reasoning later.
Specialized tasks near the end.

Curriculum design can improve stability and convergence.

Why Instruction Tuning Changed Language Models

Early large language models were often difficult to control. Users needed carefully engineered prompts to obtain reliable behavior.

Instruction tuning changed the interaction model. Instead of treating the model as a generic text completer, users could treat it as a cooperative assistant.

This shift enabled:

Capability	Impact
Conversational systems	Multi-turn dialogue
General-purpose assistants	Broad task coverage
Tool integration	API and retrieval systems
Coding assistants	Natural language programming
Educational tutors	Explanatory interaction
Agent systems	Planning and execution loops

Instruction tuning therefore transformed pretrained language models into usable interactive systems.

Summary

Instruction tuning adapts pretrained language models for task-following behavior using supervised examples of instructions and desired responses.

The model learns conditional generation:

p_\theta(\text{response} \mid \text{instruction}).

Instruction tuning improves usability, formatting, dialogue structure, reasoning style, and zero-shot task generalization.

Modern instruction-tuned systems rely on structured prompts, diverse datasets, chain-of-thought supervision, synthetic data generation, and parameter-efficient adaptation methods.

Instruction tuning greatly improves interaction quality, but it does not fully solve factuality, robustness, or safety. Later alignment stages further shape model behavior beyond supervised imitation.