CoT Distillation & Rejection-Sampling FT
Module FT15 · Course 3 — LLM Fine-Tuning Masterclass
75 minutes · Pillar 4 — the cheaper-than-RL routes to a reasoning small model
FT14 gave you GRPO — the most powerful, most expensive, most unstable thing in the course. This module: two ways to get reasoning into a small model without paying the full RL cost — CoT distillation (SFT on a strong teacher's traces) and RFT (retrain on positively-rewarded candidates only).
Pillar 4 — Reasoning Models
The problem: reasoning is expensive
GRPO (FT14) works. It is also a full RL loop: policy model, rollouts buffer, reward, critic-free advantage, KL anchor — and it loves to diverge when rewards flatten, KL is mis-set, or rollouts collapse.
The FT15 question: can you get reasoning into a small model without paying the full RL cost? Yes — two ways.
| Route | How | RL? | The result that proved it |
|---|
| CoT distillation | SFT student on a strong teacher's <think> traces | No | R1-Distill-Qwen-32B beats o1-mini |
| RFT / RAFT | Generate → filter by reward → SFT on winners | No (reward only) | arXiv:2504.11343: competitive w/ GRPO |
| GRPO (FT14) | Full policy-gradient RL | Yes | most powerful, most expensive |
Both cheaper routes are terminal techniques, not just warmup steps. That distinction is the load-bearing finding of this module.
CoT distillation: borrow the reasoning
Structurally simple. A strong reasoning teacher (R1, o1, Claude/GPT with thinking) emits <think>...</think> traces. You curate them. You SFT a smaller student on the (problem, trace, answer) tuples. No reward model, no policy gradient, no KL anchor.
Data generation (once)
Teacher does the expensive search at data-gen time. Traces collected, verified, deduplicated, difficulty-balanced.
Student training
Ordinary SFT — the trace is part of the target text, so the student learns to emit the chain of thought token-by-token. At inference it generates its own <think> before answering.
Why this is steering, not teaching (FT00): the student's base already saw the math/code in pretraining. Distillation routes probability mass through the reasoning pathways it already has — reliably, in a structured trace. Limit: the student cannot exceed the teacher. Imitation has a ceiling.
The headline: DeepSeek-R1-Distill-Qwen-32B
The result that made distillation the standard, accessible technique.
| Step | What happened |
|---|
| Teacher | DeepSeek-R1 — a frontier reasoner trained with full RL on verifiable rewards (the FT14 lineage) |
| Data | ~800K curated reasoning samples (math + code + general), traces verified correct |
| Student | Qwen-32B base, trained with SFT only — no RL on the student |
| Result | Beats OpenAI o1-mini on AIME 2024 and MATH-500 |
A 32B model, trained with nothing more than SFT on a stronger model's traces, outperforms a frontier compressed-reasoning product. This is why the open ecosystem got small reasoning models cheaply. Distillation became the default recipe: strong teacher → generate traces → curate → SFT.
The R1-Distill family (Qwen 1.5B–70B, Llama 8B/70B) showed the recipe scales from consumer-GPU sizes to frontier-adjacent quality.
Rejection-sampling FT (RFT / RAFT) — the middle path
No strong teacher? Want the model to improve on its own attempts? Let the current model generate, keep only what the reward accepts, retrain on the winners.
| # | Step | What |
|---|
| 1 | Generate | Sample N candidates per problem (temp > 0, several rollouts) |
| 2 | Filter by reward | Verifiable only: math checks, code passes, answer matches. Keep the winners. |
| 3 | Retrain | SFT on accepted candidates only. Cross-entropy, no policy gradient. |
| 4 | Repeat | Generate with the improved model → filter → retrain. Each round trains on its own successes. |
Same data-generation machinery as GRPO (rollouts, verifiable reward) — radically different training step. GRPO learns from success AND failure via advantage × log-prob. RFT discards the failures and supervises on the successes only. You keep the reward; you discard the gradient noise.
arXiv:2504.11343 — simpler than expected
The finding: a simple rejection-sampling baseline is SURPRISINGLY COMPETITIVE with GRPO and PPO on math reasoning — same base, same data, same compute.
The field expected full RL to dominate — that RFT was a warmup, a poor man's RL. Not always equal, not always better, but competitive enough that the simplicity and stability advantages make it a legitimate terminal choice.
Why RFT is more stable than GRPO (structural, not incidental): the training step is SFT — cross-entropy on accepted candidates. SFT does not diverge the way policy gradients do. GRPO's failure modes — rollout collapse, KL drift, flat-reward noise — are all policy-gradient pathologies. RFT's worst case is a small accepted set or a wrong reward — both diagnosable data problems, inspectable by reading what you accepted.
The lesson: simpler methods should not be dismissed. RFT trades expressivity (ignores failures) for stability + diagnosability. Not just a warmup — a destination.
The decision: distillation vs RFT vs GRPO
Three techniques, one goal. Choose by three axes.
| Axis | Question | Gates |
|---|
| 1. Teacher? | Strong reasoning teacher available (R1, o1, Claude/GPT w/ thinking)? | Yes → distillation on the table (cheapest, consider first) |
| 2. Reward? | Verifiable (math, code, logic w/ checks)? | Yes → RFT & GRPO available. No → FT13 (preference, not reasoning FT) |
| 3. Budget? | Constrained / must-not-diverge, or large / willing to tune KL? | Constrained → RFT. Large + push frontier → GRPO |
Distillation
Cheapest. Have a strong teacher, don't need to exceed it. No RL.
RFT
Middle. Generates own data, filters by reward, no RL instability. Self-improvement, no teacher ceiling.
GRPO
Most powerful. Most expensive. For pushing beyond what distillation/RFT achieve.
Defensible stack: distill first (cheap, most of the way) → RFT to self-improve where the teacher was weak → GRPO only if RFT plateaus and budget remains. Each stage optional; many production reasoners stop at distillation.
The thinking-mode toggle: hybrid models
A reasoning model that always emits a long <think> block pays a latency tax on every query — including the trivial ones. The frontier's answer: one model, two modes, toggled per request.
Non-thinking mode
Answer directly. Low latency. "Capital of France?"
Thinking mode
Emit <think> trace, then answer. High accuracy. "Prove this integral converges."
Qwen3 is the canonical example. The toggle is not a separate model — same weights, steered by a control token / system instruction. One deployed model serves both the cheap-fast and expensive-accurate paths.
Thinking budget → adaptive compute. Not a binary toggle but a token budget: think up to N tokens, then answer. Spend more on hard problems, less on easy ones. Implication for FT: a hybrid student trains on a mixture of thinking and non-thinking traces, so it learns both modes and the ability to choose.
Anti-patterns
- Distilling from a weak teacher. Garbage in, garbage out. The student's ceiling is the teacher's. R1-Distill worked because R1 is frontier. Distill from a 7B that can't do the math → a small model that confidently emits wrong chains.
- RFT without reward verification. RFT's entire signal is the filter. A noisy reward (fuzzy matcher, lenient LLM judge, false-positive tests) → you train on incorrectly-accepted candidates → plausible-looking wrong answers. RFT demands a verifiable reward.
- Ignoring RFT's stability advantage over GRPO. Reaching for GRPO by default because "RL is more powerful." On many tasks RFT is competitive (2504.11343), cheaper, dramatically more stable. Try RFT first when the reward is verifiable; escalate to GRPO only when RFT plateaus.
- Treating distillation as only a warmup for RL. R1-Distill-Qwen-32B is a shipping model, not a checkpoint. Distill, evaluate against your target, escalate only if it falls short.
- Distilling traces the student can't imitate. Too long, too idiosyncratic, poor tokenizer fit → degraded imitation. Curate: trim, normalize, make it imitable.
The lab — "Distill Reasoning"
Generate CoT traces from a strong teacher on 200 GSM8K math problems, SFT a small student (MiniCPM5-1B or Qwen2.5-1.5B) on those traces, and measure the GSM8K lift vs the un-distilled base. Consumer-GPU. Runnable.
Data generation
200 GSM8K problems → teacher (local reasoning model or API) emits <think> traces. Verify the final answer; keep only correct traces. This is your distillation corpus.
SFT + eval
QLoRA the student on the traces. Merge. Run GSM8K on the distilled model AND the un-distilled base. The lift is your distillation.
Expect a measurable GSM8K accuracy gain (often +10–25 pts) from 200 traces alone — the base already knows arithmetic; you steered it to reason reliably. Stretch goal: an RFT variant — generate from the distilled model, filter by math reward, retrain on the winners.
Next: FT16 — Why Uncensored: The Legitimate Use Cases
What you can now do
- State the two cheaper-than-RL routes to a reasoning small model — CoT distillation and RFT — and why both matter after GRPO is on the table.
- Describe the R1-Distill pipeline and why a 32B SFT-only student beat o1-mini — the headline that made distillation the default accessible recipe.
- Distinguish RFT from GRPO: same rollouts/reward, different training step (SFT on winners vs policy gradient on all). Explain RFT's structural stability advantage.
- Make the three-way decision — distillation vs RFT vs GRPO — from teacher availability, reward verifiability, and compute/stability constraints.
- Explain the hybrid thinking toggle (Qwen3) and thinking budgets for adaptive compute.
- Spot the anti-patterns: weak teacher, RFT without verification, ignoring RFT's stability edge.
FT14 gave you the most powerful tool. FT15 gave you the ones you will actually use first.