Course: Course 3 — LLM Fine-Tuning Masterclass Module: FT15 — CoT Distillation and Rejection-Sampling FT Duration: 75 minutes Level: Senior Engineer and above Prerequisites: FT14 (GRPO and Verifiable Rewards)
After completing this module, you will be able to:
<think>…</think> traces, the traces are curated into ~800K samples, and a smaller student SFTs on them — no RL required — yielding R1-Distill-Qwen-32B, which beats OpenAI o1-mini on several benchmarks.FT14 gave you GRPO — Group Relative Policy Optimization on verifiable rewards. It works. It is also the most expensive, most unstable thing in this course. GRPO is a real RL loop: a policy model, a rollouts buffer, a reward signal, a critic-free advantage estimate, a KL anchor to a reference, and the whole machinery of policy-gradient training that loves to diverge when the reward landscape flattens, the KL coefficient is mis-set, or the rollout distribution collapses onto a single high-reward string.
The question FT15 answers is direct: can you get reasoning into a small model without paying the full RL cost?
The answer is yes, two ways, and the open ecosystem used both to catch up to the frontier reasoning labs in a matter of months:
Both are legitimate terminal techniques, not merely warmup steps. That distinction is load-bearing — it is the finding from arXiv:2504.11343 that this module hinges on.
Chain-of-thought distillation is structurally simple. You have a strong reasoning teacher — a frontier model with thinking enabled (DeepSeek-R1, OpenAI o1, or a Claude/GPT-class model invoked with extended thinking). You give it problems. It produces solutions that include an explicit reasoning trace, conventionally wrapped in <think>…</think> tags. You collect those traces, curate them (filter for correctness, deduplicate, balance difficulty), and SFT a smaller student on the resulting (problem, think-trace, answer) tuples. The student learns to emit the reasoning trace as part of its generation.
Two things to notice immediately.
First, this is SFT, not RL. There is no reward model in the training loop, no policy gradient, no KL anchor. The teacher did the expensive search once at data-generation time; the student just imitates. The compute profile is that of an ordinary SFT run, not an RL run.
Second, the student is learning to reason in the teacher's style. The trace is part of the target text, so the student is trained to produce the chain of thought token-by-token. At inference, the student generates its own <think> block before committing to an answer. This is what makes distillation produce a genuine reasoning model, not a model that merely memorized answers.
The result that made distillation the standard, accessible technique is DeepSeek-R1-Distill-Qwen-32B. DeepSeek took their full R1 model (a frontier reasoning model trained with RL on verifiable rewards — the FT14 lineage) and used it as a teacher. They generated a large corpus of reasoning traces, curated them down to roughly 800K samples spanning math, code, and general reasoning, and SFT'd an off-the-shelf Qwen-32B base on those traces. No RL on the student. Just SFT on the distilled data.
The result: R1-Distill-Qwen-32B beats OpenAI o1-mini on several benchmarks, including AIME 2024 and MATH-500. A 32B model, trained with nothing more than supervised fine-tuning on data generated by a stronger model, outperforms a frontier compressed-reasoning product. That single fact is why the open ecosystem got small reasoning models cheaply. Distillation became the default recipe: find or train a strong teacher, generate traces, curate, SFT. The R1-Distill family (Qwen-1.5B through 70B, Llama-8B and 70B) demonstrated that the recipe scales down to consumer-GPU sizes and up to frontier-adjacent quality.
This is the clearest empirical statement in the module: for getting reasoning into a small model, distillation is the cheapest path that works at scale, and it works without RL.
Note the FT00 framing. The student is not learning new reasoning ability the base did not have — the Qwen-32B base already saw enormous amounts of math and code during pretraining. Distillation is steering the student to reliably emit a structured reasoning trace and to use the reasoning pathways the base already possesses. The teacher's trace is the steering wheel. This is why a 32B distilled model can punch above its pretraining-weight class: the capability was latent; distillation routes probability mass through it reliably.
The corollary, and the limit: distillation cannot make the student exceed the teacher. You are imitating the teacher's traces; the student's ceiling is the teacher's ceiling (minus whatever the smaller model forgets). If you need to push beyond what the teacher can produce, you need RL on the student's own rollouts — that is FT14, or RFT (15.3).
Distillation is the right call when:
It is the wrong call when you have no strong teacher (you cannot distill reasoning the teacher does not have), or when the whole point is to push past the teacher's quality.
Rejection-sampling fine-tuning — RFT, also called RAFT (Rejection sampling Fine-Tuning) in the arXiv:2504.11343 paper — is the middle path. You do not have a teacher whose traces you can copy, or you want the student to learn from its own attempts. So you let the current model generate.
The loop, iterated:
This is the same data-generation machinery as GRPO (rollouts, verifiable reward) but a radically different training step. GRPO uses the rollouts to compute a policy gradient — it learns from both the successes and the failures, pushing up the good and down the bad, with all the instability that implies. RFT throws away the failures and supervises on the successes only. You keep the reward signal; you discard the gradient noise.
The finding that gives RFT its place in this module is from arXiv:2504.11343. The authors set up a careful comparison: rejection-sampling fine-tuning versus GRPO and PPO on math reasoning, same base model, same data, same compute. The expectation in the field was that full RL (GRPO, PPO) would dominate — that rejection sampling was a warmup, a poor man's RL.
The result: a simple rejection-sampling baseline is surprisingly competitive with GRPO and PPO on math reasoning. Not always equal, not always better, but competitive — close enough that the simplicity and stability advantages of RFT make it a legitimate choice, not merely a stepping stone.
The lesson, and it is the module's second load-bearing claim: simpler methods should not be dismissed. RFT is cheaper than full RL (no policy-gradient machinery, no critic, no KL tuning), more stable (you are doing SFT, which does not diverge the way policy gradients can), and a legitimate standalone technique. It is not just a warmup step before you "graduate" to GRPO. For many reasoning tasks, RFT is the destination.
The stability advantage is not incidental — it is structural. GRPO's policy gradient is advantage × log-prob, summed over rollouts. When the rollout distribution collapses (the model finds one high-reward string and keeps generating it), the gradient becomes degenerate. When the KL coefficient is mis-set, the model drifts from the reference and the reward signal decouples from genuine improvement. When the reward landscape is flat (few rollouts get any reward), the advantage estimates are noise. Each of these is a real, observed failure mode of RL fine-tuning.
RFT has none of these. The training step is SFT — cross-entropy on accepted candidates. SFT does not diverge in the way policy gradients do. The worst case in RFT is that you accept too few candidates and the training set is small, or you accept bad candidates because the reward is wrong — both are data-quality problems you can diagnose by inspecting the accepted set. The worst case in GRPO is a silent mode collapse that ruins the model while the reward looks fine. RFT trades expressivity (it cannot learn from failures) for diagnosability (what you train on is exactly what you accepted).
RFT is the right call when:
It is the wrong call when the reward is not verifiable (RFT needs the reward to filter — a noisy reward means you train on garbage), or when you specifically need the expressivity of learning from failures (GRPO's edge on the hardest exploration problems).
This is the judgment the module exists to teach. Three techniques, one goal — reasoning in a small model. Choose by three axes.
A common, defensible stack: distill first (cheap, gets you most of the way), then RFT to self-improve on the areas the teacher was weak, then GRPO only if RFT plateaus and you have the budget. Each stage is optional; many production reasoning models stop at distillation.
A reasoning model that always emits a long <think> block pays a latency tax on every query — including the trivial ones where the answer is obvious. The frontier's answer is the hybrid model: a single model that can toggle thinking on or off, controlled by a flag in the prompt or the generation config.
Qwen3 is the canonical example. A Qwen3 model can be invoked in thinking mode (emit the trace, then answer) or non-thinking mode (answer directly), selected per request. The toggle is not a separate model — it is the same weights, steered by a control token or system instruction. This lets one deployed model serve both the cheap fast path (non-thinking, for "what's the capital of France") and the expensive accurate path (thinking, for "prove this integral converges").
The deeper idea is the thinking budget — a mechanism for adaptive compute. Rather than a binary toggle, the model is given a token budget for thinking: think for up to N tokens, then answer. This lets the model spend more compute on hard problems and less on easy ones, allocating inference cost to where it buys accuracy. Thinking-budget mechanisms are how hybrid models avoid the latency tax without giving up reasoning quality on the hard cases.
The implication for fine-tuning: a hybrid model is trained on a mixture of thinking and non-thinking traces, so it learns both modes and the ability to choose. Distillation data for a hybrid model includes both long-trace examples (thinking on) and direct-answer examples (thinking off), plus ideally examples where the difficulty warrants one or the other. This is more involved than pure CoT distillation, but it produces a model that is deployable across the whole latency/accuracy tradeoff.
Garbage in, garbage out. The student's reasoning ceiling is the teacher's. If you distill from a model that cannot actually reason well, you get a small model that imitates mediocre reasoning. The R1-Distill result worked because R1 is a frontier reasoner. Distilling from a 7B base that itself cannot do the math will produce a student that confidently emits wrong chains. The teacher must be genuinely strong on the target domain.
RFT's entire signal is the filter. If the reward is noisy — a fuzzy matcher, a lenient LLM judge, a test suite with false positives — you train on incorrectly-accepted candidates and the model learns to produce plausible-looking wrong answers. RFT demands a verifiable reward: math that checks, code that passes strict tests, logic with a ground truth. A preference-style reward is not sufficient for RFT; it is a different regime (FT13).
The most expensive anti-pattern: reaching for GRPO by default because "RL is more powerful." On many reasoning tasks, RFT is competitive (arXiv:2504.11343), cheaper, and dramatically more stable. If you skip RFT and go straight to GRPO, you pay the full RL cost — compute, tuning, divergence risk — for gains that RFT may already capture. Try RFT first when the reward is verifiable; escalate to GRPO only when RFT plateaus with budget remaining.
Distillation is a terminal technique. R1-Distill-Qwen-32B is a shipping model, not a warmup checkpoint. If you treat distillation as merely a pre-RL initialization step, you will spend RL compute to gain what may be marginal improvements over a strong distilled baseline. Distill, evaluate the distilled model against your target, and only escalate if it falls short.
If the teacher's traces are too long, too stylistically idiosyncratic, or use tokens the student's tokenizer handles poorly, the student will learn a degraded imitation. Curate the traces: trim, normalize, ensure the format is something the student can reproduce. Distillation is imitation; imitation requires the target to be imitable.
| Term | Definition |
|---|---|
| CoT distillation | SFT a smaller student on chain-of-thought reasoning traces generated by a stronger teacher model; no RL in the training loop |
| Teacher model | The strong reasoning model (R1, o1, Claude/GPT with thinking) that generates the traces distilled into the student |
<think> trace |
The explicit reasoning block a reasoning model emits before its answer; the distillation target |
| DeepSeek-R1-Distill-Qwen-32B | The 32B model distilled from R1 via SFT on ~800K curated reasoning samples; beats o1-mini on several benchmarks — the headline distillation result |
| Rejection-sampling FT (RFT / RAFT) | Iteratively SFT on only positively-rewarded candidates generated by the current model; generate → filter by reward → retrain on winners |
| Verifiable reward | A reward that can be checked exactly (math correctness, code passing tests); required for RFT and GRPO |
| arXiv:2504.11343 | The RAFT paper; finding that simple rejection-sampling FT is surprisingly competitive with GRPO and PPO on math reasoning |
| Hybrid model (thinking toggle) | A single model that can run in thinking or non-thinking mode per request (e.g., Qwen3), avoiding the always-on latency tax |
| Thinking budget | A token/length budget for thinking, allowing adaptive compute — more on hard problems, less on easy ones |
| GRPO | Group Relative Policy Optimization (FT14); full RL on verifiable rewards; most powerful, most expensive, most unstable of the three |
See 07-lab-spec.md. The "Distill Reasoning" lab: generate CoT traces from a strong teacher on 200 GSM8K math problems, SFT a small student (MiniCPM5-1B or Qwen2.5-1.5B) on those traces, and measure the GSM8K lift versus the un-distilled base. Consumer-GPU. Stretch goal: an RFT variant that filters the student's own candidates.
# Module FT15 — CoT Distillation and Rejection-Sampling FT **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FT15 — CoT Distillation and Rejection-Sampling FT **Duration**: 75 minutes **Level**: Senior Engineer and above **Prerequisites**: FT14 (GRPO and Verifiable Rewards) --- ## Learning Objectives After completing this module, you will be able to: 1. State the **two cheaper-than-RL routes** to a reasoning small model — CoT distillation and rejection-sampling fine-tuning (RFT/RAFT) — and why both matter even after GRPO (FT14) is on the table. 2. Describe the **DeepSeek-R1-Distill pipeline**: a strong reasoning teacher emits `<think>…</think>` traces, the traces are curated into ~800K samples, and a smaller student SFTs on them — **no RL required** — yielding R1-Distill-Qwen-32B, which beats OpenAI o1-mini on several benchmarks. 3. Distinguish **RFT/RAFT** from full RL: iteratively fine-tune on *positively-rewarded* candidates only (correct math, passing code), retrain on the winners — simpler and more stable than GRPO/PPO, and per arXiv:2504.11343, **surprisingly competitive** with them. 4. Make the **three-way decision** — distillation vs RFT vs GRPO — from compute budget, stability requirements, and whether a strong teacher is available. 5. Explain the **thinking-mode toggle** in hybrid models (Qwen3): a single model that can think or not, and thinking-budget mechanisms for adaptive compute. 6. Spot the anti-patterns: distilling from a weak teacher, RFT without reward verification, and ignoring RFT's stability advantage over GRPO. --- # 15.1 — The Problem: Reasoning Is Expensive FT14 gave you GRPO — Group Relative Policy Optimization on verifiable rewards. It works. It is also the most expensive, most unstable thing in this course. GRPO is a real RL loop: a policy model, a rollouts buffer, a reward signal, a critic-free advantage estimate, a KL anchor to a reference, and the whole machinery of policy-gradient training that loves to diverge when the reward landscape flattens, the KL coefficient is mis-set, or the rollout distribution collapses onto a single high-reward string. The question FT15 answers is direct: **can you get reasoning into a small model without paying the full RL cost?** The answer is yes, two ways, and the open ecosystem used both to catch up to the frontier reasoning labs in a matter of months: - **CoT distillation** — borrow reasoning traces from a strong teacher and SFT the student on them. No RL. This is the move that produced DeepSeek-R1-Distill-Qwen-32B. - **Rejection-sampling fine-tuning (RFT / RAFT)** — generate candidates with the current model, keep only the ones the reward accepts, retrain on the winners. A middle path between distillation and RL: you still need a reward, but you avoid the instability of policy gradients. Both are legitimate *terminal* techniques, not merely warmup steps. That distinction is load-bearing — it is the finding from arXiv:2504.11343 that this module hinges on. --- # 15.2 — CoT Distillation: Borrow the Reasoning ## The mechanism Chain-of-thought distillation is structurally simple. You have a strong reasoning teacher — a frontier model with thinking enabled (DeepSeek-R1, OpenAI o1, or a Claude/GPT-class model invoked with extended thinking). You give it problems. It produces solutions that include an explicit reasoning trace, conventionally wrapped in `<think>…</think>` tags. You collect those traces, curate them (filter for correctness, deduplicate, balance difficulty), and SFT a smaller student on the resulting `(problem, think-trace, answer)` tuples. The student learns to *emit* the reasoning trace as part of its generation. Two things to notice immediately. First, **this is SFT, not RL.** There is no reward model in the training loop, no policy gradient, no KL anchor. The teacher did the expensive search once at data-generation time; the student just imitates. The compute profile is that of an ordinary SFT run, not an RL run. Second, **the student is learning to reason in the teacher's style.** The trace is part of the target text, so the student is trained to *produce* the chain of thought token-by-token. At inference, the student generates its own `<think>` block before committing to an answer. This is what makes distillation produce a genuine reasoning model, not a model that merely memorized answers. ## The headline result: DeepSeek-R1-Distill-Qwen-32B The result that made distillation the standard, accessible technique is DeepSeek-R1-Distill-Qwen-32B. DeepSeek took their full R1 model (a frontier reasoning model trained with RL on verifiable rewards — the FT14 lineage) and used it as a teacher. They generated a large corpus of reasoning traces, curated them down to roughly **800K samples** spanning math, code, and general reasoning, and SFT'd an off-the-shelf Qwen-32B base on those traces. No RL on the student. Just SFT on the distilled data. The result: **R1-Distill-Qwen-32B beats OpenAI o1-mini on several benchmarks**, including AIME 2024 and MATH-500. A 32B model, trained with nothing more than supervised fine-tuning on data generated by a stronger model, outperforms a frontier compressed-reasoning product. That single fact is why the open ecosystem got small reasoning models cheaply. Distillation became the default recipe: find or train a strong teacher, generate traces, curate, SFT. The R1-Distill family (Qwen-1.5B through 70B, Llama-8B and 70B) demonstrated that the recipe scales down to consumer-GPU sizes and up to frontier-adjacent quality. This is the clearest empirical statement in the module: **for getting reasoning into a small model, distillation is the cheapest path that works at scale, and it works without RL.** ## Why distillation is steering, not teaching Note the FT00 framing. The student is not learning *new reasoning ability* the base did not have — the Qwen-32B base already saw enormous amounts of math and code during pretraining. Distillation is steering the student to *reliably emit* a structured reasoning trace and to *use* the reasoning pathways the base already possesses. The teacher's trace is the steering wheel. This is why a 32B distilled model can punch above its pretraining-weight class: the capability was latent; distillation routes probability mass through it reliably. The corollary, and the limit: **distillation cannot make the student exceed the teacher.** You are imitating the teacher's traces; the student's ceiling is the teacher's ceiling (minus whatever the smaller model forgets). If you need to push *beyond* what the teacher can produce, you need RL on the student's own rollouts — that is FT14, or RFT (15.3). ## When distillation is the right call Distillation is the right call when: - You have access to a strong reasoning teacher (a frontier API with thinking, or your own R1/o1-class model). - You want a small, cheap reasoning model for deployment. - You do not need the student to exceed the teacher. - You want to avoid RL instability entirely. It is the wrong call when you have no strong teacher (you cannot distill reasoning the teacher does not have), or when the whole point is to push past the teacher's quality. --- # 15.3 — Rejection-Sampling Fine-Tuning (RFT / RAFT) ## The mechanism Rejection-sampling fine-tuning — RFT, also called RAFT (Rejection sampling Fine-Tuning) in the arXiv:2504.11343 paper — is the middle path. You do not have a teacher whose traces you can copy, or you want the student to learn from *its own* attempts. So you let the current model generate. The loop, iterated: 1. **Generate.** For each problem in your dataset, sample N candidates from the current model (temperature > 0, several per problem). 2. **Filter by reward.** Score each candidate with a verifiable reward — does the math check, does the code pass tests, does the answer match. Keep only the positively-rewarded candidates. 3. **Retrain on the winners.** SFT the model on the accepted candidates. This becomes the new current model. 4. **Repeat.** Generate with the improved model, filter, retrain. Each round, the model is trained only on its own successes. This is the same data-generation machinery as GRPO (rollouts, verifiable reward) but a radically different training step. GRPO uses the rollouts to compute a policy gradient — it learns from *both* the successes and the failures, pushing up the good and down the bad, with all the instability that implies. RFT throws away the failures and supervises on the successes only. You keep the reward signal; you discard the gradient noise. ## The arXiv:2504.11343 finding: simpler than expected The finding that gives RFT its place in this module is from arXiv:2504.11343. The authors set up a careful comparison: rejection-sampling fine-tuning versus GRPO and PPO on math reasoning, same base model, same data, same compute. The expectation in the field was that full RL (GRPO, PPO) would dominate — that rejection sampling was a warmup, a poor man's RL. The result: **a simple rejection-sampling baseline is surprisingly competitive with GRPO and PPO on math reasoning.** Not always equal, not always better, but competitive — close enough that the simplicity and stability advantages of RFT make it a legitimate choice, not merely a stepping stone. The lesson, and it is the module's second load-bearing claim: **simpler methods should not be dismissed.** RFT is cheaper than full RL (no policy-gradient machinery, no critic, no KL tuning), more stable (you are doing SFT, which does not diverge the way policy gradients can), and a legitimate standalone technique. It is not just a warmup step before you "graduate" to GRPO. For many reasoning tasks, RFT is the destination. ## Why RFT is more stable than GRPO The stability advantage is not incidental — it is structural. GRPO's policy gradient is `advantage × log-prob`, summed over rollouts. When the rollout distribution collapses (the model finds one high-reward string and keeps generating it), the gradient becomes degenerate. When the KL coefficient is mis-set, the model drifts from the reference and the reward signal decouples from genuine improvement. When the reward landscape is flat (few rollouts get any reward), the advantage estimates are noise. Each of these is a real, observed failure mode of RL fine-tuning. RFT has none of these. The training step is SFT — cross-entropy on accepted candidates. SFT does not diverge in the way policy gradients do. The worst case in RFT is that you accept too few candidates and the training set is small, or you accept bad candidates because the reward is wrong — both are data-quality problems you can diagnose by inspecting the accepted set. The worst case in GRPO is a silent mode collapse that ruins the model while the reward looks fine. RFT trades expressivity (it cannot learn from failures) for diagnosability (what you train on is exactly what you accepted). ## When RFT is the right call RFT is the right call when: - You have a verifiable reward (math, code, unit-testable logic) but no strong teacher. - You want the model to improve on its own rollouts (self-improvement without a teacher ceiling). - You want to avoid RL instability but still push past plain SFT. - Your compute budget cannot absorb a full RL run's iteration cost. It is the wrong call when the reward is not verifiable (RFT needs the reward to *filter* — a noisy reward means you train on garbage), or when you specifically need the expressivity of learning from failures (GRPO's edge on the hardest exploration problems). --- # 15.4 — The Decision: Distillation vs RFT vs GRPO This is the judgment the module exists to teach. Three techniques, one goal — reasoning in a small model. Choose by three axes. ## Axis 1: Do you have a strong teacher? - **Yes.** Distillation is on the table. It is the cheapest path and you should consider it first. - **No.** Distillation is off the table. You must generate your own signal — RFT or GRPO. ## Axis 2: Is your reward verifiable? - **Yes (math, code, logic with checks).** RFT and GRPO are both available. - **No (preference, style, open-ended).** Neither RFT nor GRPO applies cleanly. You are in preference-optimization territory (FT13), not reasoning FT. ## Axis 3: Compute budget and stability requirements - **Constrained budget, must-not-diverge requirement.** RFT. SFT stability, reward-filtered data, no policy-gradient machinery. - **Large budget, willing to tune KL and babysit rollouts, need to push the frontier.** GRPO. The most powerful, the most expensive, the most likely to find improvements RFT cannot. ## The decision, in one line each - **Distillation** — cheapest, if you have a strong teacher and do not need the student to exceed it. - **RFT** — middle ground; generates its own data, filters by reward, no RL instability. - **GRPO** — most powerful, most expensive, for pushing beyond what distillation/RFT achieve. A common, defensible stack: distill first (cheap, gets you most of the way), then RFT to self-improve on the areas the teacher was weak, then GRPO only if RFT plateaus and you have the budget. Each stage is optional; many production reasoning models stop at distillation. --- # 15.5 — The Thinking-Mode Question: Hybrid Models A reasoning model that *always* emits a long `<think>` block pays a latency tax on every query — including the trivial ones where the answer is obvious. The frontier's answer is the **hybrid model**: a single model that can toggle thinking on or off, controlled by a flag in the prompt or the generation config. Qwen3 is the canonical example. A Qwen3 model can be invoked in thinking mode (emit the trace, then answer) or non-thinking mode (answer directly), selected per request. The toggle is not a separate model — it is the same weights, steered by a control token or system instruction. This lets one deployed model serve both the cheap fast path (non-thinking, for "what's the capital of France") and the expensive accurate path (thinking, for "prove this integral converges"). The deeper idea is the **thinking budget** — a mechanism for adaptive compute. Rather than a binary toggle, the model is given a token budget for thinking: think for up to N tokens, then answer. This lets the model spend more compute on hard problems and less on easy ones, allocating inference cost to where it buys accuracy. Thinking-budget mechanisms are how hybrid models avoid the latency tax without giving up reasoning quality on the hard cases. The implication for fine-tuning: a hybrid model is trained on a *mixture* of thinking and non-thinking traces, so it learns both modes and the ability to choose. Distillation data for a hybrid model includes both long-trace examples (thinking on) and direct-answer examples (thinking off), plus ideally examples where the difficulty warrants one or the other. This is more involved than pure CoT distillation, but it produces a model that is deployable across the whole latency/accuracy tradeoff. --- # 15.6 — Anti-Patterns ### Distilling from a weak teacher Garbage in, garbage out. The student's reasoning ceiling is the teacher's. If you distill from a model that cannot actually reason well, you get a small model that imitates mediocre reasoning. The R1-Distill result worked because R1 is a frontier reasoner. Distilling from a 7B base that itself cannot do the math will produce a student that confidently emits wrong chains. **The teacher must be genuinely strong on the target domain.** ### RFT without reward verification RFT's entire signal is the filter. If the reward is noisy — a fuzzy matcher, a lenient LLM judge, a test suite with false positives — you train on incorrectly-accepted candidates and the model learns to produce plausible-looking wrong answers. RFT demands a *verifiable* reward: math that checks, code that passes strict tests, logic with a ground truth. A preference-style reward is not sufficient for RFT; it is a different regime (FT13). ### Ignoring RFT's stability advantage over GRPO The most expensive anti-pattern: reaching for GRPO by default because "RL is more powerful." On many reasoning tasks, RFT is competitive (arXiv:2504.11343), cheaper, and dramatically more stable. If you skip RFT and go straight to GRPO, you pay the full RL cost — compute, tuning, divergence risk — for gains that RFT may already capture. **Try RFT first when the reward is verifiable; escalate to GRPO only when RFT plateaus with budget remaining.** ### Treating distillation as a warmup for RL only Distillation is a terminal technique. R1-Distill-Qwen-32B is a shipping model, not a warmup checkpoint. If you treat distillation as merely a pre-RL initialization step, you will spend RL compute to gain what may be marginal improvements over a strong distilled baseline. Distill, *evaluate the distilled model against your target*, and only escalate if it falls short. ### Distilling traces the student cannot imitate If the teacher's traces are too long, too stylistically idiosyncratic, or use tokens the student's tokenizer handles poorly, the student will learn a degraded imitation. Curate the traces: trim, normalize, ensure the format is something the student can reproduce. Distillation is imitation; imitation requires the target to be imitable. --- ## Key Terms | Term | Definition | | --- | --- | | **CoT distillation** | SFT a smaller student on chain-of-thought reasoning traces generated by a stronger teacher model; no RL in the training loop | | **Teacher model** | The strong reasoning model (R1, o1, Claude/GPT with thinking) that generates the traces distilled into the student | | **`<think>` trace** | The explicit reasoning block a reasoning model emits before its answer; the distillation target | | **DeepSeek-R1-Distill-Qwen-32B** | The 32B model distilled from R1 via SFT on ~800K curated reasoning samples; beats o1-mini on several benchmarks — the headline distillation result | | **Rejection-sampling FT (RFT / RAFT)** | Iteratively SFT on only positively-rewarded candidates generated by the current model; generate → filter by reward → retrain on winners | | **Verifiable reward** | A reward that can be checked exactly (math correctness, code passing tests); required for RFT and GRPO | | **arXiv:2504.11343** | The RAFT paper; finding that simple rejection-sampling FT is surprisingly competitive with GRPO and PPO on math reasoning | | **Hybrid model (thinking toggle)** | A single model that can run in thinking or non-thinking mode per request (e.g., Qwen3), avoiding the always-on latency tax | | **Thinking budget** | A token/length budget for thinking, allowing adaptive compute — more on hard problems, less on easy ones | | **GRPO** | Group Relative Policy Optimization (FT14); full RL on verifiable rewards; most powerful, most expensive, most unstable of the three | --- ## Lab Exercise See `07-lab-spec.md`. The "Distill Reasoning" lab: generate CoT traces from a strong teacher on 200 GSM8K math problems, SFT a small student (MiniCPM5-1B or Qwen2.5-1.5B) on those traces, and measure the GSM8K lift versus the un-distilled base. Consumer-GPU. Stretch goal: an RFT variant that filters the student's own candidates. --- ## References 1. **DeepSeek-AI (2025)** — *DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning*. arXiv:2501.12948. The R1 paper; the distillation recipe and the R1-Distill model family (incl. R1-Distill-Qwen-32B) are described here. The distilled weights and the distillation data methodology are on the [DeepSeek-R1 GitHub](https://github.com/deepseek-ai/DeepSeek-R1). 2. **Pang et al. (2025)** — *Iterative Reasoning Preference Optimization via Rejection Sampling Fine-Tuning* (RAFT). arXiv:2504.11343. The finding that rejection-sampling FT is surprisingly competitive with GRPO and PPO on math reasoning; the empirical basis for RFT as a terminal technique. 3. **Yang et al. (2025)** — *Qwen3 Technical Report*. The hybrid thinking-toggle model; thinking-budget mechanisms for adaptive compute. 4. **RLHF Book** — *Reasoning chapter*. The broader framing of reasoning fine-tuning, RLHF, and the distillation-vs-RL tradeoff in the open ecosystem. 5. **FT14 — GRPO and Verifiable Rewards** (prerequisite). The full-RL counterpart to this module; GRPO is the third option in the decision tree. 6. **FT00 — The Steering Stack**. The thesis this module rests on: distillation is steering (routing the student through reasoning pathways it already has), not teaching new reasoning ability.