SmolLM3

SmolLM3 is the answer to "what does a 3B model that doesn't punt the hard problems look like?" Two choices set it apart from Llama 3.2 3B and Gemma 3 4B: every fourth layer drops positional encoding entirely (NoPE), and the training pipeline is published in unusual detail. Both choices touch lessons we already taught in Act 2.

The blocks worth examining are positional handling, attention, the MLP, and the tokenizer. Boring blocks (residual, RMSNorm, embedding lookup) are covered cold in build-a-block — the diagram links them out rather than re-explaining them here.

Positional handling.27 of SmolLM3's 36 layers apply RoPE rotation to query and key vectors. The remaining 9 — every fourth layer, counting from layer 4 — skip the rotation entirely. The NoPE lesson walks the intuition: layers seeing positionally-uninformed attention seem to generalise better past the trained context length.

Attention — GQA 16q/4kv. 16 query heads per layer collapse into 4 KV groups, a 4:1 compression ratio that matches Llama 3.2 3B. The From MHA to GQA lesson shows why this ratio sits at the sweet spot: KV cache footprint drops 4× with negligible quality cost. SmolLM3 inherits the recipe rather than experimenting.

MLP — SwiGLU, 2048 → 11008. The MLP uses SwiGLUwith hidden dimension 2048 and intermediate 11008. That's a 5.37× expansion — noticeably fatter than the canonical SwiGLU (8/3)·d ≈ 2.67× expansion used by Llama 3 at the same per-parameter budget.

Tokenizer.SmolLM3 reuses the Llama 3.2 tokenizer wholesale (128K vocab) — the BOS token aside, it's the same tokenizer. The tokenization lesson covers the BPE recipe; SmolLM3's choice here is deliberate reuse, not innovation.

Why dense, not MoE?

At 3B parameters, the routing overhead and training instability of mixture-of-experts architectures eat the parameter-efficiency gains. The MoE crossover with dense is well past 3B — DeepSeek-V3 starts at 671B total parameters, Qwen3 MoE starts at 30B total.

Why NoPE on every fourth layer?

Long-context generalisation. SmolLM3 cites Yang et al. 2025 ("Rope to Nope and Back Again"), which shows a hybrid RoPE/NoPE pattern outperforms RoPE-on-every-layer on long-context tasks. The NoPE lessonwalks the intuition: positionally-uninformed layers seem to extend the competence ceiling past the trained context length. SmolLM3's 3:1 ratio (RoPE on three, NoPE on one, repeat) is the production-scale design point.

Why over-train at 11.2T tokens?

The model sees 11.2T tokens against 3B parameters — a D/N ratio of ~3,733, far past Chinchilla-optimal's ~20. The Scaling laws lesson covers the inference-cost-aware reframing: Sardana 2024 shows quality keeps improving up to D/N ≈ 10,000 once inference cost is added to the optimisation objective.

Why a three-stage curriculum?

Capability annealing. The three-stages lesson uses SmolLM3 as the canonical example because HF publishes the data mix per stage in unusual detail: stage 1 is web-scale general data (85/12/3 web/code/math at 0→8T tokens), stage 2 lifts the math and code share (75/15/10 at 8→10T), stage 3 anneals on reasoning data (63/24/13 at 10→11.1T). Each stage locks in capabilities the next one builds on.

The 11.2T tokens split across three stages, each with a deliberate data-mix shift. Stage one is web-scale general data — broad fluency, multilingual coverage across six European languages, the long tail of human writing. Stage two raises the math and code proportion, pushing the model toward formal-reasoning capability without losing the general fluency stage one locked in. Stage three is the reasoning anneal: upsampled math and code, plus instruction and reasoning datasets like OpenMathReasoning.

The context-extension recipe is the other published-in- detail piece. SmolLM3 pretrains at 4K sequence length, then mid-trains an extension to 32K with rope_theta raised to 1.5M, then to 64K with rope_theta raised to 5M. At inference, YaRN extrapolates the 64K-trained model to 128K. The long-context lesson walks the YaRN math.

The whole pipeline is the canonical worked example for the three-stages lesson. Other 2025-era SLMs trained on similar token budgets but published far less detail about the per-stage breakdown — SmolLM3's transparency is what makes it the teaching target, not the architecture per se.