Implement SmolLM from Scratch in PyTorch

Hardmodern-architecturesauto-graded

Build a complete small language model end-to-end: tokenizer integration, transformer blocks, and autoregressive text generation.

Solve it

Open in ColabQuestion notebookSolution

Check your answer

The grader verifies properties of your implementation, so a correct solution written differently from ours still passes.

pip install torchleet

from torchleet import check
check("smollm", rotate_half, apply_rotary_pos_emb, repeat_kv, RotaryEmbedder, RopeAttention)