Grind Attention for ML/AI Interviews
Follow the guided LLM Learning Path to implement a model from scratch, browse curated Basics and Advanced lists, or set up the AI Tutor for interactive coaching. Many exercises are tagged with the real companies that ask them in interviews.
Solved 0 of 75
0% complete
Questions from real interviews at
Implement LLM from Scratch
Implement Byte Pair Encoding from Scratch
Build the BPE tokenizer algorithm that iteratively merges frequent character pairs to build a subword vocabulary.
Implement Sinusoidal Embeddings
Build the fixed sinusoidal positional encoding from 'Attention Is All You Need' using sin/cos at different frequencies.
Implement ROPE Embeddings
Build Rotary Position Embeddings that encode relative positions by rotating query and key vectors in complex space.
Implement RMS Norm
Build Root Mean Square Layer Normalization used in LLaMA and modern transformers — simpler and faster than LayerNorm.
Implement Attention from Scratch
Build scaled dot-product attention from raw matrix operations — queries, keys, values, scaling, and softmax.
Implement Multi-Head Attention from Scratch
Split attention into multiple heads with independent projections, compute attention per head, and concatenate results.
Implement Grouped Query Attention from Scratch
Build GQA where multiple query heads share key-value heads, reducing KV cache memory while preserving quality.
Implement KV Cache for Autoregressive Generation
Cache key and value tensors from previous timesteps so each new token only computes attention over one new position.
Implement Sliding Window Attention
Restrict attention to a fixed local window around each token, reducing memory from O(n²) to O(n·w) for long sequences.
Implement SmolLM from Scratch
Build a complete small language model end-to-end: tokenizer integration, transformer blocks, and autoregressive text generation.
Implement KL Divergence Loss
Compute KL divergence between two probability distributions from scratch, essential for VAEs and knowledge distillation.
Implement LoRA on a Linear Layer
Inject trainable low-rank decomposition matrices (A, B) into a frozen linear layer for parameter-efficient fine-tuning.
Apply SFT on SmolLM
Fine-tune a language model on instruction-following data using supervised fine-tuning with cross-entropy loss.
Implement DPO Loss from Scratch
Compute the Direct Preference Optimization loss that trains a policy directly from preference pairs without a reward model.
Implement PPO for RLHF
Build Proximal Policy Optimization with clipped surrogate objective, value function baseline, and KL penalty for RLHF.
Implement GRPO (DeepSeek-R1 Algorithm)
Build Group Relative Policy Optimization that scores multiple completions per prompt and uses group-relative advantages.
Implement Temperature Sampling
Divide logits by a temperature scalar before softmax to sharpen or flatten the token probability distribution.
Implement Top-k Sampling
Select the k highest-probability tokens, zero out the rest, renormalize, and sample for controlled text generation.
Implement Top-p (Nucleus) Sampling
Sort logits, compute cumulative probabilities, mask tokens below the nucleus threshold, and sample from the filtered distribution.
Implement Speculative Decoding
Draft tokens with a fast model, verify in parallel with the target model, and accept/reject to guarantee identical output distribution.
Implement Continuous Batching for LLM Inference
Dynamically slot sequences in and out of a running batch as they finish, maximizing throughput without padding waste.
Build a Complete LLM Inference Engine
Combine KV caching, continuous batching, and memory management into a production-grade inference server.
Implement Mixture of Experts Layer
Build a gated MoE with top-k routing, load balancing loss, and expert capacity constraints for sparse computation.