Implement FSDP from Scratch in PyTorch
Expertgpu-systemsauto-graded
Build Fully Sharded Data Parallel: shard parameters across GPUs, all-gather before forward, reduce-scatter gradients after backward.
Solve it
Check your answer
The grader verifies properties of your implementation, so a correct solution written differently from ours still passes.
pip install torchleet
from torchleet import check
check("fsdp", FakeDistributed, FSDPLinear)