Implement Sliding Window Attention in PyTorch
Mediummodern-architecturesauto-graded
Restrict attention to a fixed local window around each token, reducing memory from O(n²) to O(n·w) for long sequences.
Solve it
Check your answer
The grader verifies properties of your implementation, so a correct solution written differently from ours still passes.
pip install torchleet
from torchleet import check
check("sliding-window-attention", create_sliding_window_mask, full_attention, sliding_window_attention)