Implement Sinusoidal Embeddings in PyTorch

Mediummodern-architecturesauto-graded

Build the fixed sinusoidal positional encoding from 'Attention Is All You Need' using sin/cos at different frequencies.

Solve it

Open in ColabQuestion notebookSolution

Check your answer

The grader verifies properties of your implementation, so a correct solution written differently from ours still passes.

pip install torchleet

from torchleet import check
check("sinusoidal-positional-embedding", SinusoidalPositionalEmbedding)