Company
5 problems tagged for xAI.
Build a gated MoE with top-k routing, load balancing loss, and expert capacity constraints for sparse computation.
Write the tiled, fused attention kernel that computes exact attention in O(n) memory using online softmax in Triton.
Build Fully Sharded Data Parallel: shard parameters across GPUs, all-gather before forward, reduce-scatter gradients after backward.
Distribute attention computation across GPUs in a ring topology, enabling context lengths that exceed single-GPU memory.
Write a GPU kernel in Triton that fuses the softmax computation into a single pass, eliminating intermediate memory reads.
