Company
5 problems tagged for Perplexity.
Divide logits by a temperature scalar before softmax to sharpen or flatten the token probability distribution.
Cache key and value tensors from previous timesteps so each new token only computes attention over one new position.
Sort logits, compute cumulative probabilities, mask tokens below the nucleus threshold, and sample from the filtered distribution.
Dynamically slot sequences in and out of a running batch as they finish, maximizing throughput without padding waste.
Combine KV caching, continuous batching, and memory management into a production-grade inference server.