What do settings like temperature, top-p, and top-k actually do?
Reviewed by Jason Burns, Editorial Steward · Last updated
Temperature scales the model's next-token probabilities before sampling (higher = more random), top-k restricts sampling to the k most likely tokens, and top-p (nucleus sampling) restricts it to the smallest set of tokens whose cumulative probability exceeds p. As OpenAI, put it on the record: "What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic."