The dial that controls how adventurous a model's next word is. At zero it always picks the likeliest token and repeats itself; higher, it samples from the tail and gets creative, then incoherent. The name comes from physics, where hotter systems are more disordered. It is the first setting a new API user touches and the first one they set back.
Mechanically: the model produces a score for every possible next token; temperature divides those scores before they become probabilities. Low temperature sharpens the peak, high temperature flattens it. Nothing about the model's knowledge changes. Only its willingness to gamble does.
Sighted in an internal company memo, 2023, instructing staff to 'set the temperature to zero for legal use'. The record notes that temperature zero produces a confident, repeatable answer, and that repeatable is not the same as right. The memo did not distinguish.
The first thing everyone did with the GPT-2 demo in 2019 was crank temperature to the top and watch it invent words. At 1.5 it wrote like a poet having a stroke. We printed some of it out. Somebody still has the printouts.
A common belief is that temperature zero makes a model deterministic. Nearly. Batching and floating-point arithmetic on GPUs mean the same prompt can still produce different output. Several engineers have lost afternoons to this, and the record has their forum posts.
What does it mean? Write it the way you would say it out loud.
