Determinism in Transformer-Based Language Models

August 2, 2026

Determinism in Transformer-Based Language Models

Diving deeper into the commonly repeated claim that LLMs are nondeterministic, especially after implementing one while following Sebastian Raschka's Build a Large Language Model (From Scratch), made me wonder about something rather simple.

Why can selecting the highest scoring token at every step, via a temperature of zero, still produce different output?

Thankfully, Yuan et al. published a paper last year that separated several numerical sources of nondeterminism. Horace He, in collaboration with others at Thinking Machines Lab, then approached the same problem from the inference serving side. I have linked both sources below. I still wanted to write this post because their underlying ideas are useful far beyond this one problem, and I wanted a shorter explanation that another engineer could quickly grok.

I also find that writing forces me to question the assumptions I am putting on the page. Some assumptions sound completely reasonable in one's head until you try to explain them to somebody else.

Anyway, back to determinism.

Why is temperature zero not enough?

A language model does not directly return a sentence. At each step, it assigns a score, called a logit, to every token in its vocabulary. When sampling is enabled, the decoder turns those scores into a probability distribution and draws the next token. Different draws are part of the design.

At temperature zero, inference implementations normally use greedy decoding instead. The decoder selects the token with the highest logit, adds it to the input and repeats. Given the same mathematical function and the same input, that sounds deterministic. So where can another answer possibly come from?

The catch is hidden inside the phrase "same input."

To a user, the input is the prompt. To an inference server, the input also includes the model weights, numerical format, GPU type, GPU count, software versions, batch shape and the other requests being processed at that moment. Two identical prompts can therefore travel through different computational paths.

This gives us two kinds of variation. Sampling variation is intentional and a fixed seed helps control it. Numerical variation can remain even after greedy decoding removes that intentional randomness.

Floating point numbers are approximations

Transformer computations use floating point numbers. BF16, FP16 and FP32 each have a finite number of bit patterns, so none can represent every real number exactly. Values must sometimes be rounded.

That rounding means floating point addition is not always associative:

Here is a deliberately extreme example:

In the first calculation, is too small to preserve beside . It is rounded away before the subtraction happens. In the second calculation, the large values cancel first, so the survives.

This is a toy example. FP16 cannot even represent . The exact point where information disappears depends on the format, but the principle stays the same.

Transformers perform reductions constantly. Matrix multiplication, attention and RMS normalization all combine many values into fewer values. If a kernel changes the order of those operations, its rounded result can change slightly.

Usually that difference is harmless. It becomes interesting when two next tokens have nearly equal logits. A tiny numerical change can swap their ranking, which makes greedy decoding select a different token.

That token is then fed back into the model. The next prediction now has different input, which changes the prediction after that, and so on. One tiny difference can grow into another paragraph or an entirely different reasoning trace.

Suppose we ask, "What is Isaac Newton known for?" After the shared prefix "Isaac Newton is known for his contributions to," two nearly tied tokens may compete. If physics ranks first, the answer may discuss his laws of motion and universal gravitation. If rounding puts mathematics first, it may discuss his method of fluxions, an early form of calculus, and his work on infinite series. Both are accurate, but the first changed token becomes part of every later prediction.

The GPU is not simply being random

It is tempting to blame thousands of GPU threads finishing in a random order. That can cause nondeterminism in some kernels, but it is not the full explanation here.

Thinking Machines Lab points out that a typical language model forward pass can be deterministic between repeated runs. Given the same tensors and kernel configuration, common matrix multiplication kernels can return the same bit for bit result every time.

The larger serving system can still appear nondeterministic because many fast kernels are not batch invariant.

Batch invariance means that one request gets the same result regardless of how many other requests sit beside it, where it appears in the batch or how the server groups the batch.

Inference servers dynamically batch requests to keep GPUs busy. Your prompt might run alone when the server is quiet and alongside many prompts when it is busy. Those batch shapes can select different reduction strategies or kernel configurations. Each path can be deterministic by itself while producing slightly different rounded values.

From the user's perspective, server load is unpredictable. It becomes a hidden input.

Now send the same Newton prompt to an inference server. It may run alone in a quiet batch that ranks physics first. In a busy batch with unrelated weather and code review requests, another deterministic reduction strategy may instead rank mathematics first. Those requests add no facts about Newton. They only change the batch shape and potentially the computation beneath his answer. This is a simplified illustration, not a model measurement.

Put simply, server load changes the batch, the batch can change the numerical path and the numerical path can change the selected token. I think this is the most important idea in the article. The GPU does not need to roll an invisible pair of dice. We may simply be changing an input that the user never knew was an input.

Does this actually matter?

For a chatbot producing two reasonable phrasings, probably not. For benchmarks, regression tests, safety evaluations and reinforcement learning rollouts, it absolutely can.

Yuan et al. evaluated models across 12 configurations spanning batch sizes, GPU counts and GPU types. Under BF16, DeepSeek-R1-Distill-Qwen-7B showed a 9.15 percentage point standard deviation in AIME'24 accuracy. For some reasoning model settings, the average per problem standard deviation in output length exceeded 9,000 generated tokens. FP32 reduced this variation to zero or near zero in many of their experiments.

Thinking Machines Lab found the same issue from the serving side. They generated 1,000 responses with thinking disabled, temperature zero and 1,000 tokens each from Qwen3-235B-A22B-Instruct-2507. The first 102 tokens were identical. The responses first diverged at token 103 and eventually produced 80 unique completions. With batch invariant kernels, all 1,000 completions were identical.

That is hard to dismiss as the model merely being probabilistic. The variation came from the machinery running the model.

Can we make inference deterministic?

We can make the boundary much stronger and more explicit.

The first approach is higher precision computation. More precision retains more information and makes close token rankings less sensitive to rounding. Yuan et al. propose LayerCast, which stores linear layer weights and biases in BF16, then upcasts each weight to FP32 just in time for computation. Their experiments found reproducibility comparable to full FP32 while reporting 34% less memory usage.

The second approach is batch invariant kernels. Instead of only shrinking rounding errors, these kernels keep the numerical path for one request consistent as the surrounding batch changes.

vLLM currently exposes a beta mode:

VLLM_BATCH_INVARIANT=1 vllm serve <model>

The current documentation says this mode uses deterministic kernels, keeps numerical behavior consistent across batch sizes and disables optimizations that may introduce variation. It currently requires an NVIDIA H100, H200, B100 or B200 GPU with compute capability 9.0 or higher. Model coverage is still expanding and enabling it may reduce performance.

For sampling at a nonzero temperature, a fixed seed is still required. Batch invariance prevents unrelated requests from silently changing the computation underneath that seeded process.

Conclusion

Transformer based language models are not doomed to be nondeterministic.

The mathematical model can be deterministic under greedy decoding. The deployed system may not be because floating point arithmetic, precision, kernel selection and dynamic batching all sit between the mathematics and the generated text.

So I no longer think "Is this model deterministic?" is the useful question.

The useful question is: under which hardware, software, precision, batching and sampling conditions is it deterministic?

Once that boundary is defined, determinism stops being a vague promise and becomes an engineering property we can measure, test and increasingly enforce.

Sources: