Key takeaways
- Thinking Machines Lab sampled 1,000 completions at temperature 0 from Qwen3-235B-A22B-Instruct-2507 and got 80 distinct outputs, the most common appearing 78 times. Greedy decoding is not determinism.
- The divergence was traced to the batch-size dependence of reduction kernels, not to floating-point ordering alone. Your prompt's output can change because someone else's request joined the same batch.
- Determinism is purchasable and it costs throughput: 26 seconds became 55 with naive batch-invariant kernels and 42 with a better attention kernel, for 1,000 sequences on Qwen3-8B.
- A pinned model name is not a pinned model. Anthropic states that Claude 4.7 and later use a tokenizer producing roughly 30% more tokens for identical text, so the same prompt is a different request across a version boundary.
- Our rule: record 6 identifiers per run, and treat any comparison across a run that is missing one of them as an observation rather than a result.
A team investigating a failed agent run re-runs it at temperature 0 and gets a different trace. The usual conclusion is that something in the harness is wrong. Often nothing is.
Thinking Machines Lab sampled 1,000 completions at temperature 0 from a single model and a single prompt and observed 80 distinct outputs. The most common appeared 78 times out of 1,000. Divergence first showed up at token 103.
What does temperature zero actually guarantee?
That the sampler picks the highest-probability token. It guarantees nothing about which token that is.
The Thinking Machines write-up, published 10 September 2025, traces the variation to something more specific than floating-point ordering. Kernels used in inference, including matrix multiplication, RMSNorm and attention, are not batch-invariant: their numerical output for a given sequence depends on the total batch size of the operation. A reduction tree shaped for a batch of 8 sums in a different order than one shaped for a batch of 64, and the results differ in the last bits.
The operational consequence is worth stating plainly. The batch your request lands in is decided by the serving stack according to what other traffic arrived at the same moment. Your output can therefore change because of somebody else’s request. Nothing in your code, your prompt or your parameters moved.
That single fact rules out an entire debugging instinct. A trace that differs from yesterday’s is not evidence that a config changed.
Where the divergence showed up
In the published experiment the first 102 tokens were identical across all 1,000 samples. At token 103, 992 completions continued with one phrasing and 8 with another.
A long common prefix followed by a fork is the characteristic shape here, and it is also the worst possible shape for agent debugging. The run looks identical right up to the point where it stops being identical, and in an agent that fork can be a different tool with different arguments.
What has to be pinned before two runs can be compared?
Six things, and most pipelines record two of them.
| Identifier | What changes if it moves | Can you pin it? | Usually recorded |
|---|---|---|---|
| Model version, exact | Everything, including tokenisation | Yes | Yes |
| Sampling parameters | The distribution sampled from | Yes | Yes |
| Prompt version | The request itself | Yes | Rarely |
| Tool schema version | What the model may call, and the token overhead | Yes | Rarely |
| Retrieval snapshot | The facts in context | Yes, with effort | Rarely |
| Serving batch composition | The last bits of every reduction | Not on a hosted API | Never |
The last row is the floor. On a hosted API you do not control batching, so bitwise reproducibility is not available at any price, and a comparison between two runs always carries that residue. The rows above it are all within reach and are the ones commonly skipped.
The third and fourth rows deserve particular attention because they move most often. A prompt edit and a tool schema edit both change the request without changing anything a naive log would record, and a tool schema edit additionally changes the input token count, which makes the two runs different in cost as well as in behaviour.
A pinned model name is not a pinned model
Version identifiers hide real changes. Anthropic’s pricing documentation states that Claude 4.7 and later models use a newer tokenizer producing approximately 30% more tokens for the same text.
The same prompt string is therefore a different number of tokens across that boundary, with different cache behaviour and a different position for everything downstream of it. An eval suite carried across the boundary is measuring a different input than it was before, which is one of the ways an eval suite goes stale without anyone editing it.
What does determinism cost when you can buy it?
Throughput, and a known amount of it.
| Configuration | Seconds for 1,000 sequences | Relative to default | Bitwise reproducible |
|---|---|---|---|
| vLLM default | 26 | 1.00x | No |
| Batch-invariant, improved attention kernel | 42 | 1.62x | Yes |
| Batch-invariant, unoptimised | 55 | 2.12x | Yes |
Those figures apply to a stack you run yourself. They are worth knowing anyway, because they set the price of the only configuration in which “re-run it and see” is a valid debugging technique, and because the gap between 55 and 42 seconds shows how much of the cost was implementation rather than principle.
For most fleets the honest position is that determinism is unavailable and the correct response is statistical. That is the argument behind τ-bench’s pass^k, which asks what share of tasks succeed on all k identical attempts rather than on at least one. In the retail domain pass^8 falls below 25%. A single green run proves very little about a system whose repeat rate looks like that, which is the same reason a single judged score needs its variance reported alongside it.
What should a run record about itself?
The six identifiers, attached to the trace rather than kept in a deployment ticket.
The OpenTelemetry GenAI conventions already provide somewhere to put most of this. Agent name, agent version, tool name and conversation identifier are defined attributes, which means the same span that carries token counts can carry the version identifiers and a comparison becomes a query rather than an archaeology project.
Three rules make that recording worth having:
- Record the prompt by content hash, not by a label. A label called
v3changes meaning when somebody editsv3. A hash does not. - Record the tool schema the same way. It is an input to the model and it is charged as input tokens, so it belongs with the prompt rather than with the code.
- State the floor in the report. Any comparison on a hosted API carries irreducible variance from batching. A report that does not say so is claiming a precision it does not have.
The question for the next investigation is not whether the run reproduced. Ask: of the six identifiers, how many were the same between the run that failed and the run we compared it against? If the answer is fewer than five, the comparison has not shown what it appears to show.
Frequently asked questions
Is temperature 0 deterministic in LLM inference?
No. Thinking Machines Lab sampled 1,000 completions at temperature 0 from one model and one prompt and observed 80 distinct outputs, with the first divergence at token 103. Greedy decoding picks the top token, but which token is top can change.
Why do identical prompts return different answers?
Because kernels used in inference are not batch-invariant: matmul, RMSNorm and attention can produce different numerical results depending on the total batch size of the operation. The batch your request lands in depends on other traffic.
Can you make LLM inference reproducible?
On a stack you control, yes. Batch-invariant kernels produced bitwise identical completions across repeated runs in the Thinking Machines experiments, at a throughput cost. On a hosted API you do not control batching, so you cannot.
What should be recorded to make an agent run reproducible?
The model version, the prompt version, the tool schema version, the retrieval snapshot, the sampling parameters and the orchestrator version. Missing any one of them makes a later comparison between two runs uninterpretable.
Method and sources
- Defeating Nondeterminism in LLM Inference, Horace He and colleagues, Thinking Machines Lab, published 10 September 2025. Source of the 1,000 completions at temperature 0 from Qwen3-235B-A22B-Instruct-2507, the 80 distinct outputs, the most common appearing 78 times, the first divergence at token 103, the batch-invariance diagnosis, and the 26, 55 and 42 second figures for 1,000 sequences on Qwen3-8B.
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, Yao, Shinn, Razavi and Narasimhan, arXiv:2406.12045. Source of pass^k as published and of the finding that pass^8 falls below 25% in the retail domain.
- Anthropic, Claude pricing documentation, read 18 September 2026, for the statement that Claude 4.7 and later use a tokenizer producing approximately 30% more tokens for the same text.
- OpenTelemetry Semantic Conventions for Generative AI, consulted for the attribute names available to carry run identifiers on a span. The conventions moved to a dedicated repository and remain in development.
- The six-identifier list and the recording policy are our editorial judgement. We report no reproducibility rates of our own, because we have not measured them.