acceptodds
Under review as a conference paper at ICLR 2027

Same Probe, Different Numbers: Are Activation Probes Robust to Inference-Time Numerical Non-Determinism?

Abstract

Activation probes are increasingly used to monitor LLMs in deployment. A probe is typically trained once per model and dataset under one inference configuration, then used under whatever batch size and numerical precision the serving stack happens to use. Because common GPU kernels are not batch-invariant and floating-point formats round differently, the activations seen at deployment are not the ones the probe was trained on. We measure what that costs, for Llama-3.1-8B, Qwen3-8B and Gemma-3-4B, across batch sizes 4, 8 and 16 and float32, bfloat16 and float16, at four depths and four token positions. We train 768 probes on one configuration, evaluate each on every other, and compare the same probe's verdicts between two runs example by example. Our central finding is that probes are stable under these perturbations, but that aggregate accuracy is the wrong instrument for showing it: it understates how many individual verdicts change by a factor of two to nine, and that churn, once measured, proves symmetric and leaves the probe's ranking intact. At the prompt, probe accuracy never moves by more than 0.47 percentage points across transfers, and only 0.076% of individual verdicts change; under float32 with only the batch size varied, not one verdict changes in evaluations. During decoding the flip rate rises to 2.8%, but it separates cleanly: rows whose realised tokens matched flip in 0.12–0.15% of cases at every position, while rows whose tokens diverged flip in 12.9%. Reconstructing each run's greedy trajectory from stored logits shows why, and shows that the cause is the text rather than the arithmetic: a bfloat16 batch-size change flips the first generated token for 2.1% of rows and leaves the two runs on different tokens in 25% of rows by token 20, while float32 diverges nowhere. Flips are symmetric between classes, Cohen's stays above 0.94, and AUROC moves by at most 0.05 points, so what the perturbation disturbs is the handful of examples within rounding distance of the boundary. Underneath, the activations move about as much as the format's rounding: in bfloat16 a batch-size change perturbs them by a median relative of , roughly the same change in float16, matching float16's three extra mantissa bits, and for Gemma, where both runs share a GPU, more than the entire float32-to-bfloat16 conversion. Probes absorb this; the model's own next-token argmax does not. Robustness evaluations of activation monitors should therefore report per-example agreement rather than aggregate accuracy, separate representational noise from input change, and state the serving configuration alongside the result. We will release our extraction and analysis pipeline.

Primary area:
interpretability and explainable AI
Keywords:
Activation Probing, Numerical Non-Determinism, Representation Robustness, Large Language Models, AI Safety Monitoring, Batch Invariance, Precision Formats, Inference Serving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_VE4wOpenReview ↗
Rating
2(+1)
Confidence
3(+1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper focuses on a significant issue and an important area of probe research, seemingly involving a severe training-inference inconsistency problem. The authors' proposed method addresses this issue to some extent, offering a cross-precision training-inference solution for deploying models with probes.

Critical strengths
  1. The problem being solved is interesting and has practical significance.
  2. The experimental results showed a good improvement.
Critical weaknesses
  1. Fixed left-pad to 128 tokens, B ∈ {4, 8, 16}, HF + PyTorch eager, a single seed, a single revision; no vLLM/SGLang/TensorRT-LLM, no TP/PP, no ragged/continuous batching, no chunked prefill, no CUDA graphs/autotuning. Yet the paper's mechanism is precisely what these factors would change. It is a structural gap.
  2. The transfer matrix is one-directional: all probes are trained at B=4 and only tested at B=8/16; there is no symmetric validation of train at B=16, test at B=4. Although the noise is approximately symmetric, train once, deploy anywhere is currently validated at only one training point.
  3. The core object of the entire paper, probe, has no definition, no notation, and no image.
Questions for authors

please refer to "weaknesses"

2 comments

Thank you for your review, and our apologies for the slow response. We would first like to clarify the paper's scope, because the summary describes a "proposed method" offering a training–inference solution. The paper does not propose a method. It is a controlled measurement study. It asks whether a probe trained once, under one batch size and precision, gives the same verdicts when deployed under a different configuration, which is the "train once, deploy many times" pattern used for production probes (Cunningham et al., 2025; Kramár et al., 2026). Our contributions are the measurement and an evaluation recommendation: report per-example agreement rather than aggregate accuracy, and separate numerical noise from changes in generated text. We will make this clearer in the abstract and introduction.

W1: Fixed left-padding, B ∈ {4, 8, 16}, HF eager, single seed; no vLLM/SGLang/TensorRT-LLM, TP/PP, continuous batching, chunked prefill or CUDA graphs. The narrow setup was a deliberate control. Varying the batch size and precision in isolation, on a fixed stack, lets us attribute every change to those factors. A production engine changes many things at once and would confound that attribution. That said, continuous batching is itself a mechanism that varies the effective batch shape from request to request, so our batch-size axis is the controlled version of it. We agree an engine-level check would strengthen external validity, and we propose to add:

  • an experiment with vLLM or SGLang under continuous batching (with and without CUDA graphs, and chunked prefill where supported), comparing probe verdicts against our HF-extracted baselines;
  • multiple probe-training seeds, to give the noise floor already flagged as outstanding in App. C.

We will also expand the limitations section to list the serving features not covered.

W2: Transfer is one-directional (train B=4 only). Thank you; this is easy to address. Activations for every batch size are already stored, so we will train probes at B=8 and B=16 and evaluate them on all cells, including B=4, giving a full all-pairs transfer matrix. Our existing analysis suggests B=4 is not a special training point: the pairwise distances satisfy d(4,16) ≈ d(4,8) ≈ d(8,16) in all 224 triples (Sec. 4.1), so each batch size behaves as an independent draw of rounding noise. The symmetric experiment will verify this directly.

W3: "Probe" has no definition, notation or figure. Agreed. We will add a formal definition in Sec. 3: a classifier f_θ(h_{ℓ,t}(x)) on the residual-stream activation at layer ℓ and position t, with training-set standardisation. We will also add notation for the transfer quantities (Δ, flip rate P[ŷ_A ≠ ŷ_B]) and a schematic figure showing extraction under configuration A, probe training, and evaluation on activations from configuration B.

Presentation. Following your rating, we will restructure Sec. 4 and 5 around the three regimes, add a roadmap, and move secondary diagnostics to the appendix.

We hope these clarifications and additions address your concerns, and we would welcome any further suggestions.

Thanks for your clarification. It has partially addressed my concerns. The explanation regarding the use of a fixed inference environment to isolate variables has alleviated some of my concerns about the experimental design, and the formal definition of the probe has clarified the expression issue.

However, the results for the service engine validation, reverse batch-size migration, and multiple sub-experiments mentioned in the response have not yet been provided. The scope of the existing per-sample consistency analysis and the attribution during the decoding phase remain limited.

I will raise the overall score accordingly.

Reviewer_mAvhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines whether activation probes remain reliable when an LLM is deployed with a different batch size or numerical precision from the configuration used to train the probe. It studies three language models and several probe architectures, with the main probe experiments covering sentiment and spam classification.

The authors find that predictions are very stable when probes read the prompt. During generation, predictions change more often, particularly when numerical differences cause the model to generate different tokens. They also show that aggregate accuracy can hide considerably more changes in individual predictions. The main contribution is an empirical characterization of this behavior and a recommendation to evaluate agreement on individual examples while distinguishing numerical changes from changes in the generated text.

Critical strengths

The paper asks a practical question that matters as probes move into deployment. A probe is often trained under one setup and then attached to a serving system whose batching and precision may change. Knowing whether that mismatch matters is useful, even when the answer is largely reassuring. The results suggest that, in the settings studied, small numerical changes rarely disrupt prompt-based classification. This gives practitioners a concrete starting point for deciding which sources of instability deserve attention.

Critical weaknesses

The explanation of the decoding results is plausible, but the separation between numerical effects and text changes is incomplete. The paper’s strongest interpretation is that the probe remains robust and the changing text explains the changing verdicts. However, matching tokens at a few recorded positions does not establish that the entire preceding sequence was identical. A direct comparison that feeds exactly the same complete sequence through both configurations would provide a cleaner test. There is also a practical interpretation issue: a monitor may correctly change its judgment when generated content changes. Since the labels here describe the original input, the experiments do not establish whether the observed verdict changes represent appropriate responses to new content or failures of monitoring. That distinction matters for the claimed implications.

Questions for authors
  1. How should a practitioner interpret a changed verdict when the generated text also changes? Can the authors establish whether these changes track meaningful changes in the monitored property, rather than relying only on agreement and the original input labels?
1 comment

Thank you for the clear and constructive review, and our apologies for the slow reply. Please see the general response for the main issue. Below we address your specific points.

W1 / Q1: Is a changed verdict an appropriate response to new content, or a monitoring failure? We agree that agreement measured against the original input label cannot fully answer this, and we will say so explicitly in Sec. 5 and 6. Three pieces of evidence suggest these flips are mostly near-boundary reshuffling and not real changes in the monitored property:

  • Divergence only occurs at argmax near-ties, where the model rates both continuations as almost equally plausible. Prior work finds such alternatives are usually paraphrases (Kuhn et al., 2023; He & Thinking Machines Lab, 2025; Atil et al., 2024).
  • On diverged rows, probe accuracy against the input label is unchanged (gap centred on zero).
  • Flips are class-symmetric, and κ and AUROC barely move.

To answer the question directly, we propose two additions:

  1. Semantic labelling of continuations. For every diverged row pair, we will label the property of interest (sentiment or spam) on each run's continuation, and test semantic equivalence between the two continuations. We will use an NLI/LLM judge validated on a human-annotated subset. We will then report how often a verdict flip coincides with a property change, and how often the property changed without a flip.
  2. Identical-prefix replay. Feeding exactly the same complete sequence through both configurations separates arithmetic effects from text effects without relying on agreement at four logged positions.

Practical framing. In the "train once, deploy many times" setting we study, the practitioner's question is whether the serving configuration alone changes what the monitor reports. We will make clear that our results answer this for the prompt position, and that for generated-token monitors they bound how much the configuration can matter. We will also keep our recommendations: batch-invariant decoding, deterministic re-encoding, or measuring the monitor's own flip rate.

Presentation. We will streamline Sec. 4 and 5, move secondary diagnostics to the appendix, and add a short roadmap of the three regimes (benign / behaviourally consequential / validity failure) at the start of the results.

We would be grateful to know whether these additions would address your concern.

Comments on the paper

2 comments

We thank all three reviewers for their careful reading, and we apologise for how long this response has taken. We wanted to check each point against our stored data before replying.

What the experiments are designed to test. Production activation probes follow a "train once, deploy many times" pattern. A probe is trained offline on activations extracted under one configuration. It is then attached to serving stacks whose batch size, numerical precision and serving engine differ from training and keep changing (Cunningham et al., 2025; Kramár et al., 2026). The question we ask is narrow and practical: when the probe and its input are unchanged and only the serving numerics differ, does the probe's verdict change? Our grid (batch size × precision × model × layer × position) isolates exactly that variable. We do not propose a new probing method. Our contribution is an empirical characterisation and an evaluation protocol: measure per-example agreement, and separate numerical noise from changes in the input.

The main shared concern: when generated text diverges, is the probe wrong, or is it correctly tracking new content? We agree this is the most important open point, and we will tighten our claims around it. We also think the evidence, in the paper and in prior work, indicates that divergence caused by numerical non-determinism mostly changes the wording of a continuation rather than its relationship to the prompt:

  1. Divergence happens at near-ties. A rounding perturbation can only flip an argmax when the top two tokens are nearly equally likely (Sec. 4.2). By construction, the model itself rates both continuations as almost equally plausible given the prompt. Prior work finds that such alternatives are very often semantically equivalent. Kuhn et al. (ICLR 2023) show that a model's token-level uncertainty is frequently uncertainty between different phrasings of the same meaning, which is the basis of semantic entropy (Farquhar et al., Nature 2024). The canonical batch-invariance study illustrates this directly: in 1,000 temperature-0 completions of "Tell me about Richard Feynman", every run shared the same first 102 tokens. At token 103, 992 runs continued "Queens, New York" and 8 continued "New York City" (He & Thinking Machines Lab, 2025). The texts differ, but their meaning does not.
  2. Run-to-run variation is mostly in the surface form, not the answer. Atil et al. (2024/2025) separate raw-string agreement from parsed-answer agreement under nominally deterministic settings. They find answer agreement far above string agreement; for example, GPT-4o on navigation reaches 99.6% answer agreement against 46.0% string agreement over 10 runs.
  3. Our own data point the same way. (i) On rows whose tokens diverged, the decode-position probe is no less accurate against the original input's label than on matched rows. The gap is centred on zero at every position and never exceeds 0.58 pp in the mean (Sec. 5, App. G). So the diverged text stays as informative about the monitored property as the original continuation. (ii) About 87% of diverged rows keep the same verdict. (iii) The flips that do occur are symmetric between classes (6,671 vs 6,813 at token 20), κ ≥ 0.94, and AUROC moves by at most 0.05 pp. That is the signature of near-boundary examples being reshuffled, not of a systematic change in what the text says.

Overall, real label changes caused by divergence appear rare. One caveat we will state explicitly: for long chain-of-thought generations, early divergences can cascade into substantively different reasoning and different final answers (Yuan et al., 2025). Our 20-token setting does not cover that regime.

Changes we propose to make (details in the individual replies):

  • (a) Identical-prefix replay. Teacher-force each run's complete generated sequence through the other configuration. This separates arithmetic effects from text effects exactly, as Reviewers EuDV and mAvh request.
  • (b) Semantic labelling of diverged continuations. Use an NLI/LLM judge, spot-checked by humans, to test whether verdict flips track real changes in the monitored property.
  • (c) Reverse-direction and all-pairs transfer. Train at B=8 and B=16, test at B=4. This needs only stored activations.
  • (d) Per-configuration distributions, worst cases and cluster-robust CIs.
  • (e) Direct decision-margin measurements.
  • (f) A serving-engine experiment with vLLM/SGLang continuous batching.
  • (g) A formal definition and figure for "probe", plus softer wording on the deployment and mechanism claims.

Thank you for an exceptionally detailed and careful review, and we apologise for the delay in responding. We agree with essentially every point and propose concrete changes below. The general response covers the shared question of whether divergent text changes what the probe should say.

W1: The decoding analysis does not fully establish that changed text causes the extra disagreement. This is correct. "Same tokens" means agreement at the four logged positions, not an identical prefix. We will:

  • add an identical-prefix replay: teacher-force run A's full generated sequence through configuration B, and vice versa, and compare probe verdicts on exactly the same text. This gives the direct separation of arithmetic and text effects you suggest;
  • log complete token sequences in the re-run, so "same tokens" can be defined over the full prefix;
  • fix the Appendix C indexing ambiguity. You are right that the stored logit vector at position t selects token t+1. We will state the alignment between reconstructed tokens and logged activation positions exactly, re-verify it in code, and correct the text and any affected numbers. [AUTHORS: confirm the alignment used and whether any numbers change.]

We also note an argument we will add to the paper: argmax flips occur at near-ties, which prior work finds are predominantly between semantically equivalent continuations (Kuhn et al., 2023; He & Thinking Machines Lab, 2025; Atil et al., 2024). This is consistent with our finding that probe accuracy on diverged rows, against the input label, is unchanged.

W2: Deployment conclusions extend beyond the tasks and probes tested. Agreed. We will scope the claims explicitly to the tested classifiers. We will also:

  • extend the per-example agreement analysis (flip rate, κ, score-level metrics) to all layers and all training precisions, not only final-layer fp32-trained probes;
  • report low-FPR operating points prominently (TPR at 1% FPR currently moves by at most 0.34 pp);
  • add a safety-relevant probe task if time permits [AUTHORS: e.g., a deception or harmful-request dataset], while being clear that difficult examples under realistic serving loads remain future work.

W3: Pooling and statistical dependence. We agree the pooled tests treat correlated evaluations as independent. We will:

  • report per-configuration distributions and worst-case cells;
  • compute cluster-bootstrap CIs that resample examples (and probes or extraction runs) and redo the binomial and correlation tests under that dependence;
  • rephrase "at most 0.05 pp" as the observed maximum over tested cells, not a bound.

W4: Mechanistic interpretations stronger than their measurements. Agreed, and these are cheap to fix from stored activations. We will:

  • describe equal pairwise distances as consistent with independent rounding draws, not proof of it;
  • directly measure decision margins: distance to the probe boundary for flipped vs. stable examples, including nonlinear probes via the logit margin;
  • project perturbations onto the probe's sensitive directions (the LR weight vector, and input-gradients for MLPs) and compare them to margins;
  • remove the claim that lower later-position accuracy implies smaller margins, unless the margin measurements support it.

Thank you again; these suggestions will make the paper considerably sharper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.