- Rating
- 2(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper focuses on a significant issue and an important area of probe research, seemingly involving a severe training-inference inconsistency problem. The authors' proposed method addresses this issue to some extent, offering a cross-precision training-inference solution for deploying models with probes.
Critical strengths
- The problem being solved is interesting and has practical significance.
- The experimental results showed a good improvement.
Critical weaknesses
- Fixed left-pad to 128 tokens, B ∈ {4, 8, 16}, HF + PyTorch eager, a single seed, a single revision; no vLLM/SGLang/TensorRT-LLM, no TP/PP, no ragged/continuous batching, no chunked prefill, no CUDA graphs/autotuning. Yet the paper's mechanism is precisely what these factors would change. It is a structural gap.
- The transfer matrix is one-directional: all probes are trained at B=4 and only tested at B=8/16; there is no symmetric validation of train at B=16, test at B=4. Although the noise is approximately symmetric, train once, deploy anywhere is currently validated at only one training point.
- The core object of the entire paper, probe, has no definition, no notation, and no image.
Questions for authors
please refer to "weaknesses"
2 comments
Thanks for your clarification. It has partially addressed my concerns. The explanation regarding the use of a fixed inference environment to isolate variables has alleviated some of my concerns about the experimental design, and the formal definition of the probe has clarified the expression issue.
However, the results for the service engine validation, reverse batch-size migration, and multiple sub-experiments mentioned in the response have not yet been provided. The scope of the existing per-sample consistency analysis and the attribution during the decoding phase remain limited.
I will raise the overall score accordingly.
Thank you for your review, and our apologies for the slow response. We would first like to clarify the paper's scope, because the summary describes a "proposed method" offering a training–inference solution. The paper does not propose a method. It is a controlled measurement study. It asks whether a probe trained once, under one batch size and precision, gives the same verdicts when deployed under a different configuration, which is the "train once, deploy many times" pattern used for production probes (Cunningham et al., 2025; Kramár et al., 2026). Our contributions are the measurement and an evaluation recommendation: report per-example agreement rather than aggregate accuracy, and separate numerical noise from changes in generated text. We will make this clearer in the abstract and introduction.
W1: Fixed left-padding, B ∈ {4, 8, 16}, HF eager, single seed; no vLLM/SGLang/TensorRT-LLM, TP/PP, continuous batching, chunked prefill or CUDA graphs. The narrow setup was a deliberate control. Varying the batch size and precision in isolation, on a fixed stack, lets us attribute every change to those factors. A production engine changes many things at once and would confound that attribution. That said, continuous batching is itself a mechanism that varies the effective batch shape from request to request, so our batch-size axis is the controlled version of it. We agree an engine-level check would strengthen external validity, and we propose to add:
We will also expand the limitations section to list the serving features not covered.
W2: Transfer is one-directional (train B=4 only). Thank you; this is easy to address. Activations for every batch size are already stored, so we will train probes at B=8 and B=16 and evaluate them on all cells, including B=4, giving a full all-pairs transfer matrix. Our existing analysis suggests B=4 is not a special training point: the pairwise distances satisfy d(4,16) ≈ d(4,8) ≈ d(8,16) in all 224 triples (Sec. 4.1), so each batch size behaves as an independent draw of rounding noise. The symmetric experiment will verify this directly.
W3: "Probe" has no definition, notation or figure. Agreed. We will add a formal definition in Sec. 3: a classifier f_θ(h_{ℓ,t}(x)) on the residual-stream activation at layer ℓ and position t, with training-set standardisation. We will also add notation for the transfer quantities (Δ, flip rate P[ŷ_A ≠ ŷ_B]) and a schematic figure showing extraction under configuration A, probe training, and evaluation on activations from configuration B.
Presentation. Following your rating, we will restructure Sec. 4 and 5 around the three regimes, add a roadmap, and move secondary diagnostics to the appendix.
We hope these clarifications and additions address your concerns, and we would welcome any further suggestions.