- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Decoding accuracy is commonly taken as evidence that a model represents a property, yet it does not show whether a probe tracks the property or its presentation. To address this, the paper proposes semantic counterfactual validation, which adds two controls to the evaluation of a probe:
- Certified presentation changes with an equal-budget refit: the records of a symbolic theory are permuted or rewritten, and the oracle certifies that the closure is unchanged; the frozen probe is scored on the changed presentation, and a probe of the same family is refitted on it with the same budget, which tells whether a failure to transfer leaves the property decodable.
- The crossing of truth with presentation: twin theories are minimally edited so that every role symbol keeps its record positions and counts while the truth of the queried fact changes (the false twin); truth-preserving twins further separate what a probe tracks — the edit or the truth.
The authors apply the method to four models of the Qwen3 and Gemma-3 families on type-inheritance theories and on ProofWriter, and use steering and interchange interventions to ask whether the answer uses what the probes read. The main findings are:
- Decoding accuracy (IID evaluation) does not establish what a probe tracks;
- What a probe tracks follows its training contrast, and decodability does not establish use;
- In most models, probes port across presentations where the model answers competently in every order.
Critical strengths
- Rigorous and honest. The experiments were preregistered with pass or fail conditions, post-hoc analyses are labelled as such, and failed criteria are reported faithfully. The design is clear, and three lines of evidence — the model's answers, the probes, and interventions — are checked against each other.
- A novel way to construct counterfactual inputs. The oracle certificate guarantees that every entailed fact is preserved, which provides a reliable new way to build inputs that differ in presentation but not in meaning. The truth-preserving twins are cleverly designed: with record positions and counts held fixed, they separate the probe's response to the edit from its response to the truth.
Overall, the method is a valuable and carefully executed contribution, but the current evidence does not yet separate probe failures from model incompetence; I therefore rate it marginally below the threshold.
Critical weaknesses
1. Main weakness: a portability failure of the probe is confounded with the model's lack of competence
All experiments run without generated reasoning, and activations and answers are read in a single forward pass. In this setting, the model is not competent in the key cases of the type-inheritance theories, in particular with structural negatives: after a certified reordering of the records, the model's own answer gap falls to near chance, and some models miss the competence criterion even in canonical order. The interchange interventions further show that the representation the answer uses at the last token carries the edit of the links similarly whether or not it changes the truth (REWIRING-CODED). This indicates that the model itself relies on the canonical layout as a shortcut rather than computing a truth that holds across presentations.
Under these circumstances, it is almost inevitable that the probes fail to port — there may be no portable representation of truth in the model for a probe to find. The reordering results therefore diagnose the model's shortcut more than a probe that misses an existing representation, and they are insufficient to support the core claim that decoding accuracy does not establish what a probe tracks. The evidence that genuinely separates the two rests mainly on a single Gemma probe (the H2 rewriting), and the corresponding registered primary, which required this pattern in both models, was not met.
The authors acknowledge that a failed test does not show that a model lacks a representation of truth, and Section 3.4 applies the test where competence does not depend on the order. However, the only competent cells there are those whose false alternative is absent from the theory, where the presence of the queried name decides the answer; the authors themselves describe this as a presence detector. This panel is therefore not enough to remove the confound above.
2. The diagnostic value of the test itself is not validated
The paper lacks a positive control for the method itself: a setting in which a probe is known to track truth, showing that the test passes a probe that tracks the property and fails one that tracks the presentation. The failures all coincide with the model's lack of competence and are hard to attribute to the probe; the passes are either the presence cells or ProofWriter. On ProofWriter, the model's accuracy is limited while the probe's AUROC is clearly higher than the model's own performance, and only one presentation change (sentence permutation) is tested, without the crossing or twins. This pass may therefore also come from a shortcut that is invariant to the given change — as the authors note, a pass does not certify a semantic representation. Taken together, a failure does not show that the probe is at fault, and a pass does not show that it tracks the property; the diagnostic value of the test has not yet been demonstrated.
3. Questionable probe quality and capacity
- The training budget is small. The authors did train probes on worlds whose record order is independent of the label, and refitted probes on the permuted presentation, but the number of training cases is very small relative to the activation dimension, the learning curves show no saturation, and some fits stop at the solver's iteration limit without converging. Since the theories are generated automatically and more data costs little, the restriction behind "not decodable at this budget" is effectively self-imposed, and it leaves the central question — whether recoverability holds — unresolved.
- Limited capacity and sites. The paper uses only linear probe families, with activations averaged over three-block windows at fixed sites (the final period of a statement or the question-mark site). As the authors note, a probe trained at the answer site remains to be examined under the certified test. It therefore cannot be ruled out that stronger probes or other sites would port.
Questions for authors
I would be happy to raise my score if the authors resolve the main weakness, or address some of the following points:
- Validate in a non-trivial configuration the model solves in every order. Enable generated reasoning, or use a model stronger on this task, and first establish behaviorally that the model is competent with structural negatives both in canonical order and in random orders; only then test whether the probes port. Please also specify which site the probe is read from once reasoning is generated.
- Provide a positive control for the test itself. Construct a model known to hold an order-independent representation of truth (e.g., a model fine-tuned on reachability with records in random orders, or a small transformer with a known mechanism), and show that a probe tracking truth passes the certified test while a probe trained on the positional cue fails, establishing the sensitivity and specificity of the test.
- Increase the training budget and capacity of the probes. Scale the label-independent-order training by one to two orders of magnitude and report whether the learning curves saturate; add non-linear probes and fits that converge; and train a probe at the answer site under the same certified test. Only if truth remains undecodable at a sufficient budget can "not decodable at this budget" be strengthened.
- Extend the crossing beyond the type-inheritance theories. Add twins and the crossing on ProofWriter to rule out a permutation-invariant shortcut, and discuss how certificates could be approximated on natural-language truth datasets without an oracle (e.g., rewrites verified by NLI or a solver). I would also ask the authors to state explicitly which conclusions are confirmatory and which are exploratory.
- (Optional) Characterize when probes port. In more configurations where the model is competent across presentations, characterize systematically when probes port and when they do not, and how this relates to the model's competence.
Other Suggestions
- Please reduce the custom terminology and the capitalized registered readings, or add a summary table in the main text listing, for each experiment, the preregistered criterion, its reading, and what it implies, so that readers can follow the main line.
- Appendix C (logically equivalent theories differ in how accessible their facts are) is interesting but only loosely connected to the main line; consider condensing it or stating its connection more explicitly.
2 comments
Thank you for the detailed and constructive response. I appreciate the additional ProofWriter analyses, the clarification of the refit results, and your willingness to state the limitations more explicitly. These additions improve my understanding of the paper.
However, my main concern remains unresolved. On the central reordering tasks, none of the tested models is competent with structural negatives under random ordering, even at one hop, and the reasoning-mode experiment was inconclusive. Thus, the requested setting—in which the model is competent both before and after reordering—is still missing. Canonical-order competence does not fully resolve the concern that probe failure may partly reflect the model’s inability to perform the task.
The H2 refit is informative, but recoverability is not established for the central reordering case. Concerns about the probe training budget, convergence, capacity, and extraction locations also remain. Besides, the collapse of the probes under reordering provides indirect support for my concern that the probes' failure may actually reflect the model’s inability to solve the task. The ProofWriter results provide useful supporting evidence, but they do not replace a positive control on the main task or establish when probes should be expected to transfer.
I recognize the care and effort behind this work, but I regret that the current evidence is insufficient to address my principal reservations. Therefore, I do not intend to raise my score at this time.
Thank you for your review and your appreciation of the design. We answer the main weakness first then the three questions.
The paper claims that IID decoding accuracy does not establish what a probe tracks. The evidence is a set of probes with IID AUROC 0.91 to 1.000 whose ranking does not hold under certified changes that preserve the closure, the record set, the derivation length and the token count (the reordering and the twins). Whether this happens because the probe missed a portable representation or because the model has none is a question of attribution; in both cases the IID accuracy certified a layout correlate. We agree with your description of the mechanism: the twins and the interchange interventions show that the model's own answer tracks the wiring of the records in the original order, and we name it controlled shortcut learning (in Discussion). We do not claim that the probes miss an existing representation of truth; the paper states that a failed test does not show that the model lacks one, and the revision makes the attribution explicit.
The refit is what separates the two outcomes. On H2 the property stays linearly decodable in the new presentation (0.999) and the frozen probe fails; under reordering, recoverability is not established and the answer collapses as well. The revision states which inference each outcome licenses.
Section 3.4 is the design you ask for: competence is established behaviorally in canonical order and in two random orders before portability is tested, on 384 new worlds and four models. Its result is that, without generated reasoning, no model is competent with structural negatives in random order at any depth, including one hop, and that the probes trained in canonical order fall to chance there in every model.
The competent cells are the presence questions; the probes port there, and steering along them is not specific. We read this as the paper's third finding: a pass shows invariance to the tested change and not the use of a semantic representation. We agree that this leaves the test without a non-trivial positive control in these models, and we explicit it in (upcoming) updated limitations.
The reordering results do not start from a model known to be incompetent: in canonical order the answer gap separates true from false worlds at AUROC 0.91 in Qwen3-32B. That this separation is itself bound to the layout is a result of applying the certified changes to the answer, not an independent prior; "there may be no portable representation to find" is a conclusion our method delivers. We explicit it now more clearly in the text.
Two controls of the test's mechanics are in the paper. What is missing is a positive control with a known order-independent mechanism for reachability; constructing one (a model trained on reachability in random orders, or a small transformer with a known circuit) is the right next step and cannot be done within the discussion period. Before submission we ran Qwen3-32B in reasoning mode on the fresh-world forced-choice panel; the traces were truncated by the generation budget and the run was inconclusive, so the paper reports the single-forward-pass setting only.
On ProofWriter you are right that the probe reads more than the answer shows, and literal accuracy understates the comparison, so we computed the AUROC of the model's own answer gap on the same rows from the recorded logits. On the 768 entailed rows it is 0.80 [0.75, 0.85] in canonical order and 0.81 [0.77, 0.86] after permutation for Qwen3-32B, and 0.71 [0.65, 0.76] and 0.75 [0.70, 0.79] for Gemma-3-27B-it, against 0.975/0.971 and 0.891/0.902 for the probe. The answer's separation falls with depth (Qwen3-32B: 0.91, 0.72 and 0.73 at depths 1 to 3 in canonical order) while the probe holds. So the probe separates entailed from non-entailed questions better than the answer does, in both orders: on ProofWriter the pass is a pass of portability, and the gap between what is decodable and what the answer uses is the distinction operated in the paper. Whether the probe's extra separation is an unused representation of entailment or a permutation-invariant correlate is what twins on ProofWriter would decide, it is now stated as a next step.
We can answer other elements in an additional comment, but we do not want to exceed the limit. We hope we are clear enough in this comment and it will be constructive.