acceptodds
Under review as a conference paper at ICLR 2027

When Does a Probe Identify a Representation? Semantic Counterfactual Validation in Language Models

Abstract

Decoding accuracy does not show what a probe relies on; for a given probe, no existing test shows whether it tracks the property or its presentation. We propose semantic counterfactual validation, which adds two controls to the evaluation of a probe. The first modifies the presentation of symbolic theories under an exact entailment check, which guarantees that every entailed fact is preserved; a probe of the same type, refitted on the changed presentation with the same training budget, shows whether a failure to transfer leaves the property decodable. The second crosses truth with presentation for a fixed question, using minimally edited twin theories that keep every entity and target concept at the same record positions and counts. All experiments were preregistered with pass or fail conditions except those labelled post-hoc. In type-inheritance theories, a Gemma-3-27B-it probe with AUROC 1.000 falls to 0.892 on a certified rewriting while a refit reaches 0.999, and under reordering a Qwen3-32B probe falls from 0.91 to 0.61 with refits recovering only the original order. On ProofWriter, whose questions need multi-hop reasoning, reordering causes no detectable loss. Twin theories that keep the queried fact true show, in four models, that probes and answers respond to how the records are linked in the original order; a probe trained to separate truth from these edits still fails the certified test. Interchange interventions show that the answer's representation at the last token carries the edit similarly (the two recovery curves differ by at most 0.06) whether or not it changes the truth, and that the statement-site probe direction carries none of this effect. In new theories rendered in random orders, the model answers correctly in every order only when the false alternative is absent from the theory, and there the probes keep their ranking across orders. Decoding accuracy therefore does not establish what a probe tracks, and succeeding the proposed certified test does not establish that the model uses the probe.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, linear probing, truth representations, semantic counterfactuals, interchange interventions, identification criteria, out-of-distribution generalisation, formal logic, synthetic rule worlds.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9qozOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Decoding accuracy is commonly taken as evidence that a model represents a property, yet it does not show whether a probe tracks the property or its presentation. To address this, the paper proposes semantic counterfactual validation, which adds two controls to the evaluation of a probe:

  1. Certified presentation changes with an equal-budget refit: the records of a symbolic theory are permuted or rewritten, and the oracle certifies that the closure is unchanged; the frozen probe is scored on the changed presentation, and a probe of the same family is refitted on it with the same budget, which tells whether a failure to transfer leaves the property decodable.
  2. The crossing of truth with presentation: twin theories are minimally edited so that every role symbol keeps its record positions and counts while the truth of the queried fact changes (the false twin); truth-preserving twins further separate what a probe tracks — the edit or the truth.

The authors apply the method to four models of the Qwen3 and Gemma-3 families on type-inheritance theories and on ProofWriter, and use steering and interchange interventions to ask whether the answer uses what the probes read. The main findings are:

  1. Decoding accuracy (IID evaluation) does not establish what a probe tracks;
  2. What a probe tracks follows its training contrast, and decodability does not establish use;
  3. In most models, probes port across presentations where the model answers competently in every order.
Critical strengths
  1. Rigorous and honest. The experiments were preregistered with pass or fail conditions, post-hoc analyses are labelled as such, and failed criteria are reported faithfully. The design is clear, and three lines of evidence — the model's answers, the probes, and interventions — are checked against each other.
  2. A novel way to construct counterfactual inputs. The oracle certificate guarantees that every entailed fact is preserved, which provides a reliable new way to build inputs that differ in presentation but not in meaning. The truth-preserving twins are cleverly designed: with record positions and counts held fixed, they separate the probe's response to the edit from its response to the truth.

Overall, the method is a valuable and carefully executed contribution, but the current evidence does not yet separate probe failures from model incompetence; I therefore rate it marginally below the threshold.

Critical weaknesses

1. Main weakness: a portability failure of the probe is confounded with the model's lack of competence

All experiments run without generated reasoning, and activations and answers are read in a single forward pass. In this setting, the model is not competent in the key cases of the type-inheritance theories, in particular with structural negatives: after a certified reordering of the records, the model's own answer gap falls to near chance, and some models miss the competence criterion even in canonical order. The interchange interventions further show that the representation the answer uses at the last token carries the edit of the links similarly whether or not it changes the truth (REWIRING-CODED). This indicates that the model itself relies on the canonical layout as a shortcut rather than computing a truth that holds across presentations.

Under these circumstances, it is almost inevitable that the probes fail to port — there may be no portable representation of truth in the model for a probe to find. The reordering results therefore diagnose the model's shortcut more than a probe that misses an existing representation, and they are insufficient to support the core claim that decoding accuracy does not establish what a probe tracks. The evidence that genuinely separates the two rests mainly on a single Gemma probe (the H2 rewriting), and the corresponding registered primary, which required this pattern in both models, was not met.

The authors acknowledge that a failed test does not show that a model lacks a representation of truth, and Section 3.4 applies the test where competence does not depend on the order. However, the only competent cells there are those whose false alternative is absent from the theory, where the presence of the queried name decides the answer; the authors themselves describe this as a presence detector. This panel is therefore not enough to remove the confound above.

2. The diagnostic value of the test itself is not validated

The paper lacks a positive control for the method itself: a setting in which a probe is known to track truth, showing that the test passes a probe that tracks the property and fails one that tracks the presentation. The failures all coincide with the model's lack of competence and are hard to attribute to the probe; the passes are either the presence cells or ProofWriter. On ProofWriter, the model's accuracy is limited while the probe's AUROC is clearly higher than the model's own performance, and only one presentation change (sentence permutation) is tested, without the crossing or twins. This pass may therefore also come from a shortcut that is invariant to the given change — as the authors note, a pass does not certify a semantic representation. Taken together, a failure does not show that the probe is at fault, and a pass does not show that it tracks the property; the diagnostic value of the test has not yet been demonstrated.

3. Questionable probe quality and capacity

  • The training budget is small. The authors did train probes on worlds whose record order is independent of the label, and refitted probes on the permuted presentation, but the number of training cases is very small relative to the activation dimension, the learning curves show no saturation, and some fits stop at the solver's iteration limit without converging. Since the theories are generated automatically and more data costs little, the restriction behind "not decodable at this budget" is effectively self-imposed, and it leaves the central question — whether recoverability holds — unresolved.
  • Limited capacity and sites. The paper uses only linear probe families, with activations averaged over three-block windows at fixed sites (the final period of a statement or the question-mark site). As the authors note, a probe trained at the answer site remains to be examined under the certified test. It therefore cannot be ruled out that stronger probes or other sites would port.
Questions for authors

I would be happy to raise my score if the authors resolve the main weakness, or address some of the following points:

  1. Validate in a non-trivial configuration the model solves in every order. Enable generated reasoning, or use a model stronger on this task, and first establish behaviorally that the model is competent with structural negatives both in canonical order and in random orders; only then test whether the probes port. Please also specify which site the probe is read from once reasoning is generated.
  2. Provide a positive control for the test itself. Construct a model known to hold an order-independent representation of truth (e.g., a model fine-tuned on reachability with records in random orders, or a small transformer with a known mechanism), and show that a probe tracking truth passes the certified test while a probe trained on the positional cue fails, establishing the sensitivity and specificity of the test.
  3. Increase the training budget and capacity of the probes. Scale the label-independent-order training by one to two orders of magnitude and report whether the learning curves saturate; add non-linear probes and fits that converge; and train a probe at the answer site under the same certified test. Only if truth remains undecodable at a sufficient budget can "not decodable at this budget" be strengthened.
  4. Extend the crossing beyond the type-inheritance theories. Add twins and the crossing on ProofWriter to rule out a permutation-invariant shortcut, and discuss how certificates could be approximated on natural-language truth datasets without an oracle (e.g., rewrites verified by NLI or a solver). I would also ask the authors to state explicitly which conclusions are confirmatory and which are exploratory.
  5. (Optional) Characterize when probes port. In more configurations where the model is competent across presentations, characterize systematically when probes port and when they do not, and how this relates to the model's competence.

Other Suggestions

  • Please reduce the custom terminology and the capitalized registered readings, or add a summary table in the main text listing, for each experiment, the preregistered criterion, its reading, and what it implies, so that readers can follow the main line.
  • Appendix C (logically equivalent theories differ in how accessible their facts are) is interesting but only loosely connected to the main line; consider condensing it or stating its connection more explicitly.
2 comments

Thank you for your review and your appreciation of the design. We answer the main weakness first then the three questions.

  1. The paper claims that IID decoding accuracy does not establish what a probe tracks. The evidence is a set of probes with IID AUROC 0.91 to 1.000 whose ranking does not hold under certified changes that preserve the closure, the record set, the derivation length and the token count (the reordering and the twins). Whether this happens because the probe missed a portable representation or because the model has none is a question of attribution; in both cases the IID accuracy certified a layout correlate. We agree with your description of the mechanism: the twins and the interchange interventions show that the model's own answer tracks the wiring of the records in the original order, and we name it controlled shortcut learning (in Discussion). We do not claim that the probes miss an existing representation of truth; the paper states that a failed test does not show that the model lacks one, and the revision makes the attribution explicit.

  2. The refit is what separates the two outcomes. On H2 the property stays linearly decodable in the new presentation (0.999) and the frozen probe fails; under reordering, recoverability is not established and the answer collapses as well. The revision states which inference each outcome licenses.

  3. Section 3.4 is the design you ask for: competence is established behaviorally in canonical order and in two random orders before portability is tested, on 384 new worlds and four models. Its result is that, without generated reasoning, no model is competent with structural negatives in random order at any depth, including one hop, and that the probes trained in canonical order fall to chance there in every model.

The competent cells are the presence questions; the probes port there, and steering along them is not specific. We read this as the paper's third finding: a pass shows invariance to the tested change and not the use of a semantic representation. We agree that this leaves the test without a non-trivial positive control in these models, and we explicit it in (upcoming) updated limitations.

  1. The reordering results do not start from a model known to be incompetent: in canonical order the answer gap separates true from false worlds at AUROC 0.91 in Qwen3-32B. That this separation is itself bound to the layout is a result of applying the certified changes to the answer, not an independent prior; "there may be no portable representation to find" is a conclusion our method delivers. We explicit it now more clearly in the text.

  2. Two controls of the test's mechanics are in the paper. What is missing is a positive control with a known order-independent mechanism for reachability; constructing one (a model trained on reachability in random orders, or a small transformer with a known circuit) is the right next step and cannot be done within the discussion period. Before submission we ran Qwen3-32B in reasoning mode on the fresh-world forced-choice panel; the traces were truncated by the generation budget and the run was inconclusive, so the paper reports the single-forward-pass setting only.

On ProofWriter you are right that the probe reads more than the answer shows, and literal accuracy understates the comparison, so we computed the AUROC of the model's own answer gap on the same rows from the recorded logits. On the 768 entailed rows it is 0.80 [0.75, 0.85] in canonical order and 0.81 [0.77, 0.86] after permutation for Qwen3-32B, and 0.71 [0.65, 0.76] and 0.75 [0.70, 0.79] for Gemma-3-27B-it, against 0.975/0.971 and 0.891/0.902 for the probe. The answer's separation falls with depth (Qwen3-32B: 0.91, 0.72 and 0.73 at depths 1 to 3 in canonical order) while the probe holds. So the probe separates entailed from non-entailed questions better than the answer does, in both orders: on ProofWriter the pass is a pass of portability, and the gap between what is decodable and what the answer uses is the distinction operated in the paper. Whether the probe's extra separation is an unused representation of entailment or a permutation-invariant correlate is what twins on ProofWriter would decide, it is now stated as a next step.

We can answer other elements in an additional comment, but we do not want to exceed the limit. We hope we are clear enough in this comment and it will be constructive.

Thank you for the detailed and constructive response. I appreciate the additional ProofWriter analyses, the clarification of the refit results, and your willingness to state the limitations more explicitly. These additions improve my understanding of the paper.

However, my main concern remains unresolved. On the central reordering tasks, none of the tested models is competent with structural negatives under random ordering, even at one hop, and the reasoning-mode experiment was inconclusive. Thus, the requested setting—in which the model is competent both before and after reordering—is still missing. Canonical-order competence does not fully resolve the concern that probe failure may partly reflect the model’s inability to perform the task.

The H2 refit is informative, but recoverability is not established for the central reordering case. Concerns about the probe training budget, convergence, capacity, and extraction locations also remain. Besides, the collapse of the probes under reordering provides indirect support for my concern that the probes' failure may actually reflect the model’s inability to solve the task. The ProofWriter results provide useful supporting evidence, but they do not replace a positive control on the main task or establish when probes should be expected to transfer.

I recognize the care and effort behind this work, but I regret that the current evidence is insufficient to address my principal reservations. Therefore, I do not intend to raise my score at this time.

Reviewer_fWLYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper examines whether accurate truth probes capture truth or presentation-dependent features. It proposes semantic counterfactual validation using logically equivalent transformations, equal-budget probe refits, and matched theories with controlled changes to truth and record structure. Experiments on synthetic type-inheritance worlds show that some highly accurate probes fail under rewriting or reordering, while experiments on ProofWriter show no detectable degradation under sentence reordering. Additional interventions examine whether the information decoded by probes contributes to the model’s answers. The results highlight the distinction between decoding accuracy, robustness across presentations, and causal use.

Critical strengths
  1. The combination of exact entailment checks and equal-budget refits is a useful experimental control. In particular, the Gemma H2 result separates a failure of the frozen probe from the continued availability of a highly accurate linear readout under the changed condition. This makes the interpretation more informative than reporting an out-of-distribution accuracy drop alone.

  2. The truth-preserving twins directly address a meaningful ambiguity in the truth-changing pairs: a response to an edit need not be a response to its effect on truth. The manuscript also reports unsuccessful preregistered criteria and limits its claims about failed refits and causal use. These qualifications are valuable for evaluating interpretability evidence.

Critical weaknesses
  1. The empirical scope is narrow relative to the proposed diagnosis of truth probes. Most central synthetic experiments use eight-record worlds built around two short inheritance chains, and the canonical layout is acknowledged to correlate with the label. Randomizing names and using fresh worlds prevents name reuse but does not by itself establish structural generalization. The held-out rewriting counterexample rests on one Gemma probe, with the registered cross-model primary criterion unmet; the initial fresh-world portability result is evaluable for Qwen but not Gemma. Later four-model panels strengthen the evidence within this construction, but the ProofWriter reordering result does not show the same failure. A synthetic counterexample is useful, yet the manuscript does not make sufficiently clear how much the method establishes beyond detecting dependencies in this particular generator. In addition, the permutation refits use only 88 or 104 training cases. The reported learning curves are helpful, but do not settle whether the failure to recover is stable at substantially larger budgets. My concern is coverage and sample efficiency, not evidence of cherry-picking.

  2. The interpretation of 'presentation' and the exposition of the controls need greater precision. H2 changes whether the queried fact is explicit or requires a three-step derivation, so closure preservation does not preserve computational demands. Failure of a fixed probe under this change is compatible with a change in how relevant information is encoded; it does not by itself identify a superficial shortcut. Likewise, the crossing preserves selected positions and counts, rather than the complete presentation. The paper acknowledges the truth/edit ambiguity and adds useful controls, but their registered signatures remain UNCLASSIFIED for the in-panel probes, leaving the exact mechanism unresolved. These distinctions are difficult to track because data construction, training contrasts, sample units, and decision rules are spread across many panels and appendices. A worked example and a compact experiment-to-claim table in the main text would make the evidence substantially easier to assess.

Questions for authors

For the H2 result, which inference is justified beyond failure of a fixed readout across logically equivalent inputs? How do the existing controls distinguish reliance on presentation shortcuts from a change in encoding caused by different inference requirements? Please illustrate S0/S1/S2 and H2 with one complete matched example.

1 comment

Thank you for the precise reading. Your two questions point to a distinction the paper left implicit, we will explicit them.

On H2 the fixed Gemma-3-27B-it readout loses part of its ranking (1.000 to 0.892; loss at least 0.070 at the preregistered one-sided level), while a same-family refit on H2 reaches 0.999. The inference is exactly the one you name: the readout is specific to the presentations it was selected on, while the property remains linearly decodable in the new one. We agree that H2 changes the inference demands (a stated or one-step fact becomes a three-step derivation), that preserving the closure does not preserve computational demands, and that a fixed readout failing under this change is compatible with a change in encoding. The per-tier analysis only localizes the loss: it falls on the negatives whose parent records are those of the true world (NEG_PREFIX0), which either account predicts.

The revision separates the certified changes into two kinds.

  • The first kind keeps the closure but alters the inference demands.
  • The second kind keeps:
    • the closure
    • the record set
    • the derivation length
    • the token count
      • the record permutation (the same eight records)
      • and the twins (two records rewired; the false and the frequency-matched twins are token-matched to the true theory).

The claim that the probes track the layout rests on the second kind only: under certified reorderings the frozen probes fall to chance in all four models at every depth (1.000 to 0.47 to 0.53), and the twins show that probes and answers respond to the wiring of records as strongly when it preserves the truth as when it flips it, and only in the original order.

You are right that the crossing preserves selected cues rather than the complete presentation. The twins hold fixed exactly what a position-indexed account could use:

  • the record positions and counts of E, X, T and U
  • the record count
  • and only for the false and frequency-matched twins, the token count.

The records-only positional cue separates the true theory from the false, rewire and frequency-matched twins perfectly in canonical order and is at chance after reordering. The in-panel probes separate even the skip twin, which no single account predicts; we now state more explicitly in the text that which function of the layout the probes compute remains unresolved.

We agree that the construction is narrow and that fresh names establish nothing about structural generalization. The claims are bounded accordingly more explicitly: the method is a certified counterexample generator, one controlled setting suffices to show that IID evaluation can certify a correlate, and ProofWriter is the natural-language comparison. The counterexample rests on one probe only and the registered cross-model criterion failed. The reordering failure, by contrast, replicates in four models.

The refit on 88 to 104 cases is a matched comparison by design: it asks whether a probe of the same family and budget as the frozen one recovers on the changed presentation, and it is not a claim of undecodability. We should have explicited that more clearly. The registered label-independent-order result uses 288 training worlds and 112 candidates and is reported as not decodable at this budget. Following your request we add three further families on the stored activations (a one-hidden-layer MLP, an RBF-kernel SVM and a logistic regression run to convergence): on the certified permutation their cross-fitted AUROC is 0.51 to 0.60 (Qwen3-32B) and 0.51 to 0.55 (Gemma-3-27B-it), against 0.85 to 0.92 and 0.78 to 0.83 for the same refits on the canonical presentation; in label-independent order they reach 0.50 to 0.56 on held-out worlds, as the registered linear probes do. We also add a learning curve: 2,880 fresh worlds generated in label-independent order with the same generator lineage, every family trained on 1,152 and on 2,880 of them, selected on the registered validation worlds and scored on the registered test worlds. For the registered grid the test AUROC is 0.51 (288, registered), 0.54 (1,152) and 0.52 [0.45, 0.59] (2,880) in Qwen3-32B and 0.56, 0.55 and 0.51 [0.44, 0.58] in Gemma-3-27B-it, and no other family exceeds 0.52 at 2,880. The lower bound at ten times the budget stays below the upper bound at the registered budget in both models: the curve does not rise, and the failure to recover is not a matter of sample size at the statement site.

Reviewer_fzCaOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes semantic counterfactual validation to test whether a probe—a classifier reading a language model’s internal activations—tracks truth or merely presentation cues. It combines meaning-preserving rewrites and reorderings, equal-budget probe retraining, and paired theories that separate truth changes from presentation changes. Experiments across four models reveal that highly accurate probes can depend on record order and respond to edits that preserve truth, although probes remain robust on ProofWriter. The central takeaway is that decoding accuracy, robustness across presentations, and causal use by the model are distinct claims, requiring separate evidence.

Critical strengths
  • Strong experimental controls: Certified meaning-preserving transformations and equal-budget retraining distinguish probe transfer failure from information remaining decodable.
Critical weaknesses
  • Poor presentation: Introduction was very hard for me to follow. Some of the problems are listed here.
    • "apply the method to four models" (L69) -> which 4 models?
    • "In type-inheritance theories" (L94) -> what is type-inheritance theory? why does it appear here? No explanation or reference
    • What is the main findings from the experiment?
  • The main body paragraphs are not well-presented either.
    • The introduction to the task this work is experimenting with would deserve a visual explanation or a more reader-friendly explanation.
    • Captions for figures and tables are too verbose, often including unnecessary details. Frankly, this is what I often see from LLM-generated captions.
  • Overall it was hard for me to review the content of this paper, as I had hard time understanding the intention of the paper. I'd like to get clarifications from the authors and adjust my scores, but nevertheless its poor presentation alone is a reason for rejection.
Questions for authors

No questions.

2 comments

Thank you for reading the paper carefully and for pointing to where it was hard to follow. We are very sorry for the lack of clarity: while we focused on reproducibility and validity of experiments, we did not invest engough time on rewriting and reorganising for clarity.

The revision addresses each point you list; the revised PDF will be uploaded by Monday 16 November. We add an illustrative figure 1 and a summary of experiments on table 1 (new figure and table)

  • "Four models" (L69) in Section 1 now reads: "We apply the method to four open models of two families, Qwen3-32B, Qwen3-8B, Gemma-3-27B-it and Gemma-3-12B-it without generated reasoning."
  • "Type-inheritance theories" (L94) The term is now defined first: "A type-inheritance theory is a small set of records stating that an entity has a type ([type, E, A0]) or that one type is a subtype of another ([parent, A0, A1]). The theory entails every type reachable from an entity through subtype records, and a reasoning program computes this closure exactly. Figure 1 shows one."

The new Figure 1 gives one complete theory, the question asked, a certified reordering of its records and the twins used in Section 3.2.

The four findings in Section 1 are now preceded by one sentence that states the result: "In four models, probes that decode the truth of the queried fact within one presentation stop ranking correctly when the same records are reordered, and respond to edits of the links between records whether or not these change the truth; and the model's answer does the same. The test is valid only where the property is invariant to the change (presence questions, and sentence reordering on ProofWriter), and passing does not show that the answer uses the probe."

Captions are shortened to what the figure shows, limiting to crucial elements: its sample and whether it was preregistered, with one reading sentence. All other details are moved to appendix.

The nomenclature is cut down.

To clarify the dense presentation, we express that the method contributes a change of presentation whose effect on meaning is certified exactly, we add an equal-budget refit that separates a probe failing to transfer from the property becoming undecodable, and twin theories that hold the presentation cues fixed while changing the truth to clarify the validity of our results.

The main empirical result is that truth probes with perfect IID accuracy, and the models' answers, track the layout of the records.

We are happy to clarify any remaining point.

Thank you for the authors on the revision.

I appreciate your effort on the revision, but to be honest the clarity of presentation should be a minimum requirement for peer-review, which the initial manuscript fail to achieve to my standard. I therefore maintain my score solely for that reason. However, I won't object if other reviewers recommend for acceptance.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the revision and responses

We thank the three reviewers. Concerns that are shared by all three are addressed here. We conducted some experiments and are revising the pdf as described in the following paragraph.

First, the revised PDF (to be uploaded by Monday 16 November, after careful reading) does the following.

  • Section 1 is rewritten:
    • introduction of four models
    • definition of type-inheritance theory in one sentence
    • main result detailed before the four findings
  • A worked example (i.e. new Figure 1) illustrates:
    • one eight-record theory
    • its held-out interpretation
    • one certified reordering
    • the four twins of Section 3.2.
  • A compact table (i.e. new table 1) with:
    • each experiment with its preregistered criterion
    • the registered reading (to maximize validity of experiments)
    • its status (confirmatory or post-hoc, again to distinguish validity of experiments)
    • what it implies.
  • Nomenclature is reduced, addition of a glossary and use of descriptive label instead of new term when possible
  • Captions are cut to essential elements (the goal was to be thorough, especially on experiments and reproducibility, redaction did not profit from enough rewriting and reflexion).

Secondly, the paper claims that IID decoding accuracy does not establish what a probe tracks, and supports it by exhibiting probes with IID AUROC 0.91 to 1.000 whose ranking does not hold across certified changes of presentation. The revision separates two kinds of certified change:

  • changes that preserve the closure but alter the inference demands (the first three held-out families)
  • changes that preserve the closure, the record set, the derivation length and the token count (the reordering and the twins). The inference that the probes track the layout of the records is now only based on the second kind. In addition, it is more clearly explicited that the claims remain bounded to these theories, linear probes and four open models, as stated in the discussion.

Third, the equal-budget refit is a matched comparison by design, and the registered label-independent-order result is reported, as already registered , as not decodable at this budget. Following the reviewers'requests, a new appendix adds four post-hoc analyses:

  • Non-linear (MLP, RBF-SVM) probes and converged linear fits on the stored activations of the fresh-world replication reach 0.51 to 0.60 on the permuted presentation and 0.50 to 0.56 in label-independent order, same as the registered linear probes
  • A learning curve in label-independent order on 2 880 fresh worlds, scored on the registered test worlds, stays at chance from 288 to 2,880 training worlds (Qwen3-32B 0.51, 0.54, 0.52; Gemma-3-27B-it 0.56, 0.55, 0.51)
  • Probes at the answer site, on the recorded forced-choice prompts forwarded again with their logits reproduced exactly, reach 0.96 to 0.98 and 0.90 to 0.99 in canonical order and 0.50 to 0.61 and 0.54 to 0.58 on the certified permutation, with no recovery on refit and no decodability in label-independent order at this budget
  • On ProofWriter, the AUROC of the model's own answer gap on the entailed test rows is 0.80/0.81 for Qwen3-32B and 0.71/0.75 for Gemma-3-27B-it in canonical/permuted order, against 0.975/0.971 and 0.891/0.902 for the probe.

Finally, in limitations the next stated steps are:

  • a configuration in which a model is competent on structural negatives in every order (this requires generated reasoning or a trained model)
  • a positive control with a model of known mechanism (or many if risk of under-generalization)
  • twins on ProofWriter
AuthorsOpenReview ↗

Upload of the updated pdf

Dear all,

Here is a general diff for the updated pdf:

  • Section 1: the four models are named at first mention; a type-inheritance theory is defined in one sentence; the overall result is stated before the findings; each finding is now one sentence with a section pointer; the contribution list is condensed.
  • New Figure 1 (worked example): one eight-record theory with its question, a certified permutation, the seen and held-out presentations with the three negative tiers, and the false and truth-preserving twins
  • New Table 1: every experiment with its preregistered criterion, its registered reading in words and what it implies; post-hoc parts are marked.
  • Terminology (Section 2 and throughout): many simplifications and capitalized reading labels and most internal codes now appear only in the appendices.
  • Section 3.1: one sentence states that H2 also changes the inference demands, so it shows only that the readout does not transfer; the inference that probes track the layout rests on the permutation and the twins.
  • Section 3.2: we state that which function of the layout the probes compute remains open.
  • Sections 3.1, 3.3, 3.4: one sentence or clause each pointing to the post-hoc analyses of Appendix R.
  • Captions of Figures 2 & 3 and Tables 2 to 4 shortened; the removed details are in Appendix H.
  • Section 5: one sentence on attribution, addition of post-hoc new results and new limitation
  • Reproducibility statement: one sentence on the post-hoc analyses.
  • Appendix C: opening sentence connecting it to Section 3.1.
  • Appendix H: new paragraph with the details moved out of the captions.
  • New Appendix R (post-hoc analyses, Tables 23 to 25): non-linear and converged probes, four and ten times the training worlds in label-independent order, probes trained at the answer site and the model’s answer gap on ProofWriter.
  • New Appendix S: the worked example with its verbatim prompts.
  • New Appendix T: glossary of terms and codes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.