- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies when intermediate states of looped transformers form useful interfaces for downstream computation. The key setting is source-view generalization: multiple sources express the same known intermediate identity, and a newly trained reader must transfer an observed identity-dependent label from a few source views to other views of the same identity.
The main intervention, Direct-ID, temporarily attaches a fresh affine identity classifier directly to the exposed intermediate state during source-model training. The classifier is later discarded, the source model is frozen, and new downstream readers are trained from scratch. The authors compare Direct-ID with native training, additional atomic cross-entropy, representation contraction, supervised contrastive learning, and several control objectives.
Experiments cover two-hop relational composition and decomposed modular addition, using both weight-tied recurrent transformers and untied twelve-block models. The study additionally varies supervision depth, auxiliary-gradient strength, intervention timing, source-view coverage, and reader type. Direct-ID substantially improves linear identity readability and source-view generalization. Under the bounded gradient rule, it improves the graph fresh-reader endpoint by 5.2--7.6 percentage points across seven tested settings. The experiments also show that near-perfect identity decoding does not imply equally strong linear decoding of arbitrary identity attributes, and that representations with similar identity grouping can retain different amounts of source-specific information. Overall, the paper argues that the quality of an intermediate interface depends jointly on identity accessibility, training exposure, and the distinctions required by the downstream reader.
Critical strengths
-
The experimental controls are unusually careful. The paper explicitly separates supervision location, readout location, gradient magnitude, parameter sharing, and intervention timing. In particular, the supervision-depth scan over s = {0, 2, 4, 8}, together with shared and untied architectures and fixed versus bounded gradient rules, provides a substantially stronger test of the apparent supervision-distance effect than a simple auxiliary-loss comparison. I also appreciate that the paper limits its mechanistic claims and does not attribute the remaining depth effect directly to Jacobian attenuation.
-
The paper makes a useful distinction between several notions of representation quality. Identity probes, random identity-attribute readers, retained computation, nearest-view retrieval, and source-factor readers measure meaningfully different properties. The geometry-matched comparison is particularly informative: Direct-ID and Contraction can have similar W/B while preserving substantially different amounts of source-specific information. This provides concrete evidence that a single class-separation statistic is insufficient to characterize the usefulness of an intermediate state.
Critical weaknesses
-
The external validity is limited. Both task families are synthetic and highly controlled. Every evaluated identity is already known, every downstream identity-attribute label has been observed on training views, and the processing interface is a single-token state. The experiments therefore study source-view transfer for known identities, which remains substantially removed from pretrained multi-token interfaces, unseen source distributions, or unseen identities. This limits the strength of the broader representation-reuse claim.
-
The gradient-budget control does not actually equalize the realized auxiliary pressure across supervision depths. In the shared graph model, the bounded controller attains mean auxiliary/native norm ratios of 0.088 at s = 0 and 0.059 at s = 8, while the coefficient cap is active for 32.7% and 70.3% of updates, respectively. Consequently, the remaining depth gap still mixes supervision location with cumulative gradient magnitude, gradient direction, and optimizer history. The paper acknowledges this limitation, but it makes the mechanistic interpretation of the supervision-depth result suggestive rather than conclusive.
p.s.: The discussion section seems to be too long. More weights are suggested to be put in the explanation and analysis of the results. (This reflects my personal preferences only but does not form the basis for any scoring results.)
Questions for authors
Overall, the current illustrated experiments are excellent in general. However, there are two questions below. My confidence score will be raised if the questions could be fully answered.
Q1: Could you add an experiment that more closely matches the realized cumulative gradient exposure across supervision depths, for example by increasing c_max or using a controller that matches the time-integrated auxiliary/native gradient-norm ratio?
Because the s = 8 condition hits the coefficient cap very frequently, it is currently difficult to quantify how much of the residual s = 0 versus s = 8 gap in Table 9 is attributable specifically to supervision location.
Q2: Most transfer evaluations train a new reader. In the Discussion section, you describe a stronger functional-transfer test in which the native continuation is held fixed and receives a state taken from another source context. Could you include this experiment in the current synthetic setting?
It would substantially strengthen the claim that the intermediate state is directly reusable by downstream computation and would separate easy readout relearning from compatibility with an already learned continuation.
3 comments
Thank you again for your two concrete requests; both are now addressed in the revised version.
E1 equalizes realized exposure across depths in two ways. Of the 1.91-point bounded-rule gap, equalization removes 0.36 points, leaving 1.55 points. This quantifies how much of the observed gap is attributable to exposure differences rather than location, although the underlying mechanism is not fully resolved. E2 tests the frozen continuation without any refitting (Section 4.4).
We would welcome your any further questions or concerns.
Thank you for your response. E1 addresses my concern about a fair comparison across supervision depths by showing how much of the gap remains when relative gradient strength is matched. E2 answers my second question in the synthetic setting. These results show extra evidence in the findings, especially the evidence for reuse by a fixed decoder. My overall assessment remains consistent with my original score.
Response to Reviewer CMYn
Thank you for the careful reading and the two concrete requests. Both are now experiments (E1, E2), summarized in Sections 4.2 and 4.4 of the revised PDF, with details in Appendix E.
Q1 and W2, matching realized exposure (E1). We add three arms on the control cohort of Table 9.
Results in pp, s = 0 minus s = 8 (Table 13, Figure 3c):
Direct answer to Q1 (graph, looped): of the 1.91-point gap under the bounded rule, equalizing realized exposure removes 0.36 [−0.07, 0.79] pp, which is not resolved, and 1.55 pp remains at matched exposure. Across layouts this residual is 0.8–2.5 pp; under matching its interval excludes zero in three of four layouts (the fourth just includes zero), and with the raised cap in two. The larger fixed-rule gap is mostly dose: the bounded rule alone removes 64 / 65 / 63 / 15% of it while exposure is still unequal (Table 19). The gradient cosine falls from 0.37 to 0.13 over the pulse, so equal magnitude leaves direction and optimizer history unmatched. Section 4.2 now reads: most of the depth effect is dose, and the residual is associated with location but does not identify Jacobian attenuation.
Q2, functional transfer under the frozen continuation (E2). We freeze each 200k model's decoder, the native continuation of your question, and refit nothing (Section 4.4, Figure 5, Tables 14–15).
Graph results (Native / Atomic-CE / Direct-ID / Contraction / SupCon):
Direct-ID minus Atomic-CE is +22.0 [13.9, 30.1] pp, and +21.5 [18.0, 25.1] in bounded-rule models; addition is similar (94.3 vs 75.3%). Because nothing is refitted, the gain cannot come from easier readout relearning, the distinction you asked us to draw. The native decoder partly re-derives identity from the residual, whereas after identity supervision it relies on the prototype. Contraction states are the most interchangeable, consistent with tighter grouping and larger source-factor loss. This shows interchangeability under one fixed decoder, not algorithm identity.
W1, external validity. We agree. The abstract and Section 1 now state the scope and list what we do not claim. We did not add a pretrained setting in this cycle; Section 5 names structured identities, unseen sources and pretrained multi-token interfaces as the next tests.
Moreover, following your suggestion, the interpretation of each result now sits next to it in Section 4 (e.g., "Why Atomic-CE trails" and "What changing depth changes" in Section 4.2), and the Discussion is organized around three lessons drawn from the results, each related to prior work.