acceptodds
Under review as a conference paper at ICLR 2027

Residual Geometry Separates State Motion from Predictive Change in Weight-Tied Iterative Refinement

Abstract

For weight-tied networks, the relative hidden-state residual is commonly applied as a convergence diagnostic, yet it can remain large when predictions have frozen and can look similar while predictions are still changing. We explain this mismatch by decomposing a raw update into shared scale, unit-wise scale heterogeneity, and within-unit direction. The decomposition is exact and exposes which motion a readout can see. Across nine 3D refiners, the terminal residual clusters near and consists almost entirely of shared scaling, with no label flips. The fully retained double-precision bundles attribute the late directional plateau in the float32 runs to a storage floor. An extrapolating prefix-sums model and three pretrained LoopFormer checkpoints instead remain orientation- and output-active. Equal-energy interventions confirm that component energy and decision sensitivity are different: direction flips 6–13% of 3D points and 63–75% of LoopFormer tokens, while shared scale flips none. A projective constraint makes the realised state contract but does not stabilise prediction. These results motivate a geometry-and-readout halting gate that requires resolved orientation and output change to settle and refuses unsafe truncation of an unfinished internal clock. On held-out LoopFormer sequences, that refusal is necessary: matched-cost dynamic rules lose more than a nat while the native schedule loses only nats at 25% lower compute.

Primary area:
interpretability and explainable AI
Keywords:
weight-tied networks, recurrent-depth models, iterative refinement, hidden-state dynamics, convergence diagnostics, residual geometry, adaptive computation, model interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bmFuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

I am new to the field of weight-tied networks and therefore cannot provide a sufficiently reliable assessment of the paper's technical aspects.

Critical strengths

N/A

Critical weaknesses

The organization and presentation require substantial improvement to enhance readability. In particular, the paper should clearly introduce the problem setting and notation without assuming prior familiarity with the setup. Moreover, despite having ample space available, the manuscript devotes only a few lines to related work.

Questions for authors

N/A

2 comments

The organization and presentation require substantial improvement to enhance readability. In particular, the paper should clearly introduce the problem setting and notation without assuming prior familiarity with the setup. Moreover, despite having ample space available, the manuscript devotes only a few lines to related work.

We agree that the opening asks readers to interpret several residuals before giving them a reason to distinguish them. We will start §1 with one input, one update and one prediction. A weight-tied refiner applies the same learned function at every step. Let HtH_t denote the hidden features after step tt and ptp_t the class probabilities obtained from them. If the next step multiplies every feature by 1.34, the relative state change is 0.34, but its direction is identical. A normalized head can give identical probabilities. In our 3D model the logits still move while saturated probabilities and labels can stay fixed. This example explains why we measure state motion, probability change and task performance separately.

We will introduce the unit of analysis (a point or token) before the projection formulas in §2.1. A short key next to the first example will define the relative residual, norm ratio, normalized-state change and output divergence in words. Figure 1 will follow that example. Its caption will identify which curves measure state, predictions and task performance, and identify the training depth. Readers can then consult Appendix D for the proof and Appendix C for numerical resolution.

We will expand §1’s related-work paragraph around three questions a newcomer can ask. What does a fixed-point solver certify (DEQ and monDEQ)? What evidence permits an early exit (CALM’s saturation signal and Huginn’s output KL)? How is the stopping decision learned in ACT and PonderNet? These comparisons explain what each method monitors and why we report both executed updates and measured latency.

Thank you for your responses. I believe the manuscript requires substantial revision to meet ICLR's presentation standards. I therefore maintain my original scores.

Reviewer_C44BOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that the relative residual rt=∥Ht+1−Ht∥/∥Ht∥r_t=\|H_{t+1}-H_t\|/\|H_t\|, often used to judge convergence of weight-tied models, mixes kinds of motion that matter differently for predictions. It splits rtr_t exactly into norm change and orientation change, and further into shared scaling, per-unit scale differences and direction. Shared scaling is invisible to scale-invariant readouts. The paper then proposes a halting gate based on orientation and output change. In a weight-tied 3D segmentation refiner, the residual stays near 0.34 while labels freeze, because the late motion is almost pure scaling. In prefix-sums extrapolation and LoopFormer, both the direction and the outputs keep changing. On LoopFormer, dynamic stopping rules lose more than one nat versus the native schedule, mainly because they truncate the model's internal clock.

Critical strengths
  1. The decomposition is exact and cheap, and it clearly explains why the raw residual groups models with opposite behaviour (Table 1).

  2. The measurements are unusually careful: float64 reruns separate numerical floors, a positive control is included, and injection experiments localise the shared-scale null to RMSNorm.

  3. The protocol is honest: thresholds are frozen before testing, and flawed or non-reproducible results are disclosed.

Critical weaknesses
  1. The central anomaly may come from one divergent model. Only the authors' tied COSeg refiner shows it. In that model ρ≈1.34\rho\approx1.34, so the norm grows by about 10710^7--10810^8, and the task metric peaks before the training depth. With damping 0.2, ρ\rho drops to 1.008. Does the mismatch persist in that setting?

  2. The evidence is narrow and the theory is elementary. Beyond the 3D model there are only one prefix-sums and three LoopFormer checkpoints, with no DEQ, and Eq.~1 is the law of cosines. Does the same mismatch appear in a DEQ trained to a fixed point?

  3. The halting gate is not validated. Its only test is on LoopFormer, where it abstains and the loss comes from truncating the internal clock; no experiment shows it saving compute. Related criteria, such as CALM's hidden-state saturation and KL-based exit in recurrent-depth LMs, are not discussed. Can you report compute--accuracy curves against fixed depth and output-only rules on a model where the gate actually halts?

  4. The presentation is hard to follow. Key notions such as χt\chi_t and the native and retimed schedules appear only in the appendix, and LoopFormer is evaluated on training-shard slices. Can you move these definitions into the main text and use an official validation split?

Questions for authors

Please see the questions in the Weaknesses section.

2 comments

Q1. The central anomaly may come from one divergent model. Only the authors' tied COSeg refiner shows it. In that model ρ≈1.34\rho \approx 1.34, so the norm grows by about 10710^7--10810^8, and the task metric peaks before the training depth. With damping 0.2, ρ\rho drops to 1.008. Does the mismatch persist in that setting?

At damping 0.2, the persistent shared-scale effect disappeared in all three matched training seeds. We tested each checkpoint on the same 1,200 query clouds. The strict event required residual ≥0.01\geq 0.01, shared update-energy fraction ≥0.99\geq 0.99, mean output JS ≤0.0001\leq 0.0001 nat and zero label flips on each of the final three transitions. At both epochs 76 and 100, every seed had 0/1,200 strict events. Dropping only the shared-energy condition gave 52, 7 and 22 cases at epoch 100 (4.33%, 0.58% and 1.83%). The mIoU peaks occurred at depths 64, 49 and 1, respectively, against training depth 8. At damping 0.9 every query became nonfinite at steps 55–60, so its depth-64 mismatch rate is undefined (Table 2, Appendix F).

Q2. The evidence is narrow and the theory is elementary. Beyond the 3D model there are only one prefix-sums and three LoopFormer checkpoints, with no DEQ, and Eq. 1 is the law of cosines. Does the same mismatch appear in a DEQ trained to a fixed point?

Equation (1) is the law of cosines. The empirical question is whether predictions settle before the state reaches equilibrium. In a pretrained monDEQ on MNIST, all 2,000 held-out images converged at solver tolerances 10−510^{-5} and 10−610^{-6} (median 19 and 22 iterations). The median true model defect at the returned state fell from 1.16×10−51.16 \times 10^{-5} to 1.47×10−61.47 \times 10^{-6}. At the primary tolerance, 0/2,000 images showed the strict shared-scale signature anywhere. Under the broader criterion, 1,773/2,000 (88.65%) had three consecutive transitions with true defect ≥0.01\geq 0.01, adjacent accepted-readout JS ≤0.0001\leq 0.0001 nat and unchanged labels. None met that criterion in the final three transitions. This DEQ exhibits a transient gap (Table 4).

Q3. The halting gate is not validated. Its only test is on LoopFormer, where it abstains and the loss comes from truncating the internal clock; no experiment shows it saving compute. Related criteria, such as CALM's hidden-state saturation and KL-based exit in recurrent-depth LMs, are not discussed. Can you report compute--accuracy curves against fixed depth and output-only rules on a model where the gate actually halts?

Figure 4 and Appendix L give quality versus executed updates at 0%, 10%, 25% and 50% target savings for four prefix-sums and six COSeg settings. We froze thresholds on separate calibration inputs and compared the joint gate with fixed depth, state-only rules (including CALM-style cosine saturation), and JS- or KL-only exits as used in recurrent-depth models.

At the 25% target on COSeg seed 123, cap 8, the gate exited early on 703/1,200 clouds, averaging 6.02 updates (24.74% saved). Its mIoU was 0.33851 versus 0.33919 at fixed depth 6 (paired 95% CI for the difference −0.00392 to 0.00253). It gained 0.00467 mIoU over KL alone and 0.00441 over JS alone, with realized cost gaps of 1.20 and 2.00 percentage points. That gain varied across checkpoints. In online execution, 226/400 clouds exited early on the first timed pass. Median RTX 4090 latency was 406.63 ms per episode gated versus 380.88 ms at depth 6. Fixed depth was faster in every timed condition.

Q4. The presentation is hard to follow. Key notions such as χt\chi_t and the native and retimed schedules appear only in the appendix, and LoopFormer is evaluated on training-shard slices. Can you move these definitions into the main text and use an official validation split?

Section 2.2 now defines χt\chi_t as the clock check and gives an example before Eq. (5). Section 4.6 defines the three schedules. With full budget NN and kk executed loops, native takes kk steps of size 1/k1/k and reaches time 1. Truncation takes kk steps of size 1/N1/N and ends at k/Nk/N. Retiming changes the last step to 1−(k−1)/N1 - (k-1)/N and reaches time 1 at the same update count.

Table 6 now uses the publisher-defined English C4 validation split (256 documents of 1,024 tokens per checkpoint). At 25% fewer updates, native schedules increased NLL by 0.02367–0.02740 nats/token. Recalibrated JS-only exits saved 24.32–24.90% but increased NLL by 2.12867–3.26141 nats/token. The clock-aware gate made zero preterminal exits on all three checkpoints. Appendix I reports reconstructed OpenWebText validation separately and identifies retiming curves as calibration diagnostics.

Thanks for your detailed rebuttal. I will keep my original score.

Reviewer_5KYtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the relationship between hidden-state convergence and predictive convergence in weight-tied iterative networks. The authors propose a geometric decomposition of hidden-state updates into shared scale, unit-wise scale heterogeneity, and within-unit direction. Experiments on a 3D refiner show that hidden states can continue changing even when predictions have stabilized. The paper further experiments on LoopFormer but show that predictions continue changing across iterations and do not converge.

Critical strengths

The observation that hidden-state convergence and predictive convergence can differ is interesting.

Critical weaknesses

The paper is difficult to follow and the technical presentation is unnecessarily convoluted. The core idea is relatively simple: hidden-state updates can change the magnitude of the representation without changing its direction or predictions. However, the paper introduces multiple decompositions and a lot of technical details, making it difficult to understand what the main contribution is. The paper would benefit from a much clearer presentation of its motivation, main contribution, and experimental findings.

The methodology has limited novelty. The proposed geometric decomposition is based on standard vector projection and orthogonal decomposition. Similarly, the observation that shared scaling does not change predictions under a scale-invariant readout is expected. It is therefore unclear what the novelty of the proposed decomposition is.

Experiments are narrow in scope and does not adequately support the claims of the paper.

  • The result that the hidden-state residual remains large despite stable predictions is interesting. However, this is not observed in the loopformer experiments. Since the paper aims to study adaptive computation in weight-tied networks, it is unclear how useful this finding is beyond the specialized 3D network.
  • The experiments do not demonstrate that the proposed geometric-based halting heuristic leads to improved inference efficiency.
Questions for authors

Can the authors demonstrate a practical improvement in the accuracy-compute tradeoff using the proposed halting heuristic? How often does the criterion actually halt early, and how much computation does it save?

The main positive finding is observed in the 3D segmentation network but not in LoopFormer. Can the authors provide more evidence that this generalizes to other weight-tied architectures, particularly looped transformers?

3 comments

We tested the gate in two clock-free settings and audited an independent looped transformer. Q1. Can the authors demonstrate a practical improvement in the accuracy-compute tradeoff using the proposed halting heuristic? How often does the criterion actually halt early, and how much computation does it save?

We observed early exits on prefix sums and COSeg. Fixed depth gave a better accuracy–latency tradeoff on the tested hardware. The joint rule requires both normalized-state change and adjacent-output Jensen–Shannon (JS) divergence to settle. Thresholds were frozen before testing. We report 1−E[K]/N as the fraction of recurrent updates saved, where K is the executed depth and N the cap. On prefix sums, each of four settings used 2,048 final sequences. At the 25% target the gate saved 23.85–26.91% of updates. At 32 bits and cap 30, 2,015/2,048 sequences exited early (98.39%), averaging 22.84 steps. Both the gate and fixed depth 22 achieved 100% position accuracy. At batch size one, median times were 28.43 ms gated, 12.14 ms at full depth and 9.02 ms at depth 22. Batch-32 medians were 49.98, 11.94 and 8.90 ms. We excluded known released examples. The historical training set was unavailable for a complete overlap check. COSeg used 300 calibration episodes and 600 test episodes from disjoint rooms. The table reports frozen-policy replay over the same 1,200 test query clouds per condition.

COSeg checkpointCapEarly exits / 1,200Mean updatesUpdate saving
1 (original)87036.020824.74%
1 (original)641,19248.074224.88%
2 (seed 777)81,0886.008324.90%
2 (seed 777)641,20048.028324.96%
3 (seed 2026)81,0475.986725.17%
3 (seed 2026)641,19747.914225.13%

With the first checkpoint at cap 8, mean intersection over union (mIoU) was 0.33851 gated and 0.33919 at fixed depth 6. The paired difference was −0.00068 (95% room-bootstrap interval −0.00392 to 0.00253). At cap 64, the gate used 48.07 updates for mIoU 0.33481. Calibration-selected fixed depth 9 achieved 0.33913 (gate minus fixed −0.00432, interval −0.00661 to −0.00210). We also executed the exits online on an RTX 4090. On the first timed pass of 200 episodes, 226/400 query clouds actually exited early at cap 8. Across five passes mean depth was 6.08. Median latency per episode was 406.63 ms gated, 388.78 ms full depth and 380.88 ms at fixed depth 6. The paired mean overhead had a 95% interval of 24.99–28.10 ms. At cap 64, gated medians were 728.8–733.8 ms across checkpoints versus 377.8–392.8 ms at the selected fixed depths.

Q2. The main positive finding is observed in the 3D segmentation network but not in LoopFormer. Can the authors provide more evidence that this generalizes to other weight-tied architectures, particularly looped transformers?

The persistent shared-scale effect has replicated across nine independently trained COSeg-based refiners. Its transfer to looped transformers remains unsupported. We tested an additional architecture, Huginn-0125, in native BF16 on 256 frozen sequences from the official C4 validation split. We audited caps 32 and 64. Instrumented readouts reproduced native logits, and the two runs had identical records for their first 32 steps. At each cap, 0/256 sequences met our prespecified three-transition shared-scale signature anywhere in the trajectory, including its end. At cap 64, the median relative residual and normalized-state change were both 0.01455. The median shared-energy fraction was 5.36×10−5, median output JS was 1.15×10−4 nat, and the median token-flip rate was 1.173%. The measured motion was predominantly directional. A controlled perturbation gives a different piece of evidence. At step 32, a shared scaling perturbation of 10% produced median JS of 5.74×10−5, compared with 1.26×10−3 for a directional perturbation of the same amplitude. A readout can be relatively insensitive to scale even when the trajectory rarely moves along that direction. All three LoopFormers also continued changing predictions at their configured horizons. Mean terminal token-flip rates were 20.70–34.75% on C4 and 19.94–33.64% on reconstructed OpenWebText. We will revise the heading of §4.4 and give natural trajectories and interventions separate columns in Table 4. That distinction also addresses the novelty concern. Standard vector projections supply the measurement. Testing which component actually moves labels or probabilities supplies the empirical result.

Area_Chair_6zXwOpenReview ↗

Please Respond to the Authors’ Rebuttal

Dear Reviewer 5KYt,

The discussion period ends in a few hours. Could you kindly check the authors’ responses and let us know whether your concerns have been addressed?

Thank you!

Best regards, The AC

Thank you for the response. However, my main concerns regarding the unnecessarily convoluted presentation, limited novelty, and lack of demonstrated practical benefits of the proposed method remain. I therefore my maintain my score.

Comments on the paper

1 comment

We thank the three reviewers for asking when a moving hidden state matters for a prediction, and whether that distinction can save inference time. The new experiments give different answers to those questions.

1. The strongest effect depends on the operating regime. In the original nine tied 3D refiners, late updates largely rescale a state whose predicted labels have frozen. Under stronger damping, none of the three new training seeds reproduces that terminal signature. A weaker gap survives on a small fraction of query clouds. The revision will identify the original result as a finding about a particular family and training regime.

2. The timing of prediction stability differs across architectures. An independent recurrent transformer, Huginn, continues changing predictions. In a convergent monotone deep equilibrium model, labels and probabilities can settle for several iterations while the model still has a measurable fixed-point defect. The defect then falls as the solver converges. Equation (1) is the law of cosines. The empirical contribution is to identify which motion persists, whether the readout sees it, and at which point in the trajectory that distinction matters.

3. The gate can skip updates. Its overhead matters in practice. On one COSeg checkpoint, 703 of 1,200 query clouds exit before depth 8. The gate saves 24.74% of recurrent updates. On the timed subset, median end-to-end latency rises from 380.88 ms at fixed depth 6 to 406.63 ms with the gate. Fixed depth remains the stronger deployment baseline in our tests.

Changes we will make to the manuscript

  1. Abstract, §1 and §5. Open with one contrast from Table 1. The original 3D refiner has a terminal residual near 0.34 with frozen labels, while extrapolating prefix sums has a smaller residual near 0.17 with continuing label changes. Define state stationarity, stability of the predictive distribution, and task performance separately. Rewrite the contributions around these measurements and give the measured latency result alongside the update savings.

  2. §1, “Relation to prior work.” Expand the current paragraph into three short comparisons. Describe the model defect used in DEQs, the confidence or saturation signals used by CALM and recurrent-depth KL exits, and the learned stopping distributions of ACT and PonderNet. For each, identify the observed signal, whether thresholds require training or calibration, and what computation is charged. Explain why a time-conditioned recurrent model also needs a clock check.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.