acceptodds
Under review as a conference paper at ICLR 2027

Dense Supervision for Iterative Neural Computation

Abstract

Iterative Neural Computations (INCs), such as Universal Transformers and Deep Thinking networks, have demonstrated strong capabilities across algorithmic, reasoning, and perceptual tasks, by iteratively applying a shared function multiple steps. Despite their promise, training these systems remains challenging. These models are typically trained by applying a loss only at the final iterate, a recipe we call endpoint supervision. We hypothesize that this recipe is the cause behind two coupled failure modes: (1) gradients from the final loss must backpropagate through the entire chain of repeated updates, producing a learning signal that is both noisy in magnitude and unreliable in direction, and (2) intermediate iterations are left unconstrained, allowing the model to learn a fixed-horizon strategy that breaks down when more iterations are run at test time. We introduce Dense Intermediate Consistency for Endpoints (DICE), an architecture-agnostic framework that addresses both issues by supervising the full trajectory of computation. DICE attaches a shared readout head to every iterate and applies auxiliary losses at each step, providing gradient signals that bypass the deep composition while enforcing that every intermediate state remains semantically meaningful. Across three representative architectures, DICE yields consistent gains, including near-perfect algorithmic extrapolation on prefix sums, pp exact-match accuracy on maze solving, and a inference-time speedup via adaptive halting. Overall, these results indicate that dense supervision is a simple and broadly effective remedy for the failure modes of endpoint-supervised INCs.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Machine Learning, Reasoning, Adaptive Computation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_u2q6OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a method called Dense Intermediate Consistency for Endpoints (DICE). It is essentially a weighted deep supervision method, where the final readout is applied to every hidden state and those intermediate predictions are used in the loss function as well (with a particular weighting schedule). The authors provide evaluations on prefix sums, maze solving, and bAbi question answering. The authors also provide some mechanistic analysis studying the gradients in endpoint supervision and DICE.

Critical strengths
  1. DICE shows improvement over the chosen baselines on the datasets.
  2. Section 4.5 and 4.6 provide some mechanistic analysis giving intuition on why the method works better than endpoint supervision.
Critical weaknesses

1. Minimal experimental suite. The current evaluations on prefix sums, maze solving, and question answering are limited. I would appreciate a much larger experimental suite on more datasets and tasks.

Concrete suggestions:

  • a) Since the authors cite TRM, which evaluates on ARC-AGI, further evaluations on ARC-AGI would strengthen this work.
  • b) The authors can perform evaluations in a language modeling setup, similar to Saunshi et al. 2025 (Reasoning with Latent Thoughts...). In that work, the authors pretrain on language modeling with the Pile dataset and evaluate on Closed book QA, open book QA, math word problems, etc. (See Table 3). There are a number of tasks in each setting in that work.

I think the experimental setup is too minimal as it stands.

2) Lacking novelty. The authors mention that in prior work, "a loss is computed only at the final iterate, and gradients are propagated back through the entire chain of updates". However, this is not entirely true. The authors cite the work of Bansal et al. 2022 ("End-to-end algorithm synthesis with recurrent networks: Logical extrapolation without overthinking"). That work proposes a "progressive loss", which is like a one-state deep supervision instead of an every-state supervision (this work). The motivations for both are very similar as well (Bansal et al. 2022 mentions "incremental progress", which is similar to the "unconstrained intermediate iterations" problem in this work). I could not find where in the paper the authors mention this similarity; if the authors could point this out, I would appreciate it because I may have missed it.

Some works to distinguish from, each of which has similar deep supervision losses:

  • a) "End-to-end algorithm synthesis with recurrent networks: Logical extrapolation without overthinking" Bansal et al. 2022 (see above)
  • b) "LOOPED TRANSFORMERS ARE BETTER AT LEARNING LEARNING ALGORITHMS" by Yang et al. 2024.
  • c) "Scaling Latent Reasoning via Looped Language Models" by Zhu et al. 2025.
Questions for authors

As I understand it, DICE is a major contribution of this work; however, I'm having a hard time understanding what is novel about DICE. Can the authors explain what makes DICE novel compared to prior work? If DICE itself is not the contribution, can the authors clarify what the main contribution of this work is?

3 comments

Thank you for your careful review and the concrete suggestions on experiments and related work.

W1a — ARC-AGI-1 with TRM (10k epochs, official hyper-parameters, EMA)

ARC-AGI-1 evaluation set (400 puzzles), %. pass@k from the official voting evaluator.

Model5k ep: pass@1pass@2Accuracy10k ep: pass@1pass@2Accuracy
Stock TRM3.756.250.9917.0020.887.94
Full-grad, no intermediate loss10.3812.753.3228.6332.5019.14
DICE (α=1.0)15.1319.755.2232.0036.2521.15

This is a limited-budget ARC result, at 10% of the official training schedule. It establishes promising applicability, instead of performance at convergence.

W1b — Language-model setting

We applied DICE to a 3.5B-parameter recurrent-depth language model (Huginn), continued-pretrained on wikitext-103 at K=8 recurrent steps, 3 seeds, with a single uniform weight schedule shared by every method that requires intermediate supervision. Evaluation uses a fixed seed for the latent initialization, so every model is scored on identical inputs and initial states; the metric is top-1 next-token accuracy when forced to emit after 1 recurrent step.

MethodAccuracyΔ vs endpoint
Endpoint0.288 ± 0.001-
RLTT0.293 ± 0.002+0.005
GRPO0.296 ± 0.001+0.009
LSRL0.318 ± 0.001+0.030
DICE0.341 ± 0.000+0.053

Since DICE is trained to decode at every step while the baselines are not, a step-1 number alone cannot show whether standard LM quality improves. In domain, at every depth (3 seeds, perplexity / top-1):

Methodk=1k=2k=4k=8
Endpoint59.00 / .287425.76 / .381913.47 / .469310.10 / .5137
DICE38.34 / .340620.33 / .414112.55 / .478310.12 / .5124

Off-distribution and downstream. We evaluated at every depth k=1…8 on six held-out corpora (LAMBADA, C4, OpenWebText, arXiv, PubMedQA, GitHub) and zero-shot on HellaSwag/PIQA (3 seeds). At k=1 DICE lowers mean out-of-domain perplexity by 4.54±0.05 (77.83→73.29), leading on 4–5 of 6 corpora in every seed, and gains +3.2±0.3 pp on HellaSwag and +2.1±0.9 pp on PIQA. GRPO in particular degrades sharply off-distribution (49.0 vs endpoint's 19.5 perplexity at k=8), consistent with our Related Work argument that RL post-training must undo what the base model already learned.

In conclusion, DICE substantially improves the early-exit quality and compute–quality trade-off of looped language models, both in-domain and out-of-domain, while approximately preserving full-depth performance in-domain. This suggests DICE is particularly useful when variable inference depth or early exiting is important.

W2 — Progressive loss (Bansal), looped-TF loss (Yang), LoopLM loss (Zhu) vs DICE

See the 7MKQ Q1 table and the 7vrs design-space table.

W2 / Q — Novelty and main contribution

We will add the comparison to Bansal et al., Yang et al., and Zhu et al. in Related Work. The losses differ as follows:

LossStates supervisedWeightingGradient flow
Progressive (Bansal)one state after a random no-grad prefixmixed with max-iteration losstruncated
Looped-TF (Yang)last T iterationsuniformtruncated to window
Exit-gate (Zhu)alllearned exit distribution + entropyfull
DICEallfixed schedule (uniform/linear/exp)full

Our main contribution is not intermediate supervision itself, but a controlled study of how it is implemented: which supervision designs (which states are supervised, how they are weighted, whether gradients flow across steps, shared vs. separate readout) are effective for optimization and test-time extrapolation, why they help, and when the benefit persists across architectures (recurrent nets, recursive/Universal Transformers, TRM, a 3.5B looped LM). For example, on 512-bit prefix sums under one network, uniform DICE reaches 96.2% versus 30.3% for progressive loss and 30.9% for exit-gate loss, so intermediate supervision alone does not guarantee extrapolation.

The detailed design-space table and the language-model ablation are in our response to Reviewer 7vrs.

I thank the authors for including the experiments I requested. I have updated my score to 2 to reflect this. My concerns about novelty remain. As the authors mention, supervising intermediate iterates is already established (Bansal et al., 2022, Yang et al., 2024, Zhu et al., 2025, TRM). By the rebuttal’s own comparison, DICE differs from these mainly in which iterates it supervises and how it weights them. If the primary contribution is a controlled study of how intermediate supervision is implemented, these prior losses should be compared with tuned baselines across all tasks in the main paper. Currently, this comparison is too minimal, so I cannot recommend acceptance.

Thank you for raising your score. We agree that, given the prior losses, the comparison must be broader. Below we extend it to another setting, and we will include the comparison against Bansal et al., Yang et al., and Zhu et al. with tuned baselines across all tasks in the main paper.

Table 1: Universal Transformer on bAbI (10k). Test accuracy (%) averaged over the 20 tasks, ± std over 5 seeds.

MethodMean accTasks 1–5# Tasks below 95% accuracy
Endpoint86.57±1.3590.027
Progressive loss α=1 (Bansal)74.92±3.3976.7213
Window loss T=3 (Yang 2024)76.23±1.9881.9113
Exit-gate loss β=0.05 (Zhu 2025)86.33±1.6485.167
DICE exponential w_k87.97±1.2292.944

DICE has the best mean over 20 tasks and tasks 1–5 accuracy, and has smallest number of tasks the perform less than 95% accuracy.

Reviewer_7MKQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DICE, which adds auxiliary losses at intermediate iterates of iterative neural computations (INCs). All intermediate losses are decoded through the same readout head used at the endpoint. The paper also proposes Adaptive Stability Halting (ASH), which stops inference once the ℓ1 change between consecutive softmax predictions stays below a threshold. The motivation is two failure modes attributed to endpoint supervision: gradient degradation through long Jacobian chains, and unconstrained intermediate states that hurt test-time extrapolation.

Critical strengths
  1. The method is simple and broadly applicable. It needs no architectural change and is applied to three structurally different iterative systems (convolutional recurrent, recursive Transformer, Universal Transformer). The prefix-sum result is striking: 100% at 128 bits where the endpoint model is at 0%, and 99.95% at 256 bits when ASH is allowed up to 80 iterations (Table 3).

  2. The paper attempts to isolate mechanisms rather than only report final accuracy. The four-way ablation in Table 7 is well designed. Comparing against a final-state repeated-loss control and a separate-heads control is the right way to separate added loss weight, gradient-path shortening, and the shared-readout constraint.

Critical weaknesses
  1. The relation to existing intermediate supervision in iterative models is under-discussed. Related work only contrasts DICE with deeply supervised feedforward nets. However, several of the paper’s own base systems and cited works already supervise intermediate iterates: (1) TRM’s outer N_sup loop applies a loss after each outer step through the shared output head. The “Endpoint” baseline in Tables 4–5 runs 16 such outer steps, so it is not endpoint-supervised in the sense of Eq. (2). DICE adds losses at the H-cycles within an outer step. (2) Bansal et al. (2022), use a progressive loss for Deep Thinking networks to enable extrapolation. (3) ASH is close to patience-based early exiting in multi-exit networks. The paper should state precisely what DICE adds beyond these approaches and compare against them where applicable.

  2. Baselines appear weak, and the strongest relevant baselines are missing. On prefix sums, the endpoint model reaches only 11.7% at 64 bits. DT-Recall with progressive loss (Bansal et al., 2022) and the Lipschitz-constrained variant (Bear et al., 2024), both cited, are reported to extrapolate well beyond 32-bit training lengths. Neither is compared. The RL baselines (GRPO, LSRL, RLTT) were designed for LLM post-training. Their initialization, configuration, and compute are not described. Comparing a dense supervised cross-entropy signal against sparse RL rewards is not a like-for-like comparison.

  3. The gradient diagnostic (Sec. 4.5) does not measure the claimed pathology. g_k is the gradient of a readout loss that is never optimized under endpoint training, so its misalignment with g_K is expected. The unnormalized variance comparison is also confounded by DICE’s much smaller per-step losses.

  4. The DEQ experiment is internally inconsistent. Sec. B.2 states that DEQ gradients flow through the implicit function theorem, yet attributes the collapse to amplification through 27 Jacobian products of the Broyden chain. In addition, 39.65% is called “near random,” but chance on CIFAR-10 is 10%.

Questions for authors
  1. How does DICE differ from TRM’s built-in outer-step deep supervision and from Bansal et al.’s progressive loss? Can you add DT-Recall + progressive loss and Bear et al. (2024) as baselines on prefix sums?

  2. Can you redo Sec. 4.5 by measuring the contribution of the endpoint loss to the gradient at iteration k (through J_{k→K}) under both regimes, and report normalized variance?

  3. Which bAbI tasks and splits are used in Tables 6 and 7? The endpoint UT baseline (89.72%, hidden size 128) seems low for bAbI 10k. How does it compare with the results reported by Dehghani et al. (2019)?

  4. How are the GRPO, LSRL, and RLTT baselines initialized and configured, and with what compute budget?

4 comments

Thank you for your constructive feedback. We clarify it below.

W1/Q1. Relation to TRM outer-step supervision, progressive loss, and early exiting.

  • TRM. The "Endpoint" row in Tab. 4–5 is stock TRM: a loss after each of the 16 outer steps with the carry detached, and the first H−1 cycles run without gradient. DICE changes two things: it adds the shared-head loss at each H-cycle, and it backpropagates through all H-cycles. To separate them we ran the missing control: full gradient through all H-cycles and endpoint-only loss, with TRM's outer-step supervision unchanged.
TRM, maze 30×30 (batch 64, 1.5k epochs)Exact match
Stock TRM0.580
+ gradient through all H-cycles, endpoint loss only0.743
+ shared-head loss at each H-cycle (DICE)0.862

Full gradient propagation accounts for 16.3 pp of the improvement, while adding per-cycle dense supervision yields another 11.9 pp under the same gradient configuration.

We will rename the row "TRM (outer-step deep supervision)" and add this control.

  • Progressive loss. It supervises one random iterate, after a detached prefix. DICE supervises every iterate with gradient through the whole chain. We now compare against it directly (W2).
  • ASH. It is patience-based early exiting (PABEE-style) applied across iterations of one shared map. What DICE adds: (1) The readout is not a set of separate classifiers but the same g. (2) Using the exits for halting beyond the training horizon (K_test > K_train). A multi-exit network cannot run past its last layer, but a looped model can. We use intermediate supervision to make extra test-time iterations useful, and ASH to decide when to stop in that regime.

Q1 / W1 / W2 — DT-Recall + progressive loss (Bansal et al. 2022) and DT-L (Bear et al. 2024) on prefix sums

The models were trained on 32-bit inputs with 30 iterations. 3 seeds, final-epoch checkpoint, 10k test instances per width, exact match (%).

(Width 256, batch 64, 80 epochs).

Objective64 @40 it72 @40128 @40256 @40256, best ≤512 it512, best ≤512 it512 @512 it
Endpoint4.9±4.12.3±2.00.00.00.00.00.0
Progressive loss, α=1 (Bansal)99.9±0.199.9±0.196.4±4.96.2±8.748.0±40.030.3±41.228.3±38.4
DT-L (Bear et al.)85.2±2.640.7±17.20.00.087.3±17.977.1±32.477.1±32.4
Looped-TF window loss (Yang et al. 2024)82.2±23.771.3±34.32.0±2.70.011.7±15.30.00.0
Exit-gate loss (Zhu et al. 2025)60.8±43.235.8±36.829.2±40.50.6±0.932.7±45.230.9±43.730.9±43.6
DICE, linear w_k (paper setting)99.7±0.299.8±0.2100.0±0.011.8±4.595.0±6.964.9±42.331.3±44.0
DICE, uniform w_k86.1±11.656.2±31.333.3±47.16.4±9.099.5±0.496.2±4.790.0±13.5

128 @40: accuracy on 128-bit inputs when the network runs exactly 40 iterations

512, best ≤512 it: accuracy on 512-bit inputs, taking the best value reached at any iteration count from 1 to 512. We run each model for up to 512 iterations. We then compute exact-match accuracy over the whole test set after every iteration, giving one accuracy per iteration. We pick the single iteration t* ≤ 512 with the highest test accuracy, and every instance is scored at t*. This selection is applied identically to all methods; the "512 @512 it" column involves no selection.

Progressive loss is a strong baseline and matches DICE at 64–72 bits. DICE (linear) is best at 128 bits with 40 iterations, and DICE (uniform) is best when more test-time iterations are allowed (256/512 bits: 99.5/96.2 vs. 48.0/30.3 for progressive loss and 87.3/77.1 for DT-L).

Q2 / W3 — Contribution of the endpoint loss to the gradient at iteration k through J_{k→K}, normalised variance

Setup. Deep Thinking, prefix sums, K=30. Median over k in the band, mean over 3 seeds. The network applies the same weights θ at every one of its 30 iterations. For the diagnostic, each iteration gets its own copy θ_k (all copies hold identical values), so the gradient can be split by iteration.

c_k is the gradient of the endpoint loss for one example with respect to θ_k. Because the loss is only at iteration 30, this is exactly the signal that has to travel back from the output through iterations k+1…30 (the Jacobian chain J_{k→K}) to reach iteration k.

m_k is the average of c_k over the 256 examples: the "signal" part of that contribution.

Normalised variance is E‖c_k − m_k‖² / ‖m_k‖²: how much individual examples scatter around that mean, relative to the size of the mean. A value of 200 means the per-example noise is 200 times the squared signal. (higher = noisier).

Total gradient G is the sum of m_k over all 30 iterations, which is the ordinary gradient of the endpoint loss with shared weights. The cosine column compares m_k with G.

Trained withepochk=1–10k=11–20k=21–29k=30cos(m_k, total grad), k≤10
Endpoint5203.7181.185.07.6−0.13
DICE596.774.516.513.40.21
Endpoint20165.294.66.52.30.05
DICE204.13.74.32.90.38
Endpoint75151.3167.564.313.6−0.02
DICE7567.047.820.710.80.23
Endpoint150 (final)166.4131.0122.6175.30.21
DICE150 (final)120.6163.0142.5111.30.39

Conclusion:

  • The long path is noisy under endpoint training. Early in training, noise grows with distance from the loss: 7.6 at the last iteration versus 203.7 for iterations 1–10 at epoch 5. The early-iteration signal is also essentially uncorrelated with the overall gradient direction (cosine near 0) until the end.
  • DICE can substantially improve gradient signal-to-noise ratios, particularly early and midway through training. For iterations 1–10 the normalised variance is about 2× lower at epoch 5, 40× lower at epoch 20 and 2× lower at epoch 75, and the cosine stays positive (0.21–0.39) throughout. However, the effects depend on training stage and iteration depth: at the final epoch the two regimes are comparable (DICE is lower at k=1–10 and k=30, higher at k=11–29).

Q3 — bAbI tasks/splits; comparison with Dehghani et al. (2019)

Tab. 6–7 use bAbI en-valid-10k, which takes the standard 10k training set and holds out 10% of it, giving 9,000 training, 1,000 validation and 1,000 test examples per task. We evaluate on tasks 1–5, one model per task.

Dehghani et al. (2019) use 10k, evaluate on 20 tasks, one model per task, and report the best seed per task. We were not able to reproduce their reported numbers with their released codebase.

So same benchmark and test set, 10% fewer training examples because of the held-out validation set, and a subset of five tasks. The new 20-task runs below remove the task-subset difference but keep the 9k/1k split.

New: all 20 tasks, per-task training, 5 seeds, test accuracy (%).

ConfigEndpointDICE
Mean over 20 tasks86.8289.14
Tasks 1–590.8094.06

Q4 — RL baseline initialisation, configuration, compute

InitSampling / rewardOptimiser, budget
Prefix sums, GRPOendpoint checkpoint (80 ep)1 sample at step K; exact match; batch-mean baselineAdam 1e-5, +20 ep (lr swept {3e-6, 1e-5, 3e-5})
Prefix sums, RLTTsame1 sample per step; exact match at K; log-probs summed over KAdam 1e-5, +20–30 ep
Prefix sums, LSRLsame1 sample per step; per-step position accuracy, linear w_k; std-normalisedAdam 1e-5, +20 ep
bAbI UT, GRPO / RLTT / LSRLendpoint checkpoint (≤100 ep, with early stop)1 sample, T=1; exact answer match (LSRL: per-step match vs ground truth)Adam 1e-4, +≤30 ep, clip 1.0
DICE / Endpointscratch—80 ep (prefix sums), ≤100 ep (bAbI)

Like-for-like supervised baselines (progressive loss, DT-L, window loss, exit-gate loss) are in the Q1 table.

W4 — DEQ

Thank you, the text is wrong. We will remove the Jacobian-chain explanation and the "near random" wording from Sec. B.2.

At epoch 13, the baseline DEQ run exhibits a transient training instability: test accuracy drops from ∼85% to 39.65% before recovering over the next few epochs. We did not observe a comparable drop in the intermediate-supervision runs.

Reviewer_7MKQOpenReview ↗

Feedback

Thank you to the authors for replies. These new experiments have resolved most of my concerns.

If the authors could clarify the motivation of this paper more clearly, I believe the quality of the paper would be higher.

I will reconsider my decision after reviewing the rebuttal.

Thank you for the follow-up, and we are glad the new experiments resolved most of your concerns. We clarify the motivation below.

Motivation. Iterative models are meant to improve by running more iterations. That requires (a) a useful learning signal at every iteration and (b) meaningful iterates when K_test > K_train. Endpoint supervision guarantees neither. On the other hand, each family has developed its own intermediate-supervision fix: progressive loss for Deep Thinking (Bansal et al., 2022), outer-step deep supervision in TRM, window loss for looped Transformers (Yang et al., 2024), and an exit-gate loss for looped LMs (Zhu et al., 2025). Each mixes several design choices, and they have not been compared with each other, so a practitioner cannot tell which choice matters for which failure. This is a real problem, because the methods rank differently across settings. For example, progressive loss is very strong on prefix sums at 64–128 bits but drops to 74.9% on bAbI (Endpoint: 86.6%).

What this paper does. We do not claim to be the first to supervise intermediate iterates. Our contribution is to (i) name one common cause, endpoint supervision, behind failures that earlier work treated as architecture-specific; (ii) give a minimal objective in which each component targets one failure mode: the dense loss shortens the gradient path for (a), and the shared readout g constrains the iterates for (b); (iii) test each component separately with controls.

Reviewer_7vrsOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an architecture-agnostic dense supervision approach to training recurrent architectures like transformers and deep thinking networks. By introducing a readout layer at each iteration of the model and computing a loss on it, the paper shows improved results on various reasoning tasks including maze solving. The same mechanism also endows the model with a halting switch that can essentially stop the iterations when little progress has been made between subsequent iterations.

Critical strengths

The proposed intervention can indeed help and it also does provide a valid mechanism so the central thesis of the paper is valid. The authors provide various ablations to support their method and the paper is clearly written (which I appreciate even more now nice given the deluge of AI-written submissions at this conference).

Critical weaknesses

The main issue I have with the contribution of the paper is that this particular approach of using a fixed readout at each iteration for additional supervision can be found in various papers in the literature on solving combinatorial and reasoning problems. I provide some examples below but the list is not exhaustive [1,2,3]. So while I don't have any particular disagreements with the method itself or with the fact that this can work, my issue is that I find the contribution rather limited since quite similart techniques have been used before in similar contexts. Obviously the paper provides new experiments in different settings but it seems to me that for the contribution to become more meaningful and add something new to the literature, a novel application or some new spin on those established techniques would be required. Perhaps a more exhaustive study on the various ways one can implement this strategy and additional theoretical results could help set this paper apart as the one that really digs into this issue and clarifies its benefits. Otherwise, the contribution is in my eyes a little too incremental.

I am curious what the authors view is the key contribution here and how it stands out from these previous works. I am open to reconsidering my evaluation but currently it doesn't seem like the paper offers enough new material for publication.

  1. Toenshoff, Jan, et al. "Graph neural networks for maximum constraint satisfaction." Frontiers in artificial intelligence 3 (2021): 580607.
  2. Ozolins, Emils, et al. "Goal-aware neural SAT solver." 2022 International joint conference on neural networks (IJCNN). IEEE, 2022.
  3. Palm, Rasmus, Ulrich Paquet, and Ole Winther. "Recurrent relational networks." Advances in neural information processing systems 31 (2018).
Questions for authors

see above

2 comments

Thank you for identifying the relevant prior work.

“The key contribution here and how it stands out from these previous works”

We agree that supervising intermediate iterates through a shared readout is not new, and we will revise our novelty claims and related work accordingly.

Our main contribution is not the introduction of intermediate supervision itself, but a systematic investigation of how its implementation affects optimization and test-time extrapolation in iterative neural computation.

In particular, we contribute three things:

  1. A controlled comparison of supervision strategies. We compare endpoint supervision, progressive loss (Bansal et al.), window loss (Yang et al.), learned exit weighting (Zhu et al.), detached intermediate supervision, and DICE, using the same underlying model. The results show that intermediate supervision alone does not guarantee successful extrapolation. On 512-bit prefix sums, for example, uniform DICE achieves 96.2% best accuracy within 512 iterations, compared with 30.3% for progressive loss (α=1), 30.9% for learned exit weighting, and 0.8% for detached DICE. Other competitive variants are reported in our extended comparison.
  2. An analysis of the mechanisms and design choices. Our ablations distinguish the effects of intermediate gradient signals and shared-readout supervision. We additionally find that loss weighting substantially changes extrapolation behavior: linear weighting achieves 100% accuracy at 128 bits and 40 iterations but 64.9% best accuracy at 512 bits, whereas uniform weighting achieves 33.3% and 96.2%, respectively. This reveals an important trade-off between short-horizon performance and long-horizon generalization.
  3. Evidence across different iterative architectures and tasks. Beyond algorithmic extrapolation, we demonstrate improvements on recursive Transformers, Universal Transformers, ARC-AGI, and a 3.5B-parameter looped language model. These experiments evaluate the broader applicability and limitations of dense intermediate supervision rather than treating it as an architecture-specific technique. We believe the main value of this work is therefore to move beyond demonstrating that intermediate supervision can help, toward characterizing which supervision designs are effective, why they help, and when their benefits persist across architectures and inference horizons.

In addition, we find that in large looped language models, dense supervision improves early-exit performance while approximately preserving full-depth performance in-domain. We also report an ablation in the looped language-model setting below.

Regarding the cited works, RRN’s objective is a special case of DICE: a uniform schedule and α = K−1. RUN-CSP and QuerySAT replace the label with a label-free objective defined by the instance itself, and need extra gradient control. Up to this choice of per-step loss, each corresponds to one fixed (α, w) setting of Eq. (3) within a single architecture. DICE generalizes to arbitrary INCs.

"A more exhaustive study on the various ways one can implement this strategy"

We evaluate all methods on prefix sums using the same network, 3 seeds, changing only the objective. We report exact-match accuracy (%) on 512-bit inputs (16× the training length), showing both the best accuracy within 512 iterations and the accuracy at iteration 512.

Design choiceObjective128 bits @40 it512 bits, best ≤512512 bits @512
No intermediate lossEndpoint0.00.00.0
One random intermediate state per stepProgressive loss (Bansal)96.4±4.930.3±41.228.3±38.4
Random window of T states, prefix detachedWindow loss (Yang)2.0±2.70.00.0
All states, learned exit weightsExit-gate loss (Zhu)29.2±40.530.9±43.730.9±43.6
All states, fixed weights, no backprop across stepsDICE-detach72.8±19.00.8±0.70.1±0.2
All states, fixed linear weights, full backpropDICE linear100.0±0.064.9±42.331.3±44.0
All states, fixed uniform weights, full backpropDICE uniform33.3±47.196.2±4.790.0±13.5

Language-modelling task

We re-ran the four-way ablation on a language-modelling task: continued pretraining of Huginn (3.5B recurrent-depth LM) on wikitext-103 at K=8. Each mode was run with 3 seeds, varying the parameter initialization, data order, and the latent initialization.

Two design choices make the controls more informative. For final-repeated, each extra loss term uses a fresh batch, so it measures the effect of additional supervision rather than a duplicated loss on the same minibatch. For separate heads, each g_k is an independent copy of the head transform initialized from the pretrained head and free to diverge during training, vocabulary projection shared. At initialization, this condition is therefore identical to DICE. All four are evaluated in the same way, decoded by the model's own shared head g at k = 1.

ModeCE@1 ↓acc@1 ↑
(1) Endpoint4.078 ± 0.0060.288 ± 0.001
(2) Final-repeated (loss scale only)4.066 ± 0.0050.290 ± 0.001
(3) Separate heads (short gradients only)3.721 ± 0.0030.329 ± 0.000
(4) Shared head (full DICE)3.646 ± 0.0030.341 ± 0.000

The total CE gain (Δ = 0.432) can be decomposed into three incremental contributions:

  • additional supervision: ~3% (Δ=0.012). Adding the same extra loss weight on the final state, using fresh batches, gives almost no gain. In this setting, the improvement does not come from more endpoint training or a larger effective loss scale. On bAbI (Table 7), the corresponding control accounts for a larger share of the gain, and we will discuss this difference in the revision.
  • shorter gradient path: ~80% (Δ=0.345). Supervising intermediate states, even through separate heads, produces most of the gain.
  • shared readout: ~17% (Δ=0.075). Decoding every state through the same head g, the head actually used at eval, is a real, separable benefit on top of the short path.

At full depth (k=32), in-domain CE is 0.014 higher than with endpoint training. In this setting, the benefit of dense supervision is therefore concentrated in early exit rather than full-depth. We will note this in the Limitations section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.