- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper considers how the placement of intermediate supervision affects neural algorithm learning under a fixed label budget. To do this, the authors introduce a synthetic contextual permutation-composition task and compare two recurrent state interfaces. One is native hidden-state feedback, and the other is probability-based “soft” feedback. Each sparsely supervised example receives exactly two intermediate labels plus the final label, but the main intervention either allows or excludes supervision at the first transition.
The authors find that this placement choice has a very large effect. Namely, at 8,000 updates, retaining supervision on the first step improves 8-step endpoint accuracy by +87.4 pp with soft feedback and +55.0 pp with native feedback. They also find a large difference in long-horizon behavior. Successful soft-feedback models achieve 100% complete-trajectory accuracy on 32-step rollouts, while the native-feedback models fail to generalize comparably despite strong 8-step performance.
The paper further studies optimization-budget effects and a gradient-cut intervention, arguing that supervision placement, the carried state representation, and optimization budget should be considered jointly when evaluating algorithm learning.
Critical strengths
-
The experimental design is clean and carefully controlled, which allows the authors to isolate where supervision is placed rather than how much supervision is given.
-
The reported effects are large and consistent enough to be convincing, at least within the paper's synthetic setting (retaining first-step supervision improves 8-step endpoint accuracy by +87.4 pp with soft feedback and +55.0 pp with native feedback).
-
The longer-rollout evaluation is useful. It shows that strong performance at the training horizon does not necessarily mean the model has learned a stable computation that can be reused for many more steps.
Critical weaknesses
-
The biggest concern I have is that the main finding is very narrow. The paper studies one synthetic permutation-composition task, one specific recurrent architecture, and one sparse supervision setting. It is not clear at all whether the very large first-step supervision effect would hold on other algorithm-learning tasks, architectures, or realistic process-supervision settings. The authors also explicitly avoid making claims about pretrained language models.
-
Could the authors highlight what is the broader takeaway here? Step 1 is structurally special in this setup because it starts from a provided state, while later steps operate on model-generated states. Given this asymmetry, it is perhaps not surprising that supervising the first transition can matter more. The paper does not give a more general principle for where supervision should be placed beyond this specific setting. For these reasons, I am not convinced about the practical significance of the results.
-
The mechanism behind the long-horizon native/soft gap is still unresolved. The longer-rollout result is interesting, but the paper mainly establishes the behavioral difference rather than explaining why soft feedback generalizes while native feedback does not. The authors themselves note that direct interventions on the carried states would be needed to identify the cause.
Taking everything into account, the paper seems to be a well-designed study around one specific synthetic task, but in my opinion, the findings are not general/significant enough to meet the bar for ICLR.
Questions for authors
Refer to the Critical Weaknesses above. In particular, I would be interested in the authors' view on the broader significance and generality of the first-step supervision result beyond the specific synthetic setup studied here.
Response on generality, supervision placement, and recurrent reuse
Your questions led us to test three aspects of the contribution: whether the placement effect extends to another task and architecture, whether useful supervision can move away from step 1, and whether feedback itself causes the long-horizon gap. Section and table references below refer to the revised manuscript.
1. Placement effects beyond the original task and architecture. We added a table-free task combining addition modulo 16 and bitwise XOR, learned by a 79,184-parameter GRU. Across eight paired data blocks, the retain/remove conditions share initial weights, examples, sample order, two intermediate labels plus the endpoint label, and 24,000 updates. All eight fully supervised controls exceed 95% complete-trajectory accuracy on both training and development sequences.
Retain/remove eight-step endpoint accuracy is 97.50%/9.07%, a difference of 88.44 percentage points [85.14, 91.43] (descriptive 95% block-bootstrap interval). Every paired block favors retain (Sec. 5.9, Table 6; Appendix N). Thus, the placement effect also occurs without supplied transition tables and with a different recurrent architecture. Every retain model learns all 512 local transitions correctly; 32-step complete-trajectory accuracy averages 76.34%, with a 28.27-97.44% range across blocks. This separates acquiring the local rule from reliably reusing it over longer sequences.
2. A testable principle for where supervision is useful. Your observation that step 1 starts from a provided state suggests a prediction: moving a supplied-state boundary should change which label helps the learner acquire a reusable transition. We test this by supplying an independent random state's embedding immediately before transition 3 or 5, with the task reset at the same boundary. The sparse targets are {L,7,8}, with L=3 or 5; data, initialization, label count and updates are paired.
Evaluation removes the reset and tests the original 32-step task. After reset-before-3 training, labels at 3/5 yield 100%/0.31% complete trajectories; after reset-before-5 training, they yield 0.41%/100%. The preference reverses in every block, and all aligned models reach 100% local-transition accuracy (Sec. 5.8, Table 5; Appendix L).
The original first-step label also remains effective. The new boundary therefore creates another useful supervision location. Without a reset, the label-5 condition yields 0.006% complete trajectories, or 0.015% with a gradient cut after output 4. The reset changes both the supplied input and dependency length; gradient truncation alone does not reproduce its effect. The resulting design principle is to identify labels that teach a transition from an available input state, then evaluate whether that transition transfers to other positions and longer executions.
3. Direct interventions on recurrent feedback. We hold the main models' weights and first-eight-step outputs fixed, then change only later feedback. Native retain models' 32-step complete trajectories rise from 0.26% to 93.49% with soft feedback and 94.95% with hard predicted-symbol feedback. A norm-only intervention gives 7.36%; a norm-matched wrong codebook gives 0%. The reverse intervention reduces trained soft retain models from 100% to 0.90% (Sec. 5.3, Fig. 3; Appendix J.1/K.1).
The GRU provides a second fixed-weight test: at 8,000 updates, re-encoding its own predicted symbols raises 32-step complete trajectories from 14.63% to 94.89%, without retraining or oracle feedback (Appendix N). Together, these interventions show that the feedback mapping directly controls whether learned transitions remain usable during execution.
Native feedback can also succeed. With two post-residual LayerNorms, native retain models reach 99.75% on 32-step complete trajectories, while native/soft placement effects remain 85.56/87.14 points (Sec. 5.7, Table 4). The placement effect therefore persists when native long-horizon execution succeeds.
These experiments connect supervision placement, state interfaces and training budgets through a concrete distinction: learning a correct transition and repeatedly using that transition require different evidence. The contribution is a tested account of this distinction across two symbolic task/model settings, with movable supervision opportunities and direct feedback interventions.