acceptodds
Under review as a conference paper at ICLR 2027

Auxiliary supervision shapes optimization geometry for recurrent sequence models

Abstract

Recurrent sequence-to-vector models are widely used for tasks such as time series forecasting and sequence classification or regression, where supervision is often applied only after the complete input sequence has been processed. This can create a temporal parameter-information bottleneck, leaving important parameter directions weakly informed and thereby causing poor conditioning, potentially slow and numerically sensitive training, and undesirable rollout behavior on unseen sequences. We show that auxiliary supervision can alleviate this bottleneck by exposing parameter information at intermediate positions before it becomes invisible or is attenuated by the recurrent dynamics. Our sensitivity-based analysis establishes that suitably chosen auxiliary observations can improve local parameter observability and prevent deterioration of generalized Gauss–Newton conditioning with increasing sequence length, thereby supporting faster and more numerically robust training. At the same time, auxiliary supervision acts as an implicit regularizer that can promote desired intermediate or rollout behavior. Moreover, we characterize the resulting primary-task performance trade-off and show that compatible auxiliary objectives need not sacrifice globally optimal primary-task performance. Analytical results and extensive experiments with modern Mamba models on the standard ListOps sequence-classification benchmark corroborate the identified mechanisms and demonstrate the practical benefits of appropriately chosen auxiliary supervision for training efficiency and primary-task prediction accuracy.

Primary area:
learning theory
Keywords:
Auxiliary supervision, recurrent sequence models, state space models, sequence learning, parameter observability, generalized Gauss--Newton methods, optimization conditioning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_d4P4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is motivated based on the phenonmenon where in recurrent sequence model, due to terminal supervision, this caused parameter-information bottleneck to form, where intermediate parameter direction are diminished. They theoretically show that auxiliary loss applied on intermediate readout could mitigate the bottleneck by exposing weakly observed parameter direction. Finally, they show the primary-auxiliary trade-off and demonstrate empirically that auxiliary loss allow for faster and more stable training on long range dependency task.

Critical strengths
  1. This paper is well motivated on diminishing effect of intermediate parameter direction as sequence length increase. They clearly show that how auxiliary loss will help in exposing the parameter direction under reasonable assumption.

  2. The empirical result on ListOps is significant both in term of demonstrating terminal observation only weakly representing parameter direction and also the effect of auxiliary loss in aiding the convergence and conditioning of the training.

I find this paper while mathematically dense, it is clearly written and accompanied with intuitive explanation.

Critical weaknesses
  1. ListOps dataset have semantically meaningful subexpression groundtruth that is strongly correlated with the terminal groundtruth, this might be highly favourable to the proposed method. In practice however, such intermediate annotation are rare. The paper only discuss supervised auxiliary loss, prior work like [1] and [2] also utilized unsupervised auxiliary loss ([2] use auxiliary loss more like a regularizer). As Proposition 2 relies on complementary auxiliary loss, how often and stable we can obtain with unsupervised or self-supervise loss that satisfies the condition is a key to its practicability.

  2. The empirical experiment scope for this paper is rather narrow, only 1 dataset and 1 model. It is important to at least repeat across 1 or 2 more datasets to solidify the claim. It would be even better to test on different recurrent neural network like original Mamba or xLSTM [3]. All training are done on N=72N=72, is it possible to have result at different NN?

[1] Trinh, Trieu, et al. "Learning Longer-term Dependencies in RNNs with Auxiliary Losses." International Conference on Machine Learning. PMLR, 2018.

[2] Kag, Anil, and Venkatesh Saligrama. "Training recurrent neural networks via forward propagation through time." International Conference on Machine Learning. PMLR, 2021.

[3] Beck, Maximilian, et al. "xlstm: Extended long short-term memory." Advances in Neural Information Processing Systems 37 (2024): 107547-107603.

Questions for authors
  1. Is it possible that the author could have a simple experiment showing that the proposed framework, especially Proposition 2 will hold under unsupervised on semi-supervised auxiliary loss?

  2. While the rebuttal period is rather tight, is it possible that the author could show result on 1 or 2 more datasets? Showing that the result in Figure 2. will hold on other datasets will significantly make the claim stronger.

  3. How stable is the proposed method under different NN? Does λ\lambda requires huge changes as NN changes? (Minor)

I am happy to raise my score if my concerns are resolved sufficiently (not necessarily all of them but especially 1. and 2. to a acceptable extend).

6 comments
AuthorsOpenReview ↗

Response to Reviewer d4P4

Thank you for the careful review and for suggesting these experiments. The revised manuscript adds self-supervised forecasting on two real-world datasets with xLSTM, a conditioning analysis, and results across three lookbacks.

Weakness 1 and Question 1 — Self-supervised auxiliary targets and complementary curvature

We agree that it is important to test whether auxiliary targets derived from the data can supply curvature in directions that are weakly observed under terminal supervision. To test this, we added two forecasting experiments with self-supervised auxiliary targets. At selected intermediate positions, we reuse the terminal forecast readout and use the next 96 observations as targets. Both the primary and auxiliary losses use mean squared error.

At a lookback of 192, we compare the terminal and combined GGN matrices at the same weights, on the same minibatch, and on the same retained terminal subspace. Adding the self-supervised loss improves projected conditioning in all 18 forecasting comparisons across two datasets, three seeds, and three checkpoints (Section 3.3 and Figure 3). In each of these comparisons, the weighted auxiliary term also contributes more curvature than the terminal term along every direction in the span of the five weakest retained terminal directions (Appendix C.3 and Table 5). This provides direct empirical evidence for the complementary-curvature mechanism in Proposition 2 with self-supervised targets.

Weakness 2 and Question 2 — Additional datasets, xLSTM, and the Figure 2 analyses

We agree that evaluating the framework on additional datasets and with a different recurrent architecture strengthens the empirical evidence. Following your suggestion, we use causal xLSTM in the new univariate forecasting experiments on one hourly series from each of the Electricity and Traffic benchmarks (Section 3.2 and Appendix C.2). On both datasets, we reproduce the qualitative patterns from Figure 2. Sensitivity retention decreases with distance to the terminal readout, and intermediate readouts increase the number of singular values above the 1% cutoff relative to the row-matched terminal control (Figures 6 and 7). At a lookback of 192, the 12-hour schedule reduces mean updates to the primary reference loss by 15.8% on Electricity and 11.1% on Traffic (Table 1 and Figure 8). It also lowers both test MSE and MAE for all three paired seeds on each dataset (Table 1).

To assess the effect of lookback length, we trained separate forecasting models at N=96N=96, 192192, and 336336. With the 12-hour auxiliary schedule, both test errors improve at all three lookbacks on both datasets. The reduction in mean updates to the reference loss is larger at longer lookbacks on both datasets (Table 4), consistent with the motivation for intermediate supervision when sensitivity from earlier positions must propagate farther to reach the terminal readout. We also report a length-generalization test on ListOps, evaluating models trained at N=72N=72 on held-out sequences of length N=288N=288 (Table 3).

Question 3 — Lookback length and auxiliary-loss weight

We agree that it is important to understand whether the auxiliary-loss weight needs to change with input length. In the forecasting experiments, we used the same weight, λ=0.3\lambda=0.3, at lookbacks of N=96N=96, 192192, and 336336. With the 12-hour schedule, both test errors decrease in all six dataset–lookback combinations, and the mean number of updates to reach the reference loss decreases in five of six. On Traffic at N=96N=96, the mean update count increases by 2.7% (Table 4). These results indicate that the auxiliary-loss weight can be kept fixed as the lookback varies across the tested range.

The ListOps ablation examines sensitivity to λ\lambda at the fixed training length N=72N=72. Moderate weights give the strongest results, and the largest weight requires more updates to reach the reference loss than terminal-only training (Figure 5, panel A). This illustrates why the auxiliary loss should have enough weight to contribute to learning while preserving progress on the primary task.

Thank you very much for the detailed response. My concerns in 2. have been partially addressed and 3. have been substantially addressed.

Below are my remaining concerns:

  1. I thank the authors for the effort in adding two additional datasets and showing that the proposed method is effective with data-derived auxiliary losses. However, this does not fully address my concern, as the intermediate objective in the Electricity and Traffic forecasting benchmarks is essentially the same as the terminal forecasting objective, only applied at different positions. This still limits the evidence for the applicability of the proposed method to more general datasets, for example, sequence classification, where such intermediate targets are not naturally available. For example, [1] utilized MNIST, pMNIST, and CIFAR-10 for sequence classification and showed that auxiliary reconstruction losses can still be effective, even when the auxiliary objective differs from the terminal classification objective.

  2. The selected additional benchmarks still greatly favour the proposed method, as mentioned in 1. Since the paper originally utilized ListOps from the Long Range Arena benchmark, why not consider other datasets from the same benchmark, such as Image (CIFAR-10) or Text (IMDb)? I believe that showing the proposed method is effective on these datasets, where the auxiliary objective differs from the terminal classification objective, would significantly strengthen the claim of general applicability.

[1] Trinh, Trieu, et al. "Learning Longer-term Dependencies in RNNs with Auxiliary Losses." International Conference on Machine Learning. PMLR, 2018.

Thank you for clarifying your remaining concerns and for suggesting this experiment. Following your suggestion, we have added sequential CIFAR-10 classification with unsupervised auxiliary reconstruction, using the Long Range Arena image setting and following the LSTM architecture and the auxiliary reconstruction loss design proposed by Trinh et al. (2018). Thus, the auxiliary objective differs from terminal classification and requires neither intermediate class labels nor additional annotations. In the revised manuscript, we describe the setup in Appendix C.3 and incorporate the results into Section 3.3.

The new experiment supports the mechanisms studied in our paper:

  • Dynamical forgetting: In the trained terminal-only classifier, median sensitivity retention decreases from 79% in the nearest distance bin to 12% in the farthest bin, consistent with the attenuation mechanism in Theorem 1 (new Figure 9 in the revised manuscript).
  • Complementary curvature: Across three seeds, we evaluate the shared initialization and two checkpoints from training with auxiliary supervision. In each comparison, we evaluate the terminal GGN matrix GpG^{\mathrm p} and the combined GGN matrix GλG_\lambda at the same weights, on the same minibatch, and on the same retained terminal subspace. Adding reconstruction improves projected conditioning in all nine comparisons (see the revised Figure 3). The smallest projected eigenvalue increases by a larger factor than the largest in every comparison (Table 5 and Appendix C.4).

At the common 100-epoch budget, mean test accuracy is essentially unchanged (56.41% with terminal-only training and 56.44% with auxiliary supervision), and we find no consistent reduction in updates to the reference loss. Our shorter training schedule may partly explain the difference in accuracy outcomes compared with Trinh et al. (2018), who use auxiliary pretraining followed by 1,000 epochs of joint training. These results distinguish the local curvature improvement from its eventual effect on optimization and prediction.

The added experiment therefore extends the empirical evidence to precisely the setting you highlighted: sequence classification with an unsupervised auxiliary objective, distinct from the primary classification objective. It illustrates the dynamical-forgetting mechanism in Theorem 1 and the complementary-curvature mechanism in Proposition 2, two central theoretical contributions of our paper. Together with the ListOps and forecasting experiments, these results support the applicability of these mechanisms across different recurrent architectures and forms of auxiliary supervision.

We would like to thank the reviewer again for their constructive feedback throughout the discussion, which we believe has substantially strengthened the paper.

Reviewer_fMfdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper discusses a common setting of RNN where it only has one terminal supervision. The author does an interesting analysis that this basic setting suffers from the parameter sensitively degradation problem faced with longer sequence length in exponential speed, while theoretically proves that with auxiliary supervision, this degradation can be alleviated and mitigated if these auxiliary supervision is able to complement the weak subspace of the original terminal supervision. The experiments on Mamba and one synthetic task demonstrated improved performance.

Critical strengths

It provides a solid theoretical interpretation of why auxiliary supervision to train RNN is meaningful, from the perspective of better optimization landscape. The theory is sound and novel.

Critical weaknesses

The experiments are generally weak. There is only one synthetic dataset. AFAIK this dataset naturally has some structured intermediate signals that can be leveraged. Even though a deeper analysis related to readout sharing, singular-value spectrum, and other optimization landscape related metrics are deep-dived, it remains a question that whether new practical insights can be given, of designing auxiliary supervision if there isn't a pre-existed intermediate rewards structure.

Questions for authors

Do you think the theory can hold to the RL setting where there is only one terminal reward and the modern SSM models are used as backbone?

2 comments
AuthorsOpenReview ↗

Response to Reviewer fMfd

Thank you for the careful review, for highlighting the theoretical contribution, and for your overall positive assessment. We agree that the paper benefits from a broader empirical evaluation and a concrete example of how to design auxiliary supervision without intermediate labels. In the revision, we add real-world forecasting experiments with causal xLSTM and self-supervised auxiliary targets, together with a conditioning analysis across ListOps and both forecasting datasets.

Weakness 1 — Empirical scope and practical design of auxiliary supervision

We agree that it is important to show how to design auxiliary supervision for tasks without intermediate labels. In the revised manuscript, we therefore add real-world forecasting experiments with causal xLSTM on one hourly series from each of the Electricity and Traffic benchmarks (Section 3.2 and Appendix C.2). At selected intermediate positions, we reuse the terminal forecast readout, with the next 96 observations in the series serving as self-supervised targets.

With the 12-hour readout schedule and a lookback of 192, the models reach the primary reference loss in 15.8% fewer updates on Electricity and 11.1% fewer on Traffic, on average, relative to terminal-only training. Both test MSE and MAE improve for all three paired seeds on each dataset at the same update budget (Table 1). The self-supervised auxiliary loss also improves projected conditioning at matched weights on both datasets (Section 3.3 and Figure 3).

The experiments also show that placement matters for both training progress and prediction accuracy. On Traffic at a lookback of 192, the 24-hour schedule reaches the reference loss sooner on average and increases both test errors relative to terminal-only training. The 12-hour schedule reduces both the update count and the test errors (Table 4).

We agree that the choice of auxiliary target is task-dependent. In our ListOps ablation, repeating the terminal label at intermediate positions requires more updates on average than terminal-only training (Table 3). This is consistent with prior work showing that poorly chosen auxiliary losses can interfere with primary-task learning; see, e.g., Du et al. (2018).

Together, the ListOps and Traffic results motivate evaluating candidate targets and readout schedules on both held-out primary-task performance and the number of updates to reach a reference loss.

Question 1 — Reinforcement learning with terminal rewards

Thank you for this question. We agree that RL with a single terminal reward is a promising setting for auxiliary supervision. For example, Lu et al. (2023, Section 4.1) use S5 as the recurrent backbone of an actor–critic agent on the bsuite memory-length task. The agent sees an initial cue and receives its only reward when it makes the final decision. In this setting, an auxiliary readout could predict the initial cue from the S5 hidden state at selected intermediate positions. The cue is available in the recorded observations, so this prediction loss can train the shared backbone while the environment reward remains terminal.

On a recorded trajectory, our framework can quantify how much curvature the auxiliary cue predictor adds along parameter directions that are weakly observed at the final readout. This gives a way to study the same mechanism in a recurrent policy trained with a terminal reward. Extending Proposition 2 to the full actor–critic objective would also account for the policy and value losses at multiple positions and for how policy updates can change the actions and observations in future trajectories.

The same fixed-input sensitivity analysis applies when training an SSM world model on recorded observation–action sequences. Wang et al. (2025, Section 2.2), for example, use Mamba-2 as the sequence backbone of a learned world model. If a world model is trained with a terminal prediction loss and auxiliary prediction losses at intermediate readouts, its objective has the structure studied in our paper. Proposition 2 applies to that prediction objective under its stated assumptions.

We thank you for raising this interesting question. We see extending our framework to reinforcement learning with terminal rewards as a promising direction for future work.

AuthorsOpenReview ↗

Follow-up on our response

Since our initial response, we have added sequential CIFAR-10 classification with an LSTM and unsupervised auxiliary reconstruction. Addressing your concern about designing auxiliary supervision without intermediate labels, a separate decoder reconstructs previously observed pixels from an intermediate recurrent state, using the input itself as the supervision signal.

The experiment provides further evidence for the mechanisms in Theorem 1 and Proposition 2 (Section 3.3 and Appendices C.3–C.4). The latest additions are highlighted in red.

We would be grateful to hear your thoughts on the revised manuscript and whether any concerns remain. Thank you again for your time and helpful suggestions.

Reviewer_xuGbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how auxiliary supervision can improve the training of recurrent sequence models. This paper argues that terminal-only supervision may lose useful parameter information as it propagates through recurrent dynamics, leading to poor optimization geometry. They develop a sensitivity-based theoretical framework showing that intermediate auxiliary supervision can improve parameter observability and conditioning, while potentially preserving primary-task performance. Experiments on Mamba models provide empirical support for these claims.

Critical strengths
  1. The paper provides a clear theoretical perspective on why auxiliary supervision can benefit recurrent models.

  2. The analysis connects parameter observability, recurrent information decay, and optimization conditioning in a coherent way.

  3. The experiments on Mamba are well aligned with the theoretical claims and provide useful empirical support.

Critical weaknesses
  1. The empirical results do not directly establish that improved optimization conditioning is responsible for the observed training acceleration.

  2. The theoretical guarantees rely on restrictive assumptions whose validity for practical recurrent models, including Mamba, remains unclear.

  3. The performance gains may largely arise from the additional information provided by intermediate labels, rather than the proposed optimization-geometry mechanism.

Questions for authors
  1. The theoretical results rely on strong assumptions, such as exponential stability and the absence of subsequent re-excitation. Although the experiments show improvements, they do not directly establish the connection between these improvements and the proposed theoretical mechanisms. I suggest that the authors provide stronger empirical evidence to bridge this gap.

  2. The paper cites several related studies on auxiliary supervision and optimization geometry, but the theoretical novelty over these works is not sufficiently clear. Could the authors further clarify what new insights the proposed framework provides beyond existing analyses?

    References:

    • Awasthi et al. (2025). Length Generalization via Auxiliary Tasks. NeurIPS.
    • Zhao et al. (2024). Theoretical Characterisation of the Gauss-Newton Conditioning in Neural Networks. NeurIPS.
    • Zhang et al. (2026). NITP: Next Implicit Token Prediction for LLM Pre-training. ICML.
  3. The experiments mainly focus on Mamba-2 and the synthetic ListOps benchmark, where intermediate supervision targets are readily available. I suggest that the authors provide additional experiments on more realistic tasks or different recurrent architectures to demonstrate the generality of the proposed framework.

5 comments
AuthorsOpenReview ↗

Response to Reviewer xuGb (1/3) — Weaknesses 1–2

Thank you for the careful review and for highlighting the connection between parameter observability, dynamical forgetting, and conditioning. We agree that more direct empirical evidence for these mechanisms would strengthen the paper. In the revision, we add real-world forecasting experiments with causal xLSTM and self-supervised auxiliary targets, compare conditioning at fixed parameters across all three datasets, and clarify how our analysis relates to prior work.

Weakness 1 — Conditioning and optimization speed.

We agree that it is important to understand what role improved conditioning plays in the observed reduction in training updates. In the new Section 3.3, we isolate the effect of adding the auxiliary loss on local curvature by comparing the terminal and combined GGN matrices at identical weights, on the same training minibatch, and on the same retained terminal subspace. For each of three seeds per dataset, we evaluate the shared initialization and two checkpoints from terminal-only training.

Adding the auxiliary loss lowers the projected condition number in all 27 comparisons across ListOps, Electricity, and Traffic (Figure 3). In every comparison, the smallest projected eigenvalue increases by a larger factor than the largest. In these same comparisons, the weighted auxiliary term supplies more curvature than the terminal term along every direction in the span of the five weakest retained terminal directions (Table 5). This directly demonstrates the complementary curvature mechanism described in Proposition 2.

The lower projected condition number is favorable for optimization because it reduces the disparity in local curvature. However, in stochastic nonlinear optimization, the number of updates needed to reach a reference loss depends on several interacting factors, including how curvature evolves along the training trajectory and how the optimizer responds to the gradients it encounters. The choice of auxiliary target is another important factor because it shapes the gradient signal at intermediate positions. In the ListOps ablation discussed under Weakness 3, the two auxiliary objectives give identical projected conditioning at matched weights and reach the same primary reference loss after substantially different numbers of updates.

Weakness 2 — Scope of the theoretical assumptions.

We agree that it is important to understand how the assumptions of Theorem 1 relate to trained recurrent models. Sensitivity attenuation over long time spans is a common practical issue in recurrent network training and is closely related to the vanishing-gradient phenomenon. In Theorem 1, exponential stability controls how sensitivity already present at an intermediate position propagates to the terminal readout. The no-re-excitation condition excludes later contributions along the particular parameter direction considered in the theorem. As discussed after Theorem 1, this condition can be relaxed when subsequent excitation along that direction decays sufficiently fast. Together with the other stated assumptions, these conditions yield an exponential bound on terminal curvature along that direction.

To assess whether the predicted attenuation occurs in practice, we track sensitivity already accumulated at an intermediate position separately from contributions introduced later. In trained Mamba-2 models on ListOps (Figure 2, left) and xLSTM models on Electricity and Traffic (Figure 6, right panels), propagated sensitivity generally decreases with distance to the terminal readout. Median sensitivity retention also generally decreases with distance on all three datasets (Figure 4 and Figure 6, left panels).

Proposition 2 gives a separate guarantee based on complementary curvature. The new conditioning analysis in Section 3.3 probes this mechanism by comparing the terminal and combined Gauss–Newton matrices on the same retained terminal subspace at fixed parameters. Adding the auxiliary loss improves projected conditioning in all 27 comparisons across the three datasets (Figure 3).

Theorem 1 and Proposition 2 give sufficient conditions for exponential attenuation and uniformly bounded conditioning, respectively. Our experiments show that the underlying mechanisms, sensitivity attenuation during recurrent propagation and complementary curvature from auxiliary supervision, are observable in trained Mamba-2 and xLSTM models.

AuthorsOpenReview ↗

Response to Reviewer xuGb (2/3) — Weakness 3 and Question 1

Weakness 3 — Information in the auxiliary targets.

We agree that the information supplied by the auxiliary targets matters for optimization. In the ListOps ablation, true subexpression targets reach the reference loss in 27,500 updates on average, compared with 37,100 for repeated terminal labels and 35,500 for terminal-only training. True subexpression supervision reaches the reference loss in fewer updates than both comparison variants in all three paired seeds and yields higher mean validation and length-generalization accuracy than repeated labels (Table 3).

The two auxiliary variants use the same readout, positions, and loss weight. When evaluated at the same weights and on the same examples, they have identical GGN matrices because the cross-entropy Hessian with respect to the logits depends only on the predicted probabilities. The projected conditioning improvement reported for ListOps in our reply to Weakness 1 therefore applies to both target choices at those weights. The auxiliary gradients, however, depend on the targets and can steer training along different trajectories.

In ListOps, repeating the terminal label can give the wrong target for a completed subexpression. For example, in min⁡(max⁡(1,2),0)\min(\max(1,2),0), it assigns a target of 00 to the intermediate readout for max⁡(1,2)=2\max(1,2)=2. The auxiliary gradient then favors an incorrect intermediate value, which can interfere with primary-task learning. This offers a plausible explanation for the slower progress toward the reference loss and is consistent with prior work on harmful auxiliary gradients; see, e.g., Du et al. (2018)

Question 1 — Empirical evidence for the theoretical mechanisms.

We agree that further empirical evidence would strengthen the link between the theory and the experiments. We therefore tested the mechanisms identified by the theory in trained models across different settings. In addition to Mamba-2 on ListOps with cross-entropy losses, we now evaluate causal xLSTM on two real-world forecasting datasets, Electricity and Traffic (see the new Section 3.2 in the revised manuscript for further details). Both the primary and auxiliary forecasting losses use mean squared error, and the self-supervised auxiliary targets are drawn directly from subsequent observations in the training series.

The forecasting experiments reproduce the distance-dependent sensitivity attenuation observed on ListOps, consistent with the mechanism described in Theorem 1 (see Weakness 2). Between the nearest and farthest distance bins, median sensitivity retention falls from 49.7% to 0.10% on Electricity and from 14.1% to 0.61% on Traffic (Figure 6 and Appendix C.2). On both datasets, intermediate readouts also increase the number of singular values above the 1% cutoff relative to the row-matched terminal control (Figure 7).

The fixed-parameter experiment described under Weakness 1 probes the complementary curvature mechanism in Proposition 2. Adding the auxiliary loss improves projected conditioning at matched weights across all three datasets (Figure 3). The weak-direction result also holds at all three checkpoints on each dataset (Table 5 and our reply to Weakness 1).

AuthorsOpenReview ↗

Response to Reviewer xuGb (3/3) — Questions 2–3

Question 2 — Relation to prior work

We thank the reviewer for raising this clarity issue. We agree that the distinction from these related works can be made more explicit. Indeed, they address complementary aspects of our setting, but focus on different theoretical questions:

  • Awasthi et al. (2025) derive conditions under which performance on task-structured auxiliary tasks guarantees length generalization; they do not study the optimization geometry induced by auxiliary supervision.
  • Zhao et al. (2024) characterize Gauss–Newton conditioning for deep linear neural networks, with extensions to two-layer ReLU networks, examining the influence of network architecture and data properties rather than auxiliary supervision or recurrent temporal propagation.
  • Zhang et al. (2026) analyze a specific auxiliary-supervision construction for LLM pre-training: NITP augments next-token prediction with a particular representation-level objective, and their analysis shows that this objective introduces curvature in representation directions that are weakly constrained by next-token prediction. This provides an important concrete example of how a suitably chosen auxiliary objective can improve optimization geometry, but their analysis focuses on the specific next-token pre-training setting and the corresponding representation-level auxiliary objective.

Our work studies this phenomenon in a general recurrent sequence-model setting, without committing to a particular task, recurrent architecture, or auxiliary-loss construction. We characterize terminal and auxiliary supervision through parameter sensitivities and GGN geometry, which yields a general parameter-observability perspective (Proposition 1). More importantly, this framework exposes a recurrent mechanism that is not addressed in the cited works: parameter information available at intermediate positions can be progressively attenuated by the recurrent Jacobian flow before reaching terminal supervision (Theorem 1). We show that this can lead to exponential deterioration of GGN conditioning with increasing sequence length (Corollary 1), while auxiliary observations satisfying explicit complementarity conditions can prevent this deterioration (Proposition 2). Thus, our main theoretical contribution is not merely that auxiliary supervision can add curvature, but that recurrent propagation itself can systematically attenuate parameter information with temporal distance, and that suitably placed auxiliary observations can capture this information before it is lost through the recurrent dynamics.

We have revised the related-work discussion in the updated manuscript to make these distinctions explicit.

Question 3 — Real-world forecasting with a different recurrent architecture.

We agree that it is important to test the framework across different tasks and architectures. In the new Section 3.2 and Appendix C.2, we extend the ListOps classification experiments with Mamba-2 and cross-entropy to real-world forecasting with causal xLSTM. We use one hourly series from each of the Electricity and Traffic benchmarks. Both the primary and auxiliary losses use mean squared error, and the auxiliary targets are self-supervised, drawn directly from subsequent observations in the training series.

On both forecasting datasets, we observe the same qualitative patterns as on ListOps. Sensitivity retention decreases with distance to the terminal readout, intermediate readouts reveal additional sensitivity directions in the row-matched comparison, and the auxiliary loss improves projected conditioning (Figures 3, 6, and 7). At a lookback of 192 and a forecast horizon of 96, the 12-hour schedule reduces mean updates to the primary reference loss by 15.8% on Electricity and 11.1% on Traffic and improves test MSE and MAE in all three paired seeds on each dataset (Table 1). Appendix C.2 reports the results across three lookbacks and four readout schedules.

The recurrence of these patterns across different tasks, architectures, losses, and sources of auxiliary supervision supports the relevance of the mechanisms identified by our theory.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised manuscript in response to reviewer feedback

We thank the reviewers for their careful and valuable feedback. We have revised the manuscript in response and highlighted the main changes in blue.

  • Real-world forecasting with xLSTM (Section 3.2 and Appendix C.2). We extend the ListOps experiments with Mamba-2 and cross-entropy loss to Electricity and Traffic forecasting with causal xLSTM, mean squared error, and self-supervised auxiliary targets. On both forecasting datasets, we reproduce the qualitative patterns from Figure 2 (Figures 6–8). At a lookback of 192, the 12-hour schedule reduces mean updates to the reference loss by 15.8% on Electricity and 11.1% on Traffic and lowers both test MSE and MAE (Table 1). We also test three lookbacks at the same auxiliary-loss weight (Table 4).
  • Complementary curvature across tasks (Section 3.3 and Appendix C.3). At matched weights and on the same retained terminal subspace, adding the auxiliary loss improves projected conditioning in all 27 comparisons across ListOps, Electricity, and Traffic. The auxiliary loss adds curvature along weak terminal directions with both subexpression labels and self-supervised forecasting targets (Figure 3 and Table 5).

Overall, we observe the sensitivity attenuation and complementary curvature described by our theory across the tested tasks, recurrent architectures, losses, and forms of auxiliary supervision.

AuthorsOpenReview ↗

Summary of Rebuttal and Manuscript Revisions

Dear Area Chair and Reviewers,

We thank the reviewers for their constructive feedback throughout the discussion. We have responded to their questions individually and substantially expanded the empirical evaluation. Earlier revisions are highlighted in blue and red, and the latest additions in cyan.

Our central contribution is a sensitivity-based framework explaining dynamical forgetting, complementary auxiliary curvature, and the primary-task performance trade-off. The revisions strengthen its empirical support across different tasks, recurrent architectures, losses, and forms of auxiliary supervision:

  • Real-world forecasting with xLSTM (Section 3.2 and Appendix C.2). Electricity and Traffic reproduce the qualitative mechanisms observed on ListOps, using a different architecture, loss, and self-supervised auxiliary targets. At lookback 192, the 12-hour schedule reduces mean updates to the reference loss by 15.8% and 11.1%, respectively, while lowering test MSE and MAE for every paired seed. We also evaluate three lookbacks and four readout schedules.
  • Sequential CIFAR-10 with unsupervised auxiliary reconstruction (Section 3.3 and Appendix C.3). We added sequence classification with an unsupervised auxiliary objective distinct from classification. With 100 epochs of unsupervised pretraining followed by 100 epochs of joint training, mean test accuracy increases from 56.53% to 59.62% compared with terminal-only supervision, improving in all three paired joint-training seeds. Final mean primary training loss is 17% lower (Table 5 and Figure 10).
  • Controlled curvature measurements (Section 3.3 and Appendix C.4). Adding auxiliary supervision improves projected conditioning in all 36 comparisons across ListOps, Electricity, Traffic, CIFAR-10, using supervised and unsupervised auxiliary losses (Figure 3 and Table 6).

Together, these experiments illustrate Theorem 1 and Proposition 2 across different recurrent architectures, tasks, and auxiliary objectives, alongside practical improvements in training and prediction.

We thank the reviewers again for the fruitful rebuttal discussion, which we believe has substantially strengthened the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.