- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper is motivated based on the phenonmenon where in recurrent sequence model, due to terminal supervision, this caused parameter-information bottleneck to form, where intermediate parameter direction are diminished. They theoretically show that auxiliary loss applied on intermediate readout could mitigate the bottleneck by exposing weakly observed parameter direction. Finally, they show the primary-auxiliary trade-off and demonstrate empirically that auxiliary loss allow for faster and more stable training on long range dependency task.
Critical strengths
-
This paper is well motivated on diminishing effect of intermediate parameter direction as sequence length increase. They clearly show that how auxiliary loss will help in exposing the parameter direction under reasonable assumption.
-
The empirical result on ListOps is significant both in term of demonstrating terminal observation only weakly representing parameter direction and also the effect of auxiliary loss in aiding the convergence and conditioning of the training.
I find this paper while mathematically dense, it is clearly written and accompanied with intuitive explanation.
Critical weaknesses
-
ListOps dataset have semantically meaningful subexpression groundtruth that is strongly correlated with the terminal groundtruth, this might be highly favourable to the proposed method. In practice however, such intermediate annotation are rare. The paper only discuss supervised auxiliary loss, prior work like [1] and [2] also utilized unsupervised auxiliary loss ([2] use auxiliary loss more like a regularizer). As Proposition 2 relies on complementary auxiliary loss, how often and stable we can obtain with unsupervised or self-supervise loss that satisfies the condition is a key to its practicability.
-
The empirical experiment scope for this paper is rather narrow, only 1 dataset and 1 model. It is important to at least repeat across 1 or 2 more datasets to solidify the claim. It would be even better to test on different recurrent neural network like original Mamba or xLSTM [3]. All training are done on , is it possible to have result at different ?
[1] Trinh, Trieu, et al. "Learning Longer-term Dependencies in RNNs with Auxiliary Losses." International Conference on Machine Learning. PMLR, 2018.
[2] Kag, Anil, and Venkatesh Saligrama. "Training recurrent neural networks via forward propagation through time." International Conference on Machine Learning. PMLR, 2021.
[3] Beck, Maximilian, et al. "xlstm: Extended long short-term memory." Advances in Neural Information Processing Systems 37 (2024): 107547-107603.
Questions for authors
-
Is it possible that the author could have a simple experiment showing that the proposed framework, especially Proposition 2 will hold under unsupervised on semi-supervised auxiliary loss?
-
While the rebuttal period is rather tight, is it possible that the author could show result on 1 or 2 more datasets? Showing that the result in Figure 2. will hold on other datasets will significantly make the claim stronger.
-
How stable is the proposed method under different ? Does requires huge changes as changes? (Minor)
I am happy to raise my score if my concerns are resolved sufficiently (not necessarily all of them but especially 1. and 2. to a acceptable extend).
6 comments
Thank you very much for the detailed response. My concerns in 2. have been partially addressed and 3. have been substantially addressed.
Below are my remaining concerns:
-
I thank the authors for the effort in adding two additional datasets and showing that the proposed method is effective with data-derived auxiliary losses. However, this does not fully address my concern, as the intermediate objective in the Electricity and Traffic forecasting benchmarks is essentially the same as the terminal forecasting objective, only applied at different positions. This still limits the evidence for the applicability of the proposed method to more general datasets, for example, sequence classification, where such intermediate targets are not naturally available. For example, [1] utilized MNIST, pMNIST, and CIFAR-10 for sequence classification and showed that auxiliary reconstruction losses can still be effective, even when the auxiliary objective differs from the terminal classification objective.
-
The selected additional benchmarks still greatly favour the proposed method, as mentioned in 1. Since the paper originally utilized ListOps from the Long Range Arena benchmark, why not consider other datasets from the same benchmark, such as Image (CIFAR-10) or Text (IMDb)? I believe that showing the proposed method is effective on these datasets, where the auxiliary objective differs from the terminal classification objective, would significantly strengthen the claim of general applicability.
[1] Trinh, Trieu, et al. "Learning Longer-term Dependencies in RNNs with Auxiliary Losses." International Conference on Machine Learning. PMLR, 2018.
Thank you for clarifying your remaining concerns and for suggesting this experiment. Following your suggestion, we have added sequential CIFAR-10 classification with unsupervised auxiliary reconstruction, using the Long Range Arena image setting and following the LSTM architecture and the auxiliary reconstruction loss design proposed by Trinh et al. (2018). Thus, the auxiliary objective differs from terminal classification and requires neither intermediate class labels nor additional annotations. In the revised manuscript, we describe the setup in Appendix C.3 and incorporate the results into Section 3.3.
The new experiment supports the mechanisms studied in our paper:
- Dynamical forgetting: In the trained terminal-only classifier, median sensitivity retention decreases from 79% in the nearest distance bin to 12% in the farthest bin, consistent with the attenuation mechanism in Theorem 1 (new Figure 9 in the revised manuscript).
- Complementary curvature: Across three seeds, we evaluate the shared initialization and two checkpoints from training with auxiliary supervision. In each comparison, we evaluate the terminal GGN matrix and the combined GGN matrix at the same weights, on the same minibatch, and on the same retained terminal subspace. Adding reconstruction improves projected conditioning in all nine comparisons (see the revised Figure 3). The smallest projected eigenvalue increases by a larger factor than the largest in every comparison (Table 5 and Appendix C.4).
At the common 100-epoch budget, mean test accuracy is essentially unchanged (56.41% with terminal-only training and 56.44% with auxiliary supervision), and we find no consistent reduction in updates to the reference loss. Our shorter training schedule may partly explain the difference in accuracy outcomes compared with Trinh et al. (2018), who use auxiliary pretraining followed by 1,000 epochs of joint training. These results distinguish the local curvature improvement from its eventual effect on optimization and prediction.
The added experiment therefore extends the empirical evidence to precisely the setting you highlighted: sequence classification with an unsupervised auxiliary objective, distinct from the primary classification objective. It illustrates the dynamical-forgetting mechanism in Theorem 1 and the complementary-curvature mechanism in Proposition 2, two central theoretical contributions of our paper. Together with the ListOps and forecasting experiments, these results support the applicability of these mechanisms across different recurrent architectures and forms of auxiliary supervision.
We would like to thank the reviewer again for their constructive feedback throughout the discussion, which we believe has substantially strengthened the paper.
Response to Reviewer d4P4
Thank you for the careful review and for suggesting these experiments. The revised manuscript adds self-supervised forecasting on two real-world datasets with xLSTM, a conditioning analysis, and results across three lookbacks.
Weakness 1 and Question 1 — Self-supervised auxiliary targets and complementary curvature
We agree that it is important to test whether auxiliary targets derived from the data can supply curvature in directions that are weakly observed under terminal supervision. To test this, we added two forecasting experiments with self-supervised auxiliary targets. At selected intermediate positions, we reuse the terminal forecast readout and use the next 96 observations as targets. Both the primary and auxiliary losses use mean squared error.
At a lookback of 192, we compare the terminal and combined GGN matrices at the same weights, on the same minibatch, and on the same retained terminal subspace. Adding the self-supervised loss improves projected conditioning in all 18 forecasting comparisons across two datasets, three seeds, and three checkpoints (Section 3.3 and Figure 3). In each of these comparisons, the weighted auxiliary term also contributes more curvature than the terminal term along every direction in the span of the five weakest retained terminal directions (Appendix C.3 and Table 5). This provides direct empirical evidence for the complementary-curvature mechanism in Proposition 2 with self-supervised targets.
Weakness 2 and Question 2 — Additional datasets, xLSTM, and the Figure 2 analyses
We agree that evaluating the framework on additional datasets and with a different recurrent architecture strengthens the empirical evidence. Following your suggestion, we use causal xLSTM in the new univariate forecasting experiments on one hourly series from each of the Electricity and Traffic benchmarks (Section 3.2 and Appendix C.2). On both datasets, we reproduce the qualitative patterns from Figure 2. Sensitivity retention decreases with distance to the terminal readout, and intermediate readouts increase the number of singular values above the 1% cutoff relative to the row-matched terminal control (Figures 6 and 7). At a lookback of 192, the 12-hour schedule reduces mean updates to the primary reference loss by 15.8% on Electricity and 11.1% on Traffic (Table 1 and Figure 8). It also lowers both test MSE and MAE for all three paired seeds on each dataset (Table 1).
To assess the effect of lookback length, we trained separate forecasting models at N=96, 192, and 336. With the 12-hour auxiliary schedule, both test errors improve at all three lookbacks on both datasets. The reduction in mean updates to the reference loss is larger at longer lookbacks on both datasets (Table 4), consistent with the motivation for intermediate supervision when sensitivity from earlier positions must propagate farther to reach the terminal readout. We also report a length-generalization test on ListOps, evaluating models trained at N=72 on held-out sequences of length N=288 (Table 3).
Question 3 — Lookback length and auxiliary-loss weight
We agree that it is important to understand whether the auxiliary-loss weight needs to change with input length. In the forecasting experiments, we used the same weight, λ=0.3, at lookbacks of N=96, 192, and 336. With the 12-hour schedule, both test errors decrease in all six dataset–lookback combinations, and the mean number of updates to reach the reference loss decreases in five of six. On Traffic at N=96, the mean update count increases by 2.7% (Table 4). These results indicate that the auxiliary-loss weight can be kept fixed as the lookback varies across the tested range.
The ListOps ablation examines sensitivity to λ at the fixed training length N=72. Moderate weights give the strongest results, and the largest weight requires more updates to reach the reference loss than terminal-only training (Figure 5, panel A). This illustrates why the auxiliary loss should have enough weight to contribute to learning while preserving progress on the primary task.