- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the imbalanced regression problem, where the label space is continuous. The authors introduce a training strategy named ECA, which combines a data exposure curriculum with an adjusted learning rate scheduler. The proposed method aims to adjust the weight dynamics during optimization, allowing the model to learn from the entire distribution early on before shifting focus later in training to improve predictions in the tail regions. However, this work contains several weaknesses, which are discussed below.
Critical strengths
- The paper is relatively easy to read and understand.
- The results seem to indicate that the proposed method can be integrated with other models and can exhibit improvements over existing baselines.
- The method is tested across different modalities.
Critical weaknesses
- The paper reads more like a recipe of how to train a model in case of DIR rather than a new method with very weak or limited explanation. For example, it is unclear why this specific curriculum works better than a simpler two-stage training alternative (e.g., pre-training on the full distribution and fine-tuning on an undersampled/rebalanced distribution).
- The results are not consistent with the paper claims. On IMDB-WIKI-DIR, the Vanilla+ECA improves the performances on the many regions in comparison to the other baseline but performs worse than the baselines for the medium and few shot region (Table 2). Furthermore, methods like ConR are missing in that table (Table 2). While a broader comparison is provided in Table 1, several stronger method combinations established in the literature are missing.
- While Section 4.4 aims to justify the paper's claims regarding precision and optimization dynamics, the takeaway remains unclear. For example, the authors do not clearly demonstrate how the singular value spectrum of the weights evolves, nor how this strategy fundamentally differs from standard fine-tuning.
- The improvements brought by ECA are not consistent across all experimental settings; in at least one case, adding ECA performs worse than the vanilla model.
Small weaknesses:
- Figure 1 is not very informative.
- Some language should be improved. For example in the abstract some sentences are not clear, such as ‘description length view of capacity’ or when saying ‘currently do so by restricting the model and therefore sacrifice orthogonality.’ Or ‘majority tax of static rebalancing’.
- Please update the references of the papers proceedings when available rather than their arXiv version.
Questions for authors
Please see above.
- In the ablations, using WSD seems to help a lot in the results without any consideration of the imbalance, could you elaborate on it?
- How are N_Warmup and N_decay chosen for each dataset
3 comments
W3 – Singular value spectrum and difference to standard fine-tuning. We computed the singular value decomposition of every weight matrix of the ResNet-50 on AgeDB-DIR at the start of the decay phase and at the end of training, and analysed the update ΔW in between (mean over all weight tensors, four seeds):
| Method | ‖ΔW‖ / ‖W‖ | Stable rank of ΔW | Share of ΔW in the top-10% singular subspace of W | Change of the top-10% singular values |
|---|---|---|---|---|
| V. + ECA | 0.158 | 15.3 | 0.129 | 4.9% |
| V. + WSD | 0.152 | 14.8 | 0.135 | 4.7% |
| V. + LDS + WSD | 0.145 | 14.0 | 0.137 | 4.3% |
| V. + SQINV + WSD | 0.146 | 14.0 | 0.137 | 4.3% |
| V. + LDS, constant LR | 0.211 | 12.6 | 0.172 | 9.1% |
| V. + SQINV, constant LR | 0.211 | 13.0 | 0.167 | 9.2% |
The two rows with a constant learning rate correspond to standard fine-tuning on a re-balanced objective. There, the update is about 40% larger, has a lower rank, is more concentrated in the dominant singular directions of the weights, and changes the largest singular values about twice as much. With a decaying learning rate the update is smaller and spread over more directions, and the dominant directions are changed less. This is the difference to standard fine-tuning we refer to in Section 4.4.
In these spectral statistics ECA and V. + WSD are almost identical: the update has the same size and the same structure. This is expected, since the spectrum describes how much and in which directions the weights are changed, which is determined by the learning rate schedule that both share. It cannot show what the update is used for. That is visible in the shot regions: with the same update budget, ECA reaches a lower medium- and few-shot MAE than V. + WSD (7.90 vs. 8.22 and 10.35 vs. 11.67) at an unchanged many-shot MAE (6.25 vs. 6.20). So the data curriculum does not change the weights more, or more coarsely, than WSD alone. It changes on which samples the small updates of the decay phase are spent. This is consistent with our interpretation in Section 4.4, although we want to be clear that it remains an interpretation and not a direct measurement of weight precision. We will add this analysis to the appendix and make the takeaway of Section 4.4 explicit.
W4 – In one case ECA performs worse than Vanilla. We agree that in one of the eleven experiments (Targa '13) ECA performs worse than Vanilla, and we flagged that as a limitation. We investigated this case further: Vanilla + WSD without the data curriculum shows the same degradation (bMASE 0.2166 vs. 0.2176 for ECA and 0.1113 for Vanilla), so it is caused by the learning rate schedule and not by the curriculum. If the schedule is selected per dataset by validation loss, the warmup is removed and ECA reaches 0.1274 on this dataset.
Small weaknesses. Thank you, we will revise Figure 1, clarify the mentioned sentences in the abstract and update the references to the proceedings versions.
Q – WSD seems to help a lot without any consideration of the imbalance. WSD improves the model overall, it gains in every region, and it is the only component in Table 5 that improves the many-shot region. It is a better general-purpose schedule than the step schedule that is established in the DIR literature, independent of the imbalance. We now quantify this on both AgeDB and the IMDB benchmarks: WSD alone accounts for 7.68 to 7.04 on AgeDB-DIR, 8.06 to 7.79 on IMDB-WIKI-DIR. The data curriculum then adds the part that is specific to the imbalance: on AgeDB-DIR the few-shot MAE drops from 11.67 to 10.35 and the medium-shot MAE from 8.22 to 7.90 at an unchanged many-shot MAE, and on IMDB-WIKI-DIR the overall MAE drops further to 7.40.
Q – How are N_warmup and N_decay chosen for each dataset? By a grid search over the warmup and decay fraction in {0.0, …, 0.5} with selection by validation loss, as described in Appendix D.3. For MuViS-DIR one shared configuration is used for all nine datasets. Selecting the schedule per dataset improves the geometric mean bMASE further from 0.1848 to 0.1698.
We want to thank the reviewer again for their feedback and their questions. We especially appreciate the input regarding the singular value decomposition, as these results additionally support the findings of the paper.
I thank the authors for providing additional experiments in their rebuttal. After reviewing the rebuttal and reading the comments from other reviewers, I find myself aligning with the core concerns raised by Reviewer pLV5.
While the motivation is logical, the method mainly combines existing components (progressive data curriculum and the WSD schedule), leaving the overall contribution feeling somewhat incremental for a main-conference publication.
Therefore, while I appreciate the thoroughness of the authors response, I choose to maintain my recommendation of Weak Reject.
We thank the reviewer for the feedback and address the points in order.
W1 – The paper reads more like a recipe than a new method. We agree that ECA is close to a training recipe, and we would even see this as a strength: it changes neither the model nor the loss, introduces no hyperparameters of its own, and can be put on top of an existing training pipeline. What makes it a method and not only a collection of settings is the coupling. The data exposure is tied to the learning rate at every step, so that the re-balanced samples are seen exactly when the updates become small. To our knowledge we are the first to introduce such a continuous coupling between a data curriculum and the learning rate.
W1 - Why does this perform better than a simpler two-stage alternative? Pre-training on the full distribution and fine-tuning on a re-balanced distribution is, in the DIR field, the same as RRT (Yang et al.), which is part of our comparison. ECA outperforms RRT in all regions on both age datasets:
(AgeDB-DIR numbers are ten-seed means from Table 1. The IMDB-WIKI-DIR numbers for Vanilla and RRT are taken from Yang et al. and are single-seed numbers.)
Another option to introduce such a two-stage scheme is an abrupt switch of the data distribution within one training run. We tested this in Appendix J.3 (Table 29), where the ceiling is dropped together with the two steps of the established step schedule. This variant reaches 7.40 / 6.75 / 8.00 / 11.86 and stays behind ECA in all four regions. After the abrupt switch the best validation checkpoint is reached within a few hundred steps, i.e. the model cannot make use of the re-balanced phase. This is the reason why we couple the ceiling continuously to the learning rate.
W2 – Inconsistency in the results. ECA does not perform worse than the Vanilla baseline on IMDB-WIKI-DIR, it improves on it in all four regions. In comparison with the other single add-on methods, ECA performs best in the overall and many-shot column and third best in the medium- and few-shot column, where it is beaten by SQINV and BMSE, which sacrifice their many-shot performance for their medium- and few-shot performance:
ConR is missing in Table 2 because of the inconsistent reporting across papers. We took the IMDB-WIKI-DIR numbers from the paper with the best reported baselines (UVOTE). We nevertheless compare against the numbers reported by ConR in the following table. As before, these are single-seed numbers for ConR and ten-seed means for ECA:
ConR and ECA are, however, not alternatives. ConR regularizes the representation, ECA changes the data exposure, and the two can be combined. On AgeDB-DIR we trained ConR and RankSim with the same schedule as ECA, once with WSD only and once with ECA on top (five seeds, Appendix G.1 of the revised PDF):
There, ConR + ECA is the best orthogonal configuration overall and in the few-shot region, and it improves on ConR in every region. We have not yet run this combination on IMDB-WIKI-DIR. Given the AgeDB-DIR result, we see ConR + ECA on IMDB-WIKI-DIR as the natural next step and will add it to the final version.