acceptodds
Under review as a conference paper at ICLR 2027

Allocating Weight Precision for Deep Imbalanced Regression with Edge-Case Annealing

Abstract

Deep Imbalanced Regression (DIR) addresses the mismatch between naturally imbalanced training distributions and the more balanced performance desired across the target range. Methods that resolve the underlying capacity conflict, in which parameters shared across the target range receive updates dominated by majority samples, currently do so by restructuring the model and therefore sacrifice orthogonality. We introduce Edge-Case Annealing (ECA), a training-schedule method that leaves the loss and the architecture untouched. ECA couples a data exposure curriculum one to one to the learning rate schedule: a ceiling on the target histogram tracks the learning rate, so that the effective training distribution reaches balance exactly as the step size vanishes. The design follows a minimum description length view of capacity: under the imbalanced training distribution, fine-grained structure for rare targets does not justify the weight precision it requires, whereas under the balanced evaluation distribution it does, and it can be retained once the learning rate, and with it the gradient noise, has decayed. On AgeDB-DIR, IMDB-WIKI-DIR, and the MuViS-DIR time series benchmark, ECA is the strongest orthogonal addition, improves the tail without the majority tax of static rebalancing, and composes with existing methods up to a new state of the art in the medium and few-shot regions. Under increasing weight noise, the annealing gains are erased in rare target regions, consistent with being stored as small, precision-fragile adjustments and supporting the proposed capacity view.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
deep imbalanced regression, long-tailed regression, learning-rate schedules, curriculum learning, age estimation, virtual sensing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_duKaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the imbalanced regression problem, where the label space is continuous. The authors introduce a training strategy named ECA, which combines a data exposure curriculum with an adjusted learning rate scheduler. The proposed method aims to adjust the weight dynamics during optimization, allowing the model to learn from the entire distribution early on before shifting focus later in training to improve predictions in the tail regions. However, this work contains several weaknesses, which are discussed below.

Critical strengths
  • The paper is relatively easy to read and understand.
  • The results seem to indicate that the proposed method can be integrated with other models and can exhibit improvements over existing baselines.
  • The method is tested across different modalities.
Critical weaknesses
  • The paper reads more like a recipe of how to train a model in case of DIR rather than a new method with very weak or limited explanation. For example, it is unclear why this specific curriculum works better than a simpler two-stage training alternative (e.g., pre-training on the full distribution and fine-tuning on an undersampled/rebalanced distribution).
  • The results are not consistent with the paper claims. On IMDB-WIKI-DIR, the Vanilla+ECA improves the performances on the many regions in comparison to the other baseline but performs worse than the baselines for the medium and few shot region (Table 2). Furthermore, methods like ConR are missing in that table (Table 2). While a broader comparison is provided in Table 1, several stronger method combinations established in the literature are missing.
  • While Section 4.4 aims to justify the paper's claims regarding precision and optimization dynamics, the takeaway remains unclear. For example, the authors do not clearly demonstrate how the singular value spectrum of the weights evolves, nor how this strategy fundamentally differs from standard fine-tuning.
  • The improvements brought by ECA are not consistent across all experimental settings; in at least one case, adding ECA performs worse than the vanilla model.

Small weaknesses:

  • Figure 1 is not very informative.
  • Some language should be improved. For example in the abstract some sentences are not clear, such as ‘description length view of capacity’ or when saying ‘currently do so by restricting the model and therefore sacrifice orthogonality.’ Or ‘majority tax of static rebalancing’.
  • Please update the references of the papers proceedings when available rather than their arXiv version.
Questions for authors

Please see above.

  1. In the ablations, using WSD seems to help a lot in the results without any consideration of the imbalance, could you elaborate on it?
  2. How are N_Warmup and N_decay chosen for each dataset
3 comments

We thank the reviewer for the feedback and address the points in order.

W1 – The paper reads more like a recipe than a new method. We agree that ECA is close to a training recipe, and we would even see this as a strength: it changes neither the model nor the loss, introduces no hyperparameters of its own, and can be put on top of an existing training pipeline. What makes it a method and not only a collection of settings is the coupling. The data exposure is tied to the learning rate at every step, so that the re-balanced samples are seen exactly when the updates become small. To our knowledge we are the first to introduce such a continuous coupling between a data curriculum and the learning rate.

W1 - Why does this perform better than a simpler two-stage alternative? Pre-training on the full distribution and fine-tuning on a re-balanced distribution is, in the DIR field, the same as RRT (Yang et al.), which is part of our comparison. ECA outperforms RRT in all regions on both age datasets:

MethodOverallManyMedianFew
AgeDB-DIR: Vanilla7.686.678.9913.52
AgeDB-DIR: V. + RRT7.366.588.3012.15
AgeDB-DIR: V. + ECA6.916.257.9010.35
IMDB-WIKI-DIR: Vanilla8.067.2315.1226.33
IMDB-WIKI-DIR: V. + RRT7.817.0714.0625.13
IMDB-WIKI-DIR: V. + ECA7.406.7312.8224.00

(AgeDB-DIR numbers are ten-seed means from Table 1. The IMDB-WIKI-DIR numbers for Vanilla and RRT are taken from Yang et al. and are single-seed numbers.)

Another option to introduce such a two-stage scheme is an abrupt switch of the data distribution within one training run. We tested this in Appendix J.3 (Table 29), where the ceiling is dropped together with the two steps of the established step schedule. This variant reaches 7.40 / 6.75 / 8.00 / 11.86 and stays behind ECA in all four regions. After the abrupt switch the best validation checkpoint is reached within a few hundred steps, i.e. the model cannot make use of the re-balanced phase. This is the reason why we couple the ceiling continuously to the learning rate.

W2 – Inconsistency in the results. ECA does not perform worse than the Vanilla baseline on IMDB-WIKI-DIR, it improves on it in all four regions. In comparison with the other single add-on methods, ECA performs best in the overall and many-shot column and third best in the medium- and few-shot column, where it is beaten by SQINV and BMSE, which sacrifice their many-shot performance for their medium- and few-shot performance:

MethodOverallManyMedianFew
V. + RankSim7.726.9314.4825.38
V. + SQINV7.877.2412.4422.76
V. + BMSE8.087.5212.4723.29
V. + DenseWeight7.857.1413.7025.38
V. + ECA7.406.7312.8224.00

ConR is missing in Table 2 because of the inconsistent reporting across papers. We took the IMDB-WIKI-DIR numbers from the paper with the best reported baselines (UVOTE). We nevertheless compare against the numbers reported by ConR in the following table. As before, these are single-seed numbers for ConR and ten-seed means for ECA:

MethodOverallManyMedianFew
V. + ECA7.406.7312.8224.00
V. + ConR7.336.7511.9922.22
V. + FDS + RankSim + ConR7.446.7811.7921.32
V. + SQINV + ECA7.567.0511.5121.74

ConR and ECA are, however, not alternatives. ConR regularizes the representation, ECA changes the data exposure, and the two can be combined. On AgeDB-DIR we trained ConR and RankSim with the same schedule as ECA, once with WSD only and once with ECA on top (five seeds, Appendix G.1 of the revised PDF):

MethodOverallManyMedianFew
V. + ConR (step, Table 1)7.316.388.5812.51
V. + WSD7.036.148.3111.89
V. + ConR + WSD7.016.138.3111.68
V. + ECA6.906.247.8910.40
V. + ConR + ECA6.836.187.8910.06

There, ConR + ECA is the best orthogonal configuration overall and in the few-shot region, and it improves on ConR in every region. We have not yet run this combination on IMDB-WIKI-DIR. Given the AgeDB-DIR result, we see ConR + ECA on IMDB-WIKI-DIR as the natural next step and will add it to the final version.

W3 – Singular value spectrum and difference to standard fine-tuning. We computed the singular value decomposition of every weight matrix of the ResNet-50 on AgeDB-DIR at the start of the decay phase and at the end of training, and analysed the update ΔW in between (mean over all weight tensors, four seeds):

Method‖ΔW‖ / ‖W‖Stable rank of ΔWShare of ΔW in the top-10% singular subspace of WChange of the top-10% singular values
V. + ECA0.15815.30.1294.9%
V. + WSD0.15214.80.1354.7%
V. + LDS + WSD0.14514.00.1374.3%
V. + SQINV + WSD0.14614.00.1374.3%
V. + LDS, constant LR0.21112.60.1729.1%
V. + SQINV, constant LR0.21113.00.1679.2%

The two rows with a constant learning rate correspond to standard fine-tuning on a re-balanced objective. There, the update is about 40% larger, has a lower rank, is more concentrated in the dominant singular directions of the weights, and changes the largest singular values about twice as much. With a decaying learning rate the update is smaller and spread over more directions, and the dominant directions are changed less. This is the difference to standard fine-tuning we refer to in Section 4.4.

In these spectral statistics ECA and V. + WSD are almost identical: the update has the same size and the same structure. This is expected, since the spectrum describes how much and in which directions the weights are changed, which is determined by the learning rate schedule that both share. It cannot show what the update is used for. That is visible in the shot regions: with the same update budget, ECA reaches a lower medium- and few-shot MAE than V. + WSD (7.90 vs. 8.22 and 10.35 vs. 11.67) at an unchanged many-shot MAE (6.25 vs. 6.20). So the data curriculum does not change the weights more, or more coarsely, than WSD alone. It changes on which samples the small updates of the decay phase are spent. This is consistent with our interpretation in Section 4.4, although we want to be clear that it remains an interpretation and not a direct measurement of weight precision. We will add this analysis to the appendix and make the takeaway of Section 4.4 explicit.

W4 – In one case ECA performs worse than Vanilla. We agree that in one of the eleven experiments (Targa '13) ECA performs worse than Vanilla, and we flagged that as a limitation. We investigated this case further: Vanilla + WSD without the data curriculum shows the same degradation (bMASE 0.2166 vs. 0.2176 for ECA and 0.1113 for Vanilla), so it is caused by the learning rate schedule and not by the curriculum. If the schedule is selected per dataset by validation loss, the warmup is removed and ECA reaches 0.1274 on this dataset.

Small weaknesses. Thank you, we will revise Figure 1, clarify the mentioned sentences in the abstract and update the references to the proceedings versions.

Q – WSD seems to help a lot without any consideration of the imbalance. WSD improves the model overall, it gains in every region, and it is the only component in Table 5 that improves the many-shot region. It is a better general-purpose schedule than the step schedule that is established in the DIR literature, independent of the imbalance. We now quantify this on both AgeDB and the IMDB benchmarks: WSD alone accounts for 7.68 to 7.04 on AgeDB-DIR, 8.06 to 7.79 on IMDB-WIKI-DIR. The data curriculum then adds the part that is specific to the imbalance: on AgeDB-DIR the few-shot MAE drops from 11.67 to 10.35 and the medium-shot MAE from 8.22 to 7.90 at an unchanged many-shot MAE, and on IMDB-WIKI-DIR the overall MAE drops further to 7.40.

Q – How are N_warmup and N_decay chosen for each dataset? By a grid search over the warmup and decay fraction in {0.0, …, 0.5} with selection by validation loss, as described in Appendix D.3. For MuViS-DIR one shared configuration is used for all nine datasets. Selecting the schedule per dataset improves the geometric mean bMASE further from 0.1848 to 0.1698.

We want to thank the reviewer again for their feedback and their questions. We especially appreciate the input regarding the singular value decomposition, as these results additionally support the findings of the paper.

I thank the authors for providing additional experiments in their rebuttal. After reviewing the rebuttal and reading the comments from other reviewers, I find myself aligning with the core concerns raised by Reviewer pLV5.
While the motivation is logical, the method mainly combines existing components (progressive data curriculum and the WSD schedule), leaving the overall contribution feeling somewhat incremental for a main-conference publication.

Therefore, while I appreciate the thoroughness of the authors response, I choose to maintain my recommendation of Weak Reject.

Reviewer_rPQVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies Deep Imbalanced Regression and proposes Edge-Case Annealing. It couples Data Exposure Curriculum with Warmup-Stable-Decay learning rate to balance the sample distribution as the learning rate decays. The author introduces weight-precision hypothesis based on Minimum Description Length, suggesting that small updates late in training can help improve performance on minor samples while preserving performance on majority. Experiments include AgeDB-DIR, IMDB-WIKI-DIR, and MuViS-DIR.

Critical strengths
  1. The method is simple yet widely applicable. ECA modifies only the data sampling strategy, making it very easy to integrate into existing systems.
  2. The experiments include a wide range of tasks across multiple image and time-series datasets, and the results show notable improvements.
  3. The paper provides extensive ablations and analyses of the proposed mechanism. It examines the effects of DEC, WSD, backbone size, and schedule synchronization.
  4. The paper is clearly organized and provides good support for reproducibility. The authors report detailed training configurations, hyperparameter choices, and results across ten random seeds for most main experiments.
Critical weaknesses
  1. The independent contribution of ECA has not been sufficiently proven. On AgeDB-DIR, WSD alone reduces the overall MAE from 7.68 to 7.04, while ECA further reduces it to 6.91; similarly comprehensive WSD-only comparisons are missing for the other benchmarks.
  2. The method has limitations in robustness and performance. ECA shows performance degradation on Targa '13 and can worsen overall performance when combined with UVote.
Questions for authors
  1. Could the authors provide Vanilla + WSD baselines on IMDB-WIKI-DIR and MuViS-DIR using the same learning rate schedules as ECA, to clarify the independent contribution of DEC when WSD is fixed?
  2. Could the authors investigate the performance degradation on Targa '13 and report sensitivity analyses for the bin count, minimum bin size, and cumulative sample exposure, etc.
3 comments

We thank the reviewer for the feedback and the two concrete questions, which we answer with new experiments below.

W1 – Independent contribution of ECA. We agree that WSD is already a better schedule than the one the standard DIR literature uses. But especially in Table 5 we see that WSD is the component that improves the many-shot region (6.67 to 6.20), which then also translates to the overall column. The interesting results for DIR are in the medium- and few-shot columns, where WSD is outperformed both times by the data curriculum (8.08 vs. 8.22 and 11.11 vs. 11.67) and where ECA shows its biggest effect. The WSD-only comparisons for the other benchmarks are given under Q1.

W2 – Robustness. We agree that on UVOTE, ECA sacrifices many-shot performance for better few-shot performance. Targa '13 was kept in the benchmark deliberately, as we think it is a good example of a potential limitation. We investigated it further under Q2: the degradation is caused by the learning rate schedule and not by the data curriculum.

Q1 – Vanilla + WSD on IMDB-WIKI-DIR and MuViS-DIR. We trained Vanilla + WSD with exactly the schedule used for ECA on both benchmarks.

IMDB-WIKI-DIR (MAE):

MethodOverallManyMediumFew
Vanilla8.067.2315.1226.33
V. + WSD7.797.0913.6524.20
V. + ECA7.406.7312.8224.00

With WSD fixed, the data curriculum reduces the overall MAE by a further 0.39, the many-shot MAE by 0.36, the medium-shot MAE by 0.83 and the few-shot MAE by 0.20. This shows that on IMDB-DIR ECA brings similar gains on top of the WSD-schedule than on AgeDB-DIR, with the difference of ECA also outperforming WSD in the many-shot region.

MuViS-DIR (bMASE):

DatasetVanillaV. + WSD (same setting as ECA)V. + WSD (per dataset tuned)V. + ECA (per dataset tuned)
PM100.65440.63590.62730.6387
PM2.50.56210.55180.51160.5078
Heart Rate0.13030.09230.06090.0550
Battery SoC0.02290.02530.02460.0245
Monterey0.21400.19030.10040.1056
Targa '130.11130.21660.21660.1274
Targa '140.12700.12490.12490.0963
Chem. Conc.0.70250.64520.61540.6086
Tyre Temp.0.34150.35410.35380.3404
Geometric mean0.21000.18960.18740.1698

On MuViS-DIR we first trained V. + WSD with the same schedule as in ECA and to further investigate the independent contribution we individually tuned V. + WSD and V. + ECA and did the selection based on the validation loss. ECA is with same setting outperforming WSD in 8 out of 9 datasets and in the individually trained version now the overall best in 4 out 9 cases with the smallest overall geometric mean.

Q2 – Targa '13 and sensitivity. The table shows bMASE on Targa '13 and, for comparison, on and PM2.5. The default is 50 bins, a minimum bin size of 1, warmup 0.1 and decay 0.2. The decay fraction controls how long the ceiling is lowered and therefore the cumulative sample exposure.

VariantPM2.5Targa '13
Vanilla0.56210.1113
V. + WSD (no data curriculum)0.51160.2166
V. + ECA (default)0.51480.2176
Bin count 10 / 25 / 100 / 2000.5143 / 0.5178 / 0.5088 / 0.51420.2213 / 0.2197 / 0.2168 / 0.2240
Minimum bin size 5 / 20 / 50 / 1000.5137 / 0.5127 / 0.5235 / 0.50600.2200 / 0.2194 / 0.2288 / 0.2215
Decay fraction 0.1 / 0.3 / 0.4 / 0.50.5102 / 0.5083 / 0.5150 / 0.51220.2190 / 0.2183 / 0.2203 / 0.2211
V. + ECA without warmup0.52470.1279

Two observations. First, V. + WSD without any data curriculum shows the same degradation on Targa '13 as ECA (0.2166 vs. 0.2176), so the degradation is not caused by the curriculum. It comes from the warmup phase: without warmup, ECA reaches 0.1279 on Targa '13. In the submission we used one shared schedule for all nine datasets and did not tune it per dataset. Second, ECA is not sensitive to its own hyperparameters: over bin counts from 10 to 200, minimum bin sizes from 5 to 100 and decay fractions from 0.1 to 0.5, bMASE changes by at most about 3% on PM2.5 and Targa '14 and about 5% on Targa '13.

Following the first observation, we additionally selected the warmup and decay fraction per dataset (warmup in {0.0, 0.1, 0.2, 0.3}, decay in {0.2, 0.4}, selection by validation loss on one seed, then five seeds), for ECA and for V. + WSD. The results are shown in the answer to Q1.

We also ran the same analysis on AgeDB-DIR for the minimum bin size and the bin width (default: minimum bin size 1, bin width 1 year):

VariantOverallManyMedianFew
V. + ECA (default)6.906.247.8910.40
Minimum bin size 56.966.198.1510.92
Minimum bin size 206.876.127.9211.09
Minimum bin size 506.956.178.0411.27
Bin width 2 years6.856.177.8010.64
Bin width 5 years6.926.257.8310.73

The overall MAE is stable in all variants (6.85 to 6.96). The few-shot MAE increases with the minimum bin size (10.40 to 11.27), which is expected: a larger minimum bin size leaves the final distribution less balanced and moves ECA towards V. + WSD (11.89). The bin width has only a small effect.

We want to thank the reviewer for asking both questions which led us to investigating individually tuned baselines and an in depth robustens analysis.

I thank the authors for the new experiments. Q1 is largely resolved and Q2 is helpful. However, UVote is only acknowledged, and Targa’13 still degrades even after tuning. Since there is no borderline option, I keep a weak reject as the closest score, but this is a borderline decision.

Reviewer_pLV5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a training-schedule method called Edge-Case Annealing for deep imbalanced regression. The method combines a data exposure curriculum (DEC) and warmup-stable-decay (WSD) learning rate schedule. The authors conducted extensive experiments on two face age datasets and nine MuVis-DIR datasets. The experiments are solid and show the effectiveness.

Critical strengths
  • The method is simple and can be applied to existing baselines with little overhead.

  • The experiments are extensive and solid, including two face age DIR datasets and nine MuVis-DIR datasets from environment, health and other domains.

Critical weaknesses
  • My major concern is the technical novelty. As already mentioned, the idea of ECA is built on progressive balanced learning in [1], and WSD alone is not new [2]. The major contribution is the continuous coupling of the ceiling to the learning rate. Table 5 show that WSD alone can improve the vanilla method significantly. Though LDS+WSD and SQINV+WSD are included in the appendix, I believe adding these ablations in the main manuscript is necessary. And the authors should also add WSD+other baselines (e.g. ConR and RankSim) with the same training settings in the main manuscript. This can help clarify the contribution.

  • The authors claim the method improves the tail without the majority tax, but combining ECA with UVote worsens many-shot and overall MAE on both age datasets. ConR+ECA also increases many-shot MAE on AgeDB-DIR The authors should reword this claim to reflect the tradeoff.

  • The paper needs more evidence to support the claim of being data agnostic and model orthogonal. The authors should include ConR+ECA in the Table 1 of main manuscript and evaluate additional existing baselines such as RankSim+ECA to strengthen the claim of model orthogonal. And the authors should also evaluate on a text benchmark STS-B-DIR in [3] to provide a fuller picture of being data agnostic.

[1] Kang et al., Decoupling Representation and Classifier for Long-Tailed Recognition, 2020

[2] Hagele et al., Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations. 2024

[3] Yang et al., Delving into Deep Imbalanced Regression, ICML 2021

Questions for authors

Did the authors run an independent grid search for WSD-only, LDS + WSD and SQINV + WSD, or did they reuse the setting selected for ECA?

3 comments

We thank the reviewer for the detailed and constructive feedback. We ran all requested experiments and report them below.

W1 - Technical novelty. We agree that the introduced technique is not a quantum leap in novelty, we would even claim the opposite. Because the technique is simple and grounded in established components, we think it is interesting that it is already this competitive against methods that introduce much heavier machinery, for example in representation learning. To our knowledge we are nevertheless the first to introduce a continuous coupling between a data curriculum and the learning rate.

W1 - Table 5. We agree that WSD is already a better schedule than the one the standard DIR literature uses. But especially in Table 5 we see that WSD is the component that improves the many-shot region (6.67 to 6.20, while the data curriculum alone even worsens it to 6.86), which then also translates to the overall column. The interesting results for DIR are in the medium- and few-shot columns, where WSD is outperformed both times by the data curriculum (8.08 vs. 8.22 and 11.11 vs. 11.67) and where ECA shows its biggest effect.

W1 - LDS + WSD and SQINV + WSD in the main paper. The reason we chose LDS and SQINV is that they outperform methods like RankSim and ConR in the few-shot region, which is the focus of DIR. With the additional space in the camera-ready version we will follow your recommendation and move them to the main paper.

W1 - ConR + WSD and RankSim + WSD. We trained ConR and RankSim with the same training settings as ECA (90 epochs, WSD with warmup 0.2 and decay 0.2), once with WSD only and once with ECA on top (all runs done on AgeDB-DIR):

MethodOverallManyMedianFew
V. + ConR (step, Table 1)7.316.388.5812.51
V. + RankSim (step, Table 1)7.286.398.4912.32
V. + WSD7.036.148.3111.89
V. + ConR + WSD7.016.138.3111.68
V. + RankSim + WSD6.986.148.1211.71
V. + ECA6.906.247.8910.40
V. + ConR + ECA6.836.187.8910.06
V. + RankSim + ECA6.876.207.8410.53

With WSD alone, ConR and RankSim end up at the level of V. + WSD (7.01 and 6.98 vs. 7.03). Adding the data curriculum on top reduces the few-shot MAE by 1.62 (ConR) and 1.18 (RankSim) and the overall MAE by 0.18 and 0.11, while the many-shot MAE stays within one standard deviation (+0.05 / +0.06). We will add these rows to the main paper.

W2 - Majority tax. Thank you for pointing that out. The claim regarding the non-existing majority tax refers to the Vanilla + ECA version (not the combination with other methods like UVOTE) and was intended in contrast to LDS, SQINV, RRT and FDS. We agree that this should be reworded and will adjust it in the camera-ready version.

W3 - ConR+ECA. Regarding ConR + ECA: the row in Table 14 was trained with ConR's original step schedule, i.e. without the coupling to WSD (to avoid confusion we left this in the appendix). In the coupled version above, ConR + ECA is now the best performing version of the ConR method.

W3 - Data agnostic and model orthogonal. The claim of being model orthogonal and data agnostic is more of an architectural nature than an empirical one: ECA only changes which samples are drawn and does not touch the model, the loss or the input, so it does not enforce any type of model or data domain. On the empirical side, the table above now adds ConR + ECA and RankSim + ECA under the same settings, and ECA improves both. We will include both rows in Table 1.

Q – Independent grid search for WSD-only, LDS + WSD and SQINV + WSD. No, in the submission LDS + WSD and SQINV + WSD were trained with one fixed WSD configuration (90 epochs, warmup 0.1, decay 0.2) and were not tuned separately. We have now run the grid of Appendix D.3 independently for all three.

MethodSelected warmup / decayOverallManyMedianFew
V. + WSD (tuned)0.4 / 0.17.016.208.0511.78
V. + ECA0.2 / 0.26.906.247.8910.40
V. + LDS- / -7.697.038.6411.32
V. + LDS + WSD (tuned)0.5 / 0.36.956.457.619.83
V. + LDS + ECA (tuned)0,4 / 0,26.906.507.429.27
V. + SQINV- / -7.697.028.6511.42
V. + SQINV + WSD (tuned)0.4 / 0.46.996.567.439.88
V. + SQINV + ECA (tuned)0,4 / 0,46.876.607.188.94

This additional experiment show that with independent grid search we were able to improve all four configurations (LDS+WSD, LDS+ECA, SQINV+WSD and SQINV+ECA) and both LDS and SQINV now have the same performance hierarchy of V. + LDS < V. + LDS + WSD < V. + LDS + ECA.

We want to thank the reviewer again for their feedback and their questions. We think the additional requested experiments are well justified and will improve the overall paper.

I thank the authors for their efforts on the rebuttal. Given the new tuned results, the authors should revise the statement that ECA is the only addition that improves on WSD. As this is a top-tier ML venue, my main concern about technical novelty still remains. And since the paper claims to be data-agnostic, the authors should also add standard text benchmark STS-B-DIR, which is likely not doable within the rebuttal period. Therefore, I keep my scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.