acceptodds
Under review as a conference paper at ICLR 2027

One Size Doesn't Fit All: Depth-aware Manifold-Gated Mixture-of-Experts for Deep Imbalanced Regression

Abstract

Existing representation-based Deep Imbalanced Regression (DIR) methods typically enforce label–feature geometry at a fixed representation depth. This one-size-fits-all strategy can overlook depth-dependent variation in label–feature alignment. We propose Depth-aware Manifold-Gated Mixture-of-Experts (DMG-MoE) to address this depth-dependent variation. It measures Depth-wise Manifold Alignment (DMA) across selected backbone depths by comparing an input's soft feature neighborhood to its soft label neighborhood via KL divergence, and uses this signal to guide the allocation of prediction weight across depth-wise experts. We distill geometry-aware supervision from a training-time DMA teacher into a label-free student gate that produces routing weights without label access at inference. We parameterize routing via a stick-breaking scheme that sequentially allocates prediction mass across depth-wise experts. This enables sample-specific, geometry-guided depth adaptation. On AgeDB-DIR and IMDB-WIKI-DIR, DMG-MoE achieves the lowest overall MAE and GM among the compared methods. We also observe gains on the tabular benchmarks (Abalone, Airfoil) and with two additional backbones (ResNet-152, ConvNeXt-Tiny). The learned routing aligns with its DMA teacher signal, supporting the geometry-aware mechanism.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Imbalanced Regression, Mixture-of-Experts

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oJMrOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper critizes that current Deep Imbalanced Regression (DIR) methods enforce representation constraints at the one fixed model depth. The authors propose "Depth-aware Manifold-Gated Mixture-of-Experts (DMG-MoE)", which attaches a lighweight depth-wise expert to different stages of depth, the idea is that the deeper the layer the more it improves the performance for medium- and few-shot cases. The final prediction is the weighted sum of all experts done with a stick-breaking routing. Their key signal is Depth-wise Manifold Alignment (DMA). During training a DMA teacher compares the label and feature neighborhood via KL-divergence and by that quantifies the alignment with more depth. For inference this is distilled into a label-free gate.

Critical strengths
  • Novel idea, very informative and nicely written/structured paper. Moving from “one depth fits all” to sample-specific depth allocation guided by a geometric signal is a new angle for DIR. The method is clearly motivated (Fig. 2) and cleanly explained.
  • Extensive ablation studies and deep analysis of the isolated components. Also tested on different model types and sizes.
Critical weaknesses
  • No comparison with UVOTE (strong recent DIR method). UVOTE (Jiang et al. 2025) reports for all/many/medium/few in their own paper 6.82/6.55/7.37/7.80 and Puetz et al. (2026) comfirmed in their experiments across 10 seeds similar results with 6.77 ± 0.08/6.38 ± 0.12/7.32 ± 0.28/8.91 ± 0.37. Beating DMG-MoE on all, medium and few-shot in both cases. Also experiments on AgeDB-DIR and IMDB-WIKI-DIR seem to be single runs, so its hard to distinguish significant results when the difference is sometimes small between two values.
  • The method uses Focal-R as the regression loss, which is in itself already considered a DIR-method and which is a deviation from the default DIR setup. To see the effect of DMG-MoE trained on MAE would make it easier to compare it other methods.
Questions for authors
  • Is A3 trained with fixed π\pi or only evaluated with fixed π\pi? The question occurs because A3 numbers are identical to the ResNet-50 Vanilla in Table A.6
  • Can you compare or discuss the result of UVOTE in context to your method?
  • Can you isolate the effect of the Focal-R loss in contrast to using an MAE?
  • Can you report mean and std across multiple seeds for AgeDB-DIR and IMDB-WIKI-DIR?
10 comments
AuthorsOpenReview ↗

Response to Reviewer oJMr

Thank you for your thoughtful review and for recognizing the novelty of our idea, the motivation behind our method, and the presentation of our paper. Your comments encouraged us to further strengthen the experimental comparison by including UVOTE and reporting results over multiple runs.

In the revised manuscript, changes addressing your comments are highlighted in light blue. Changes addressing comments from multiple reviewers are highlighted in light gray.

Q1. Training of A3

(Q1) Is A3 trained with fixed π\pi or only evaluated with fixed π\pi? The question occurs because A3 numbers are identical to the ResNet-50 Vanilla in Table A.6.

Response:

Thank you for this question. In the original manuscript, A3 was trained with fixed π\pi that assigned all prediction weight to the deepest expert. Your question prompted us to reconsider this comparison. We realized that the original setting does not directly examine the effect of routing at prediction time. In the revised analysis, we train DMG-MoE with learned routing and fix the routing weights to the deepest expert only during evaluation. We have clarified this setup in Appendix A.8.2.

W1. Comparison with UVOTE and experimental reliability

(Q2) Can you compare or discuss the result of UVOTE in context to your method?

(Q4) Can you report mean and std across multiple seeds for AgeDB-DIR and IMDB-WIKI-DIR?

Response:

Thank you for pointing out UVOTE as an important baseline. The UVOTE results you cited achieve lower All, Med. and Few MAE than the DMG-MoE results in our original submission on AgeDB-DIR. These results were obtained under different training configurations, which may contribute to the observed differences. Following your suggestion, we have added UVOTE to the comparison and evaluated it on both image datasets. We also report the mean and standard deviation over three runs in Table 1.

The updated results are shown below.

MethodAgeDB All MAEAgeDB Many MAEAgeDB Med. MAEAgeDB Few MAE
UVOTE7.20 ± 0.126.84 ± 0.147.72 ± 0.219.25 ± 0.29
DMG-MoE6.97 ± 0.016.43 ± 0.047.84 ± 0.159.62 ± 0.01
MethodIMDB All MAEIMDB Many MAEIMDB Med. MAEIMDB Few MAE
UVOTE7.50 ± 0.056.99 ± 0.0411.55 ± 0.1521.77 ± 0.32
DMG-MoE7.41 ± 0.026.84 ± 0.0112.01 ± 0.1322.82 ± 0.33

DMG-MoE achieves lower mean All and Many MAE on both datasets, while UVOTE achieves lower mean Med. and Few MAE, suggesting that UVOTE performs better on the less represented groups in these comparisons.

W2. Effect of Focal-R and MAE

(Q3) Can you isolate the effect of the Focal-R loss in contrast to using an MAE?

Response:

Thank you for this suggestion. To isolate the effect of Focal-R, we compare DMG-MoE trained with MAE and Focal-R, while keeping the other settings unchanged. The results report the mean and standard deviation over three runs. Compare to MAE, Focal-R reduces the mean All and Few MAE on both datasets, with larger reductions in the Few-shot groups.

Regression lossAgeDB All MAEAgeDB Few MAEIMDB All MAEIMDB Few MAE
MAE7.3581 ± 0.142913.7393 ± 0.37367.9119 ± 0.051927.4010 ± 1.0183
Focal-R7.2178 ± 0.094512.5110 ± 0.15437.8698 ± 0.068825.7327 ± 0.5381

Additionally, following your suggestions and those of the other reviewers, we also evaluated combinations of DMG-MoE with other DIR methods. The combinations with PRIME and RankSim greatly improve upon the corresponding individual methods on both datasets (Table A.21). Based on these results, we finally select DMG-MoE with PRIME for the revised main comparison and use MAE instead of Focal-R as the regression loss.

MethodAgeDB All MAEAgeDB Few MAEIMDB All MAEIMDB Few MAE
PRIME + SQInv7.3233 ± 0.115710.4990 ± 0.71057.5312 ± 0.011722.5945 ± 0.2348
DMG-MoE + PRIME + SQInv6.9684 ± 0.01039.6228 ± 0.01327.4143 ± 0.024422.8183 ± 0.3293
RankSim + SQInv7.2415 ± 0.052710.1696 ± 0.26427.4284 ± 0.033622.8820 ± 0.7515
DMG-MoE + RankSim + SQInv7.1419 ± 0.062410.0986 ± 0.35817.3433 ± 0.039222.7652 ± 0.3938

Thank you again for your constructive comments and suggestions. We hope that our responses and revisions address your concerns. If you have any further experiments or analyses you would find helpful, we would be glad to provide them and discuss the results with you to support your assessment of our work.

Thanks for the reply, two follow up questions:

  1. How does your training configurations for UVOTE differ from other UVOTE implementations?
  2. I don't understand the results for the MAE and Focal-R comparison. The response says that all settings are unchanged and in the paper it says that focal-r is used as the regression loss, how does the results for Focal-r now differ so much to the numbers in the paper? For example, in the rebuttal you are reporting 12.511 for few-shot on AgeDB but in the paper you are reporting 9.62

I also just realized you changed the text in A.5 describing which loss you used. In the original version you reported Focal-R and in the revised version you are reporting "absolute error". Why have you done this? In the first table in the rebuttal you report that DMG-MoE + MAE actually worsens few-shot performance on AgeDB and IMDB in comparison to Vanilla.

Reviewer_cEjFOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1(−1)
Read review
Paper summary

The paper proposes Depth-aware Manifold-Gated Mixture-of-Experts (DMG-MoE) for deep imbalanced regression. The method attaches regression heads to different depths of a backbone network and combines their predictions using stick-breaking routing weights. During training, a label-dependent teacher computes Depth-wise Manifold Alignment (DMA) scores by comparing soft label and feature neighborhoods within each mini-batch. These scores are then distilled into a label-free gate, along with a separate auxiliary loss encouraging specialization among depth-wise experts. The authors conducted experiments on two standard face age DIR benchmarks, AgeDB-DIR / IMDB-WIKI-DIR, and two tabular datasets, Abalone / Airfoil. The authors also evaluate the method with three backbones including ResNet-50, ResNet-152, and ConvNeXt-Tiny on AgeDB-DIR.

Critical strengths
  • The proposed method is simple and effective, with small computational overhead.
  • The experiments are extensive and demonstrate the effectiveness of the method on the benchmarks.
Critical weaknesses
  • The baseline experimental results seem not consistent. On AgeDB-DIR, Table 1 reports 7.67 as Vanilla overall MAE, but Table A.6 reports 7.20. It’s not a practice to use two sets of numbers in one table. I encourage the authors to consider changing all numbers to the reproduced ones in Table 1.

  • The empirical improvement in Table 1 are small, such as overal MAE on IMDA-WIKI-DIR (7.69 -> 7.57) and few-shot region MAE (9.81->9.63) on AgeDB-DIR. There are also issues related to reproduced baselines in point 1. The authors should report the mean and standard deviation over multiple runs to strengthen the empirical contribution.

  • I feel that the claim that the teacher guides routing is not strongly supported. The authors could further report the gap between teacher and student routing probabilities, instead of only reporting their rank correlation, to see how closely the student follows the teacher.

  • Missing or incomplete comparison with related work. The paper discussed PRIME [1], which also studies distribution alignment and reports strong results in the published version, but did not include it in the experimental comparison. Also, [2] and [3] study expert routing or aggregation for imbalanced classification problems. The authors should discuss how their approach differs from these related expert-based methods for imbalanced regression.

[1] Lim et al., PRIME: Deep Imbalanced Regression with Proxies. ICML 2025.

[2] Wang et al., Long-tailed Recognition by Routing Diverse Distribution-Aware Experts. ICLR 2021.

[3] Zhang et al., Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognition. NeurIPS 2022.

Questions for authors
  • Can the authors explain why the performance difference of ResNet-50 Vanilla baseline on AgeDB-DIR between Table 1 (7.67) and Table A.6 (7.20)? Are these results obtained under the same training settings? Are the results in Table A.6 from the authors' reproduction?

  • Can the authors report the mean and standard deviation over multiple runs for the experiments, especially for the vision datasets?

8 comments
AuthorsOpenReview ↗

Response to Reviewer cEjF

Thank you for your thoughtful review and for recognizing DMG-MoE as a simple and effective method. Your comments helped us strengthen the experimental evidence for DMA and improve our experimental comparisons. The discussion of other mixture of experts methods you suggested also inspired us to consider new directions for future extensions of the DMG-MoE framework.

In the revised manuscript, changes addressing your comments are highlighted in light green. Changes addressing comments from multiple reviewers are highlighted in light gray.

W1. Consistency of baseline results

(Q1) Can the authors explain why the performance difference of ResNet-50 Vanilla baseline on AgeDB-DIR between Table 1 (7.67) and Table A.6 (7.20)? Are these results obtained under the same training settings? Are the results in Table A.6 from the authors' reproduction?

Response:

Thank you for pointing out this inconsistency. The Vanilla result of 7.67 in the original Table 1 was taken from the baselines reported by SRL. The result of 7.20 in Table A.6 came from our incorrect reproduction of Vanilla. It was obtained under different training settings and did not accurately reproduce the Vanilla baseline.

Following your suggestion, we reproduced the comparison methods under a common training protocol and report the mean and standard deviation over three runs in Table 1. The updated Vanilla result on AgeDB-DIR is 7.45 ± 0.10, and the same result is now reported in Table A.5.

W2. Experimental reliability and performance improvements

(Q2) Can the authors report the mean and standard deviation over multiple runs for the experiments, especially for the vision datasets?

Response:

Thank you for this suggestion. We have updated Tables 1 and 2 to report the mean and standard deviation over three runs for all four datasets.

In the new experiments under a common training protocol, DMG-MoE achieves an All MAE of 6.97 ± 0.01 on AgeDB-DIR and 7.41 ± 0.02 on IMDB-WIKI-DIR, compared with 7.51 ± 0.09 and 8.18 ± 0.04 for SRL, respectively. SRL was the strongest baseline in the original submission, as noted in your review. On the Few-shot group of AgeDB-DIR, DMG-MoE achieves 9.62 ± 0.01, compared with 11.98 ± 0.21 for SRL. These results provide an updated assessment of the empirical improvements of DMG-MoE over SRL.

However, following your suggestions and those of the other reviewers, we also added PRIME and UVOTE as new baselines to the main comparison. On AgeDB-DIR, they achieve All MAEs of 7.32 ± 0.12 and 7.20 ± 0.12, respectively. On IMDB-WIKI-DIR, they achieve 7.53 ± 0.01 and 7.50 ± 0.05, respectively. DMG-MoE achieves lower mean All MAE on both datasets. However, a performance gap remains on the Few-shot groups compared with the best-performing baselines.

W3. Teacher and student routing probabilities

Response:

Thank you for this suggestion. We compared the teacher target and student routing probability for the same sample at each gate on AgeDB-DIR. The following table reports the mean absolute probability gap over the 1,472 valid test samples. We report the mean and standard deviation over three runs.

GateMean absolute probability gap
Gate 10.0555 ± 0.0017
Gate 20.0784 ± 0.0025
Gate 30.1121 ± 0.0030

To examine how these probability gaps affect prediction, we replaced the student probabilities at all three gates with the corresponding label-dependent teacher probabilities while keeping the trained experts unchanged. On the valid samples, this increased the All MAE by 0.1003 ± 0.0527 over three runs.

AuthorsOpenReview ↗

Response to Reviewer cEjF (continued)

W4. Comparison with PRIME and related expert methods

Response:

Thank you for pointing out these related methods.

(1) Comparison with PRIME.

We did not include PRIME in the original comparison because we could not find a link to its official implementation in the paper. Following your suggestion, we implemented PRIME based on its description in the paper and evaluated it on both image benchmarks with three seeds. Following the PRIME paper, we use SQInv weighting for PRIME on both datasets. The results are shown below, with the complete results in Table A.21 of Appendix A.11.

MethodAgeDB-DIR All MAEIMDB-WIKI-DIR All MAE
DMG-MoE7.3581 ± 0.14297.9119 ± 0.0519
PRIME + SQInv7.3233 ± 0.11577.5312 ± 0.0117
DMG-MoE + PRIME + SQInv6.9684 ± 0.01037.4143 ± 0.0244

PRIME focuses on representation learning. It uses learnable proxies to organize the feature space and improve the representations used for prediction. DMG-MoE provides a depth-aware routing framework. It uses DMA to guide how much each depth-wise expert contributes to the prediction for a given sample.

The two methods address DIR from orthogonal perspectives. In addition to comparing PRIME and DMG-MoE individually, we also evaluated their combination. DMG-MoE + PRIME achieves lower All MAE than either method on both datasets. It reduces the All MAE from 7.3233 to 6.9684 on AgeDB-DIR and from 7.5312 to 7.4143 on IMDB-WIKI-DIR compared with PRIME. These results demonstrate the potential of combining the two approaches.

(2) Comparison with RIDE and SADE.

RIDE, SADE, and DMG-MoE all use multiple experts to address imbalanced data, but their experts and routing serve different purposes. RIDE and SADE focus on long-tailed classification. RIDE trains diverse classification experts and routes ambiguous samples to additional experts. SADE trains experts for different class distributions and aggregates them to handle an unknown test distribution.

DMG-MoE focuses on imbalanced regression with continuous targets. Its experts are attached to different depths of a shared backbone, so they make predictions from representations with different levels of abstraction. A DMA-guided gate assigns sample-specific weights to these predictions. This allows a sample with a well-aligned shallow representation to receive more contribution from a shallow expert.

The expert routing and aggregation ideas in RIDE and SADE may also be adapted to regression by redesigning their experts and distribution-aware components for continuous targets. Conversely, the depth-wise routing framework of DMG-MoE may be applicable to classification if the label neighborhood used by DMA is adapted to discrete classes. We have added this comparison to Section 2 of the revised manuscript.

Thank you again for your valuable suggestions and inspiring recommendations, which have helped us improve our work. We sincerely hope that our responses and revisions have addressed your concerns. If you have any further suggestions, we would be glad to continue the discussion with you.

I thank the authors for the rebuttal, but the revision and rebuttal raises serious concerns. Therefore I lower my score.

  1. The “DMG-MoE” results in revised Table 1 are combined with PRIME + SQInv, which is only stated in the appendix. In the original submission, the methods is presented as trained with a Focal-R loss, with no mention of PRIME, and reported nearly identical numbers as in revised DMG-MoE + PRIME + SQInv (All 6.97 / Few 9.63 vs. All 6.97 / Few 9.62 on AgeDB-DIR). The original reported best setting (DMG-MoE + Focal-R loss) now gives 7.22 / 12.51 in Table A.21.

  2. Standalone DMG-MoE is worse than Vanilla in the few-shot region on both datasets. The authors mentioned PRIME is implemented during rebuttal period, but now it becomes the major module for the improvement.

  3. The claim that the teacher guides routing is weakly supported, as replacing student routing with teacher routing increases MAE.

Reviewer_VnJQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a model called Depth-aware Manifold-Gated Mixture-of-Experts (DMG-MoE) for deep imbalanced regression. It measures the alignment between label and feature neighborhoods at different network depths using KL divergence, and uses this information to train a label-free gating network. The gating network uses stick-breaking to assign prediction weights to experts at different depths. Empirical evaluations are conducted on four benchmark datasets (AgeDB-DIR, IMDB-WIKI-DIR, Abalone, and Airfoil), with additional experiments using different backbone architectures.

Critical strengths
  1. Clear motivation and meaningful idea

The paper presents an interesting idea that different samples may benefit from different representation depths in deep imbalanced regression. The proposed method uses manifold alignment to guide sample-specific routing. This is a meaningful direction beyond methods that use only the final representation.

  1. Good experimental results and detailed ablation studies

The proposed method improves overall MAE on all four datasets compared to the reported baselines. The ablation studies also show that the DMA teacher, adaptive routing, and stick-breaking design contribute to the performance gains. These results provide useful support for the proposed approach.

Critical weaknesses
  1. Limited evidence that DMA identifies the best depth for prediction

The paper assumes that higher manifold alignment indicates a more reliable representation. However, it is not clear whether higher DMA scores consistently lead to lower prediction errors. Also, the method directly uses DMA scores as targets for stick-breaking gates, but the reason for this design is not fully justified. This weakens the main motivation of the proposed method.

  1. Missing important baseline comparisons and limited experimental reliability

The paper does not compare with PRIME (ICML 2025), although PRIME is closely related to the proposed method and also uses KL divergence to match label and feature neighborhoods. This missing comparison makes it difficult to fully assess the effectiveness and novelty of the proposed approach. In addition, the performance improvements on the image benchmarks are relatively small, and results from multiple random seeds are not reported. This makes it difficult to judge the reliability of the improvements.

Questions for authors
  1. Why are absolute DMA scores used as targets for stick-breaking probabilities? How does this design encourage the model to select the most reliable depth?

  2. Is there a clear relationship between DMA scores and actual prediction errors across different depths? Could the authors report a quantitative analysis over the full evaluation set, rather than only selected examples?

  3. Why is PRIME (ICML 2025) not included in the experimental comparisons? Since PRIME also uses KL-based neighborhood alignment, a direct comparison would help clarify the benefits of the proposed method.

  4. Could the authors report mean and standard deviation over multiple random seeds?

4 comments
AuthorsOpenReview ↗

Response to Reviewer VnJQ

Thank you for your thoughtful review and for recognizing the motivation and potential of sample-specific depth routing. Your questions encouraged us to provide further evidence for the effectiveness of DMA and improve our comparisons. They also inspired us to explore a promising combination of DMG-MoE and PRIME. Below we clarify the design of DMG-MoE and provide the requested analyses and multi-run comparisons.

In the revised manuscript, changes addressing your comments are highlighted in light orange. Changes addressing comments from multiple reviewers are highlighted in light gray.

W1. Evidence for the relationship between DMA scores and prediction errors, and how we use DMA for depth-aware routing

(Q2) Is there a clear relationship between DMA scores and actual prediction errors across different depths? Could the authors report a quantitative analysis over the full evaluation set, rather than only selected examples?

Response:

We analyzed the relationship between DMA scores and prediction errors at each depth using all 2,140 images in the AgeDB-DIR test set. DMA scores are computed offline using test labels and shared test set neighborhoods. The results are reported in Appendix A.12 of the revised manuscript.

Using DMG-MoE, we computed the correlations between DMA scores and negative absolute prediction errors across all test images at each depth. The table reports the mean Spearman and Pearson coefficients over three seeds. For each coefficient, we also report the largest two-sided p value among the three seeds.

DepthSCCp valuePCCp value
L10.3974.158×10−524.158\times 10^{-52}0.4162.381×10−562.381\times 10^{-56}
L20.4621.310×10−1061.310\times 10^{-106}0.5052.154×10−1272.154\times 10^{-127}
L30.6291.702×10−2121.702\times 10^{-212}0.7145.976×10−2935.976\times 10^{-293}
L40.6811.150×10−1991.150\times 10^{-199}0.7486.973×10−2476.973\times 10^{-247}

Both SCC and PCC are positive at every depth, showing that samples with higher DMA scores tend to have lower prediction errors. These correlations are statistically significant across all three seeds at every depth, with all pp values below 10−5010^{-50}. The association is already present at L1, indicating that DMA provides information about prediction quality even at a shallow depth.

(Q1) Why are absolute DMA scores used as targets for stick-breaking probabilities? How does this design encourage the model to select the most reliable depth?

Response:

As shown in our response to Q2, DMA scores are associated with expert prediction quality at every depth across the full test set. Figure A.7 further illustrates the relationship between DMA scores and expert errors across depths for individual samples. Together, these results support using DMA to guide depth-wise routing.

We use the DMA score at each depth directly as the target for the corresponding gate. A higher DMA score provides a higher target for the gate, encouraging it to allocate a larger fraction of the remaining prediction weight to the current expert. Through stick-breaking, the remaining weight is passed to deeper experts, with the final expert receiving the remainder. This produces a sample-specific mixture of predictions across depths.

The following AgeDB-DIR results summarize the ablations of DMA supervision and stick-breaking (Table 3). Removing or shuffling the DMA teacher increases the prediction error, indicating that DMA supervision contributes to the learned routing. Replacing stick-breaking with softmax also increases the error, supporting the use of stick-breaking for depth-wise prediction.

MethodAll MAE
DMG-MoE6.9684 ± 0.0103
Without DMA teacher7.1605 ± 0.0772
Shuffled DMA teacher7.1705 ± 0.0587
Softmax routing7.1437 ± 0.0210

Due to the character limit, we have to continue our response to your remaining comments in the next comment. We apologize for the inconvenience.

AuthorsOpenReview ↗

Response to Reviewer VnJQ (continued)

W2. Comparison with PRIME and experimental reliability

(Q3) Why is PRIME (ICML 2025) not included in the experimental comparisons? Since PRIME also uses KL-based neighborhood alignment, a direct comparison would help clarify the benefits of the proposed method.

Response:

Thank you for pointing out PRIME as an important related method. We did not include it in the original comparison because we could not find a link to its official implementation in the paper. We have since implemented PRIME and evaluated it under our training protocol on both image benchmarks with three seeds. Following the PRIME paper, we use SQInv weighting for PRIME on both datasets. The results are shown below, with the complete results in Table A.21 of Appendix A.11.

MethodAgeDB-DIR All MAEIMDB-WIKI-DIR All MAE
DMG-MoE7.3581 ± 0.14297.9119 ± 0.0519
PRIME + SQInv7.3233 ± 0.11577.5312 ± 0.0117
DMG-MoE + PRIME + SQInv6.9684 ± 0.01037.4143 ± 0.0244

PRIME focuses on representation learning. It uses learnable proxies to organize the feature space and improve the representations used for prediction. DMG-MoE provides a depth-aware routing framework. It uses DMA to guide how much each depth-wise expert contributes to the prediction for a given sample.

The two methods address DIR from orthogonal perspectives. In addition to comparing PRIME and DMG-MoE individually, we also evaluated their combination. DMG-MoE + PRIME achieves lower All MAE than either method on both datasets. It reduces the All MAE from 7.3233 to 6.9684 on AgeDB-DIR and from 7.5312 to 7.4143 on IMDB-WIKI-DIR compared with PRIME. These results demonstrate the potential of combining the two approaches.

(Q4) Could the authors report mean and standard deviation over multiple random seeds?

Response:

We now report the mean and standard deviation over three runs for all four datasets in Tables 1 and 2 of the revised manuscript.

We thank you again for your thoughtful questions and suggestions. Your valuable feedback has given us the opportunity to strengthen the soundness of our work. We hope that our responses and revisions have fully addressed your concerns and will help improve your assessment of our work. If you have any further questions or suggestions, we would be glad to discuss them with you.

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer VnJQ

Dear Reviewer VnJQ,

Thank you again for your thoughtful review. With less than a day left in the discussion period, we would like to briefly recap our replies to your questions.

We analyzed the relationship between DMA scores and prediction errors at each depth across the full AgeDB-DIR test set and clarified how DMA guides depth-aware routing. We also compared DMG-MoE directly with PRIME, evaluated their combination, and reported means and standard deviations over three runs in Table 1 of the revised manuscript.

If these clarifications and results have fully addressed your concerns, we would be grateful if you could consider updating your assessment. We would be glad to discuss any remaining questions with you.

We wish you a wonderful day.

Best regards,

The authors

I thank the authors for their detailed responses and additional experiments. While the new analyses address some concerns, the relationship between DMA scores and optimal depth selection remains insufficiently established. Moreover, PRIME outperforms standalone DMG-MoE on both image benchmarks, leaving the independent effectiveness of the proposed method unclear. Therefore, I maintain my original rating.

Comments on the paper

2 comments

Dear reviewers and authors,

As the discussion period is limited, please follow up on the latest responses and comments in a timely manner to address any remaining concerns. Thank you!

Dear Area Chair,

Thank you for your reminder. We have been working continuously to address the reviewers' concerns. However, some experiments are taking longer than expected. For example, runs on datasets such as IMDB-WIKI and baselines like ConR are time-consuming. In addition, using three seeds for the comparisons means that we also need to run the other candidates in our ablation and backbone-transfer experiments with three seeds to keep the experimental settings consistent.

We take the reviewers' suggestions very seriously and are grateful for the opportunity their feedback gives us to improve our manuscript. We are working hard to complete these experiments as soon as possible and aim to submit our response within two days, so that there is still time for further discussion.

We apologize for the delay and any inconvenience it may have caused you and the reviewers. Thank you for your patience and understanding.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.