- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes UP-MOPD, which projects AdamW’s candidate parameter update onto a "feasible cone". Unlike gradient projection, UP-MOPD applies the correction after AdamW produces a candidate update, while retaining the optimizer state computed from the original mixed gradient. Experiments cover medical-general integration and a public math-code-instruction benchmark.
Critical strengths
- The paper identifies a meaningful limitation of gradient-level conflict handling, namely, AdamW can transform a seemingly safe gradient into a harmful parameter update.
- The projection is clearly formulated, has a unique solution, and admits a low-dimensional dual formulation.
- The evaluation covers two-/three-domain settings and includes comparisons with gradient projection, update rejection, and reweighting.
Critical weaknesses
- The empirical evidence is limited by the lack of multiple training seeds. The paper also reports that none of the eight paired metric differences are significant after multiple-comparison correction.
- The Euclidean projection metric is not clearly justified for AdamW.
- First-order feasibility does not guarantee that the actual finite update decreases each domain’s loss.
Questions for authors
- Are the gains consistent across multiple independent training seeds?
- Are domain-gradient passes executed serially or in parallel, and can the authors provide a detailed runtime breakdown?
- What is the effect of retaining the optimizer state computed from the original mixed gradient?
- Personally, I think that the mixed gradient can be obtained by merging domain-specific gradients, which raises a question: why do you perform repeated computation?
3 comments
Thank you for the detailed response and the additional experiments. The clarifications address my questions about the rationale for Euclidean projection, the scope of the first-order guarantee, and the execution and computational cost of the domain-gradient passes. The explanation of the separate mixed-gradient backward pass also clarifies the current implementation choice. The additional training runs provide encouraging evidence of consistency on the public benchmark.
Some minor limitations remain. In particular, repeated runs of the main baselines would strengthen the comparison, and the new loss analysis uses first-order estimates rather than directly measured post-update losses. The benefit of retaining the original optimizer moments could also be investigated more explicitly. Nevertheless, I do not consider these remaining limitations to undermine the paper’s central contribution or its appropriately scoped conclusions. Overall, the rebuttal has sufficiently addressed my main concerns.
Thank you for taking the time to review our response and the additional experiments. We’re glad they helped address your main concerns, and we appreciate your helpful suggestions for strengthening the work further.
Response to Reviewer xaoi
Thank you for your careful review and specific suggestions. We have added results from three training seeds, along with loss and runtime analyses, to address your concerns.
W1 / Q1. Training seeds and consistency of results. Under the same evaluation protocol, UP-MOPD achieves six-task average scores of 34.33, 31.47, and 32.16 across three seeds, all exceeding the reported Open-MOPD score of 30.85. The mean and sample standard deviation across the three seeds are reported below:
These repeated training runs provide additional evidence of consistency across seeds in the public benchmark setting. The conclusions of the BH-adjusted analysis in Table 12 for the medical setting remain unchanged.
W2. Euclidean metric. We adopt Euclidean distance to implement the design objective of minimally modifying the actual parameter displacement already proposed by AdamW. The candidate displacement incorporates momentum, adaptive scaling, and weight decay; projection preserves this update as closely as possible while satisfying the domain constraints. Proposition 2 guarantees a unique correction, leaves feasible candidates unchanged, and establishes that the exact projection does not increase the displacement’s Euclidean norm. Appendix C.4 presents an extension to positive-definite weighted metrics, with the dual Gram matrix. Thus, the Euclidean metric is the minimum-correction choice adopted and evaluated in this work, rather than a claim about the optimal metric for AdamW.
W3. First-order guarantees and finite-step behavior. Corollary 1 guarantees first-order non-increase on the current batch and characterizes finite-step loss changes through a second-order remainder. The new loss figure compares the original updates and their offline UP projections for the same 645 rejected AdamW candidates from the Update Rejection trajectory. A median relative correction of 1.77% removes the predicted increase in the anchor loss while retaining a median of 96.26% of the predicted medical loss decrease. The improvement in aggregate score over Update Rejection in Table 3 (59.15 → 60.03) further supports the value of this local constraint for capability integration in practice. Our theoretical claim is first-order non-increase, rather than domain-wise monotonic loss decrease under finite updates.
Q2. Runtime breakdown. In the public verl implementation, domain gradients are computed sequentially across domains, with distributed parallelism within each pass. The newly added timing figure for the medical setting shows the cost of each stage. The mean timings corresponding to Table 9 are provided below. Actor time includes forward and backward computation, domain-gradient processing, and optimizer updates.
Actor training takes 3.43 times as long as the baseline, while the end-to-end time per step is 1.42 times that of the baseline. The additional actor computation primarily comes from the domain-gradient computations required for parameter projection.
Q3. Optimizer state. AdamW continues to update its momentum using the original mixed gradient. Specifically, we retain the first and second moments computed from that gradient, so AdamW’s historical statistics continue to correspond to the original distillation objective, while the current domain constraints act directly on the committed parameter displacement. The projection correction does not directly enter the moment estimates at the current step. M-OPD and UP-MOPD therefore use the same gradient-computation and state-update rules, and their ablation comparison adds only parameter-displacement projection (59.13 → 60.03 in Table 3). Update Rejection further tests the value of correcting updates relative to skipping them entirely.
Q4. Repeated computation. Thank you for this suggestion. Mathematically, the mixed gradient satisfies g0=∑dqdgd and can be reconstructed from the domain gradients. However, under finite-precision computation such as BF16, gradients aggregated from separate backward passes may differ numerically from those obtained by directly backpropagating the mixed objective. To preserve the original M-OPD gradient-input path, our current implementation passes the result of the mixed backward pass to AdamW and monitors discrepancies through a decomposition-consistency check (Appendix C.3). After assessing the numerical impact, gradient reuse can eliminate this additional backward pass.