acceptodds
Under review as a conference paper at ICLR 2027

Normalize the Update, Not the Weights: Efficient Loss Weighting for Multi-Objective Pretraining

Abstract

Modern deep models are often pretrained with multi-objective losses whose relative weights act as hyperparameters. Selecting these weights through random search or Bayesian optimization is computationally expensive because it requires many independent training runs. We introduce GraP, an online bilevel method that adapts pretraining loss weights to improve downstream performance through gradient alignment. A central challenge is scale ambiguity: uniformly rescaling the weights changes the magnitude of the composite update and therefore the effective learning rate. Rather than normalizing the weights themselves, GraP constrains the norm of the composite update. We show theoretically that this normalization preserves the optimal alignment direction and demonstrate its empirical benefits. To enable efficient optimization, GraP computes per-loss gradients in the shared representation space and reuses their weighted combination, reducing the number of full-backbone backward passes from K to one per iteration. Across event-sequence modeling tasks, GraP improves upon adaptive-weighting baselines and matches or improves upon carefully tuned hyperparameters, while requiring only a single training run with approximately 34% additional training-time overhead. In self-supervised computer vision, GraP remains competitive with standard tuning strategies.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Bilevel optimization, loss weighting, gradient alignment, multi-objective pretraining, representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_x8EDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a bilevel optimization objective for aligning gradients between pretraining and downstream tasks. The actual implementation first takes one gradient ascend step on the normalized projection on to the downstream task gradient with respect to the loss coefficients/weights, updating the coefficients/weights to increase alignment, and then it does one gradient descent on the backbone on the combined loss.

Critical strengths
  1. The objective itself is sensible.
  2. experiments seem to be comprehensive and covering many different domains and settings
Critical weaknesses
  1. I am not sure if normalizing a backward signal is a new idea.
  2. There is no convergence guarantee on constantly shifting loss objective, especially when W can be unbounded.
  3. The experiment results are mixed and improvements are within margin. I am not sure if there is other merits to this method like more hardware efficient, but it does not seem to have comparisons with baseline methods so it remains unclear.

minor complaints on the terminologies: usually weights are referred to as the model parameters, and update referred to the optimizer step, the paper uses rather unconventional wordings and often times would confuse readers.

Questions for authors

Table 4 shows significant slowdowns, but the paper claims "efficiency"?

1 comment

We thank the reviewer for the constructive feedback and for recognizing the motivation of our objective and the breadth of our empirical evaluation.

W1. Novelty. We do not claim that gradient normalization itself is novel. Our contribution is the analysis demonstrating that conventional weight normalization distorts the optimal composite-gradient direction (Fig. 2), and the proposed solution based on composite-gradient normalization (Sec. 3.2).

Unlike Du et al. (2018), who directly use cosine similarity as individual auxiliary loss weights, GraP optimizes the cosine similarity between the composite pretraining gradient and the downstream gradient, accounting for interactions between individual objectives. Compared with OL-AUX, our normalization mitigates uncontrolled weight growth and improves initial learning speed in the RL experiment (App. G). The normalization ablation (Tab. 9) further demonstrates an improvement from 77.48 to 78.05 under identical supervision.

W2. Convergence and unbounded weights. Although the raw loss weights can theoretically grow without bound, the normalized alignment objective is invariant to their uniform positive rescaling. Moreover, the norm of the composite representation-space gradient propagated through the backbone is bounded by construction. Thus, increasing weight magnitudes does not directly lead to exploding backbone gradients.

Nevertheless, we acknowledge that this does not establish convergence of the joint optimization procedure. Obtaining such guarantees for simultaneously evolving model parameters and loss weights is an interesting direction for future work. Empirically, the learned weights generally exhibit smooth and stable trajectories (App. H).

W3. Empirical results and practical benefits. Our objective is not to universally outperform existing MTL methods, but to efficiently select pretraining loss weights targeting downstream performance. GraP achieves the best average performance across event-sequence benchmarks for all three backbones and significantly outperforms most adaptive-weighting baselines after Holm correction. More importantly, it achieves performance comparable to carefully tuned Optuna weights without multiple independent training runs.

We acknowledge that improvements in computer vision are marginal. Nevertheless, GraP identifies a redundant objective (Attn-NNCLR), demonstrating its additional diagnostic value.

Q1. Computational efficiency. The overhead reported in Tab. 4 is relative to a single standard training run. However, our primary computational reference is hyperparameter optimization, which requires multiple runs. GraP introduces an average epoch-time overhead of 34% (17–71% across datasets) while optimizing loss weights online.

Furthermore, compared with a direct implementation of gradient-based loss-weight optimization, our shared-representation approximation and gradient reuse reduce the number of full-backbone backward passes from K to one per iteration. This is particularly beneficial for expensive backbones and composite objectives with many components.

Terminology. We agree that distinguishing loss weights, model parameters, composite gradients, and actual optimizer updates would improve clarity and will revise the terminology accordingly.

We hope these clarifications better establish the methodological and practical contributions of GraP. We would be grateful if the reviewer would reconsider their assessment and consider updating the score.

Reviewer_6XHsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Gradient-aligned Pretraining (GraP), an online bilevel approach for adapting the weights of multiple pretraining losses using downstream-task feedback. Its central idea is to normalize the composite gradient rather than the loss weights, thereby separating the relative contributions of individual objectives from the overall update scale. To reduce computational cost, GraP performs gradient comparisons in the shared representation space and propagates their weighted combination through the backbone once. Experiments cover event-sequence modeling and self-supervised image representation learning, reporting improved average performance on event-sequence benchmarks and performance comparable to the default All4One configuration on vision benchmarks.

Critical strengths
  1. The paper identifies an important distinction between controlling loss-weight magnitudes and controlling the resulting gradient magnitude. As illustrated in Figure 2, constraining the weights does not directly constrain the composite gradient because its norm also depends on the magnitudes and correlations of individual gradients. Formulating the selection criterion around the normalized composite gradient gives this distinction a clear geometric interpretation.
  2. The computational design addresses a practical obstacle to online loss-weight optimization. Combining task gradients at the shared representation before propagating them through the backbone exploits the linearity of gradient aggregation and avoids repeated backbone differentiation. This provides a practical way to incorporate downstream feedback within a single pretraining run.
Critical weaknesses
  1. The empirical benefits vary across datasets and backbones, but the reasons remain insufficiently examined. Although Section 4.1 states that GraP “consistently improves upon the equal-weight baseline,” Table 1 shows essentially unchanged performance for the standard Transformer on Churn (83.50 to 83.46) and a relatively small improvement on Taobao (67.83 to 68.23). Importantly, Churn and Taobao are the two smallest event-sequence datasets, containing approximately 10,000 sequences each (Table 5). Limited data size may affect Transformer training and its ability to benefit from adaptive loss weighting, but this remains an untested hypothesis: the paper provides no learning curves, controlled data-scaling experiments, overfitting analysis, or gradient statistics to investigate this possibility. In vision, GraP also matches the default All4One configuration in average performance. The explanation that All4One is “relatively insensitive to moderate changes in its loss weights” is plausible, but a more systematic analysis is needed to establish which dataset and objective characteristics determine when GraP provides meaningful benefits.
  2. The computational advantage over relevant adaptive-weighting methods is not sufficiently established. The paper emphasizes reducing backbone backward passes “from K to one per iteration” (Section 3.3), but this comparison primarily concerns a direct per-loss implementation. DWA does not require individual task gradients, GradNorm can operate on a restricted shared parameter subset, and MGDA-UB already uses representation-space gradients. Table 4 measures GraP against standard training and estimates the cost of Bayesian/TPE search, rather than directly comparing the time and memory requirements of the adaptive baselines. Consequently, the reported overhead does not establish where GraP offers a favorable performance–computation trade-off relative to these alternatives.
  3. Several implementation choices in the supplied source code do not match the algorithm described in the paper. Section 3.4 states that “The pretraining heads are updated using their unweighted losses,” but the optimizer’s hpo_step aggregates their gradients using rescale_grad_norm_weights. In the supplied default event-sequence configurations, the CoLES entry of this vector is zero, which removes its own loss contribution from the CoLES head’s parameter update. The implementation also restores a reference gradient scale before backbone propagation, rather than simply applying the normalized representation gradient in Algorithm 1. These differences affect the training procedure, so the authors should clarify which implementation and configurations produced the reported results and whether the conclusions hold for the algorithm as written.
  4. The presentation does not provide enough initial intuition for several concepts central to the contribution. The introduction describes “separating the relative contributions of the losses from the scale of the backbone update,” but readers must wait until Section 3.2 to understand the mechanism and why conventional weight constraints are insufficient. A short explaination or an example would make the motivation substantially easier to follow. Similarly, (K) appears in the abstract and introduction before being defined, and the distinctions among representation gradients, parameter gradients, and actual optimizer updates should be made more explicit. These issues obscure the connection between the problem formulation, geometric argument, and implemented algorithm.
Questions for authors
  1. What explains the different behavior across backbones and datasets, particularly the limited Transformer gains on Churn and Taobao and the negligible average improvement in vision? Can targeted analyses distinguish the roles of dataset size, objective relevance, gradient relationships, and sensitivity to loss weights?
  2. Can the authors directly compare training time and peak memory against DWA, GradNorm, MGDA/MGDA-UB, and PCGrad under matched settings, specifying where each method computes its gradients? Such comparisons would clarify the practical efficiency advantage beyond avoiding repeated hyperparameter-search runs.
  3. Why does the supplied implementation use reference weights for head updates, including a zero CoLES entry, despite the unweighted-head-update description in Section 3.4? Please also explain the reference-scale restoration and identify the exact configurations used for the reported experiments.
3 comments

We thank the reviewer for the constructive feedback and for recognizing the conceptual and computational contributions of GraP.

W1/Q1. Performance across datasets and backbones. Churn and Taobao are the two smallest event-sequence datasets, containing approximately 10,000 sequences each (Tab. 5), and exhibit relatively high variability across runs. We hypothesize that limited data provide a weaker signal for downstream-guided weight optimization. Similar dataset-dependent performance of Transformer-based models was observed in prior work (Karpukhin & Savchenko, 2025).

Regarding vision, the comparable performance of default and equal weights suggests that All4One is relatively insensitive to loss weighting. Nevertheless, GraP identifies a low-impact objective (Attn-NNCLR), demonstrating its diagnostic value. We agree that systematically investigating the factors determining the effectiveness of adaptive loss weighting is a promising direction for further research.

W2/Q2. Computational efficiency. We distinguish three categories of methods according to their use of downstream supervision:

  • A. Direct backbone supervision (auxiliary training and SFT).
  • B. Downstream-guided hyperparameter selection (GraP, OL-AUX adapted to pretraining, and Optuna).
  • C. Task-agnostic loss balancing (DWA, GradNorm, MGDA, and PCGrad).

GraP primarily addresses efficient hyperparameter selection (category B). While Optuna requires multiple independent training runs, GraP optimizes loss weights within a single run. Compared with a direct implementation of OL-AUX, GraP reduces the number of full-backbone backward passes from K to one per iteration.

To further clarify the computational trade-offs, we distinguish three representative gradient computation strategies:

  • A single composite gradient (Equal Weights, DWA).
  • Individual parameter-space gradients (full-gradient MGDA, PCGrad, OL-AUX).
  • Individual representation-space gradients (GraP, MGDA-UB).

Our preliminary profiling indicates that computational costs are largely determined by the gradient computation strategy. We benchmark representative methods from each group on AgePred, with K = 4 pretraining loss terms and a Transformer backbone, using an RTX 4060 GPU and a batch size of 16. We exclude 20 warm-up batches and measure 40 subsequent batches under identical settings.

MethodTime/batch (ms)Peak GPU memory (GiB)
Equal Weights200.99 ± 2.772.48
MGDA (full)1845.04 ± 12.084.34
GraP (shared)527.22 ± 7.444.33

Even with only four loss terms, GraP is approximately 3.5 times faster than full-gradient MGDA, with comparable peak GPU memory consumption. As expected, methods that do not require individual gradients are faster and consume less memory.

We acknowledge that shared-representation gradient computation is not unique to GraP. Our contribution is its adaptation to downstream-guided loss-weight optimization, with the corresponding theoretical justification (App. C). Our efficiency claim concerns hyperparameter optimization rather than computational superiority over every adaptive-weighting method.

W3/Q3. Implementation discrepancies. We thank the reviewer for carefully examining the supplementary code. The rescale_grad_norm_weights mechanism, including the zero CoLES entry and reference gradient-scale restoration, is an artifact of an earlier implementation. It was removed in the version used to produce the currently reported results. The current implementation follows Sec. 3.4 and Algorithm 1. We apologize for the discrepancy and have updated the supplementary material to provide the exact implementation and configurations used in the reported experiments.

W4. Presentation. We agree with the reviewer's suggestions and have revised the manuscript accordingly. Specifically, we have defined K at its first occurrence in the abstract, introduced an explanation in the introduction of why conventional weight normalization may favor objectives with larger gradient magnitudes, and explicitly clarified the distinction between representation-space gradients, parameter-space gradients, and actual optimizer updates.

We hope that the additional computational comparison, updated supplementary material, and manuscript revisions address the reviewer's concerns. We would be grateful if the reviewer would reconsider their assessment and consider updating the score.

I appreciate the authors for the clarifications and additional evidence. The newly added computational comparisons and the narrowing of the efficiency claims have addressed my concerns regarding computational cost. The remaining mechanistic analysis of performance discrepancies across different datasets and backbones can reasonably be framed as limitations and future work. The explanation for implementation-version differences resolves my primary questions about implementation consistency. These responses have substantially alleviated my major concerns, and I will reconsider raising my score accordingly.

We sincerely thank the reviewer for reconsidering our work and for the constructive discussion.

Reviewer_wWLgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GraP, an online bilevel method for tuning the weights of a composite pretraining loss so that the learned representation performs well on a designated downstream task. Using a one-step SGD hypergradient with a first-order approximation, each loss weight is updated in proportion to the alignment between its gradient and the downstream gradient. To remove the scale ambiguity of this linear objective, GraP constrains the norm of the composite update rather than the weights. This is equivalent to maximizing the cosine similarity between the composite pretraining gradient and the downstream gradient. For efficiency, the alignment is computed in the shared representation space (gradients with respect to the backbone embedding zz), so only one full backbone backward pass is needed per iteration. A small labeled subset trains a detached downstream head that supplies the downstream gradient. Experiments cover five event-sequence datasets with three backbones and many next-token-prediction (NTP) losses plus a contrastive loss, All4One self-supervised learning on CIFAR-10/100 and ImageNet-100, and an RL auxiliary-task setting. The authors report the best average rank among adaptive-weighting baselines, parity with Optuna-tuned weights, about 34% epoch-time overhead, and identification of a low-impact objective (Attn-NNCLR) in All4One.

Critical strengths

S1. A practical, cheap mechanism that scales to many loss terms. GraP computes per-loss gradients at the shared embedding and propagates only their weighted combination through the backbone. This makes downstream-guided weighting feasible for objectives with up to 19 terms (AlfaBattle) in a single run. It is simple to implement, uses only first-order gradients, and is the paper's main practical contribution. This matters to my assessment because search-based tuning of 10–20 loss weights is otherwise expensive.

S2. The learned weights are a useful diagnostic. The weight trajectories (App. H) are informative in their own right. The All4One experiment, where GraP suppresses Attn-NNCLR and removing that objective barely changes CIFAR performance, is a nice demonstration that the method can expose low-impact components of a composite objective.

Critical weaknesses

W1. The novelty is limited, and the central "normalize the update" motivation is weak in the paper's own training setups.

  • The pieces exist in prior work.
    • Weighting auxiliary losses by the alignment of their gradients with a main-task gradient is the idea of Du et al. (2018), which already uses cosine similarity, and of OL-AUX (Lin et al., 2019), which derives it from a one-step hypergradient.
    • As App. B states, composite-gradient normalization is exactly maximizing the cosine similarity between the composite gradient and the downstream gradient. It is therefore a small change to these methods.
    • Computing gradients at the shared representation is the MGDA-UB approximation of Sener & Koltun (2018), which the paper cites.
    • The contribution is mainly the combination of these elements, and the paper should state this more plainly.
  • The scale-ambiguity argument largely does not apply to the optimizers used. The motivation is that rescaling ww changes the update magnitude and hence the effective learning rate. However, the event-sequence experiments use Adam, which is approximately invariant to gradient scale, and the vision experiments use LARS, which normalizes updates layer-wise. In both cases a uniform rescaling of the composite gradient has little effect on the actual step. The paper should explain why normalization matters under these optimizers. The modest ablation gain (Table 9: 77.94 vs. 77.67 for sum normalization and 77.48 for no normalization, on one backbone, four seeds, overlapping standard deviations) does not establish that it does.
  • Normalization also changes the backbone update. Algorithm 1 also uses the normalized wˉ\bar w for the backbone step (line 8), so GraP trains with a unit-norm composite gradient in representation space, while the baselines use raw gradients. Gains could come from this normalized-gradient training rather than from better weights. A baseline with fixed weights but the same update normalization is needed.
  • The theory is thin.
    • Theorem C.1 is a direct consequence of the operator-norm definition and is one-sided. It does not bound how far the representation-space weights are from the parameter-space optimum.
    • Theorem A.1 holds only for fixed rollout length and vanishing step size.
    • One-step hypergradients are known to suffer from short-horizon bias, which is not discussed.
  • The labeled subset may distort the representation-space objective. g~down\tilde g_{\mathrm{down}} is nonzero only on labeled examples, while G~⊤w\tilde G^\top w spans the whole batch. With small labeled fractions, the norm in the denominator is dominated by unlabeled examples that contribute nothing to the numerator. How this affects the objective is not analyzed. This weakness matters because the paper positions composite-update normalization as its key conceptual contribution, but neither the theory nor the experiments show that it matters in practice.

W2. The comparisons use labels unevenly, and the claimed gains are not robust.

  • Only GraP uses downstream labels during pretraining. The adaptive-weighting baselines (GradNorm, DWA, MGDA, PCGrad) are task-agnostic. The fair references are methods that use the same labels, and against these GraP does not win:
    • joint auxiliary training: 77.80 (Table 9)
    • Equal Weights + SFT: 78.00, which exceeds GraP's 77.94
    • GraP + SFT improves only to 78.20 The practical question, whether GraP beats simply using the labels, is answered weakly at best. OL-AUX and Du et al.-style cosine weighting, the closest label-using methods, are compared only in an ablation (OL-AUX on RL with two seeds; Sec. F reportedly for pretraining) rather than in the main tables.
  • The average gain is driven largely by one dataset where baselines collapse.
    • On MIMIC-III, several baselines degrade badly (e.g., HT-Transformer DWA 81.32, PCGrad 82.65 and GradNorm 84.48, against supervised 91.56), while GraP stays near 91.
    • Excluding MIMIC-III, GradNorm's average is at least as high as GraP's on two of three backbones:
      • Transformer: 73.40 vs. 73.29
      • HT-Transformer: 74.02 vs. 73.71
      • RNN: 74.52 vs. 74.65 for GraP
    • GraP also loses on Churn for all Transformer variants, and on Taobao and AlfaBattle for Transformer and HT-Transformer.
    • A plausible explanation for the MIMIC-III result is the downstream label signal itself (mortality labels guiding the weights), which the baselines lack.
  • The statistics are weak. Results use four seeds. The critical-difference diagram does not show significance against GradNorm after Holm correction, and the reported uncorrected p≈0.0078p \approx 0.0078 should not be emphasized.
  • There is no gain in vision. GraP's 83.49 average equals the default configuration and is below GradNorm (83.54) and PCGrad (83.53) (Table 7). The authors attribute this to All4One being insensitive to weights, which makes vision a weak test bed for the method.
  • The comparison with Optuna is limited.
    • It uses Optuna weights reported by prior work, only for NTP losses without the contrastive term, and with no equal-budget comparison.
    • The "4000%–19000%" cost figures assume 10 runs per weight, which is not measured.
    • A small-budget random or Bayesian search (e.g., 5–10 runs) is the realistic alternative and is missing.
    • GraP is also not free: memory rises by 2.8× on AgePred and 3.7× on AlfaBattle, and throughput drops by up to 79%. This qualifies the "efficient" framing.

Additional Concerns

  • Label leakage. The argument in App. F, that weights evolve smoothly so little label information leaks, is informal. Table 8 shows no clear trend across labeled fractions (Churn: 83.61 at 5%, 84.80 at 20%, 84.00 at 50%), which is consistent with seed noise more than with robustness. It would be cleaner to compare against baselines that receive the same labels (e.g., auxiliary training with each MTL method).
  • Design choices without ablations.
    • The pretraining heads are trained on unweighted losses.
    • The downstream head is trained concurrently on a moving representation, so early downstream gradients come from a poorly fit head.
  • RL experiment. It uses two seeds, and GraP's weights stay near zero (Fig. 5). This makes it unclear whether the speedup comes from the auxiliary tasks or from other implementation differences relative to DDPG and OLAUX.
Questions for authors
  • With Adam (event sequences) and LARS (vision), how does uniform rescaling of the composite gradient affect the effective step? Can you show that the normalization effect persists under SGD, or isolate it under Adam?
  • Can you add a baseline with fixed equal weights that uses the same unit-norm composite update as GraP (Algorithm 1, line 8), to separate the effect of weighting from that of update normalization?
  • How does GraP compare with label-using alternatives in the main tables: Du et al. (2018) cosine weighting, OL-AUX for pretraining, and each MTL baseline combined with auxiliary supervised training or SFT? Given that Equal Weights + SFT (78.00) exceeds GraP (77.94), what is the practical advantage?
  • Why do several baselines collapse on MIMIC-III, and how do the conclusions change if MIMIC-III is excluded or the baselines are re-tuned there?
  • How do GraP's results compare with a small-budget Optuna or random search (e.g., 5–10 runs) on the full objective, including the contrastive loss?
  • Why do GraP's RNN results differ across Tables 1, 8 and 9, and why are several standard deviations exactly zero?
  • How sensitive is the representation-space objective to the labeled fraction, given that g~down\tilde g_{\mathrm{down}} is zero on unlabeled examples but those examples contribute to ∥G~⊤w∥\|\tilde G^\top w\|?

Suggestions: Clarify the novelty relative to Du et al. (2018), OL-AUX and MGDA-UB. Discuss short-horizon bias. Report results with more seeds.

1 comment

We thank the reviewer for the constructive feedback and for recognizing GraP's computational and diagnostic advantages.

W1. Novelty and motivation for normalization (Q1–Q2).

Relation to prior work. Du et al. (2018) directly use non-negative cosine similarity as an auxiliary loss weight in [0, 1]. In contrast, GraP optimizes cosine similarity between the composite pretraining gradient and the downstream gradient, accounting for interactions between individual loss gradients. Our contribution includes identifying the limitations of conventional normalization (Fig. 2) and proposing a solution (Sec. 3.2). App. G also demonstrates that GraP mitigates uncontrolled weight growth in OL-AUX while achieving faster initial learning.

While our shared-representation approximation is inspired by MGDA-UB, MGDA minimizes the composite gradient norm, whereas GraP optimizes alignment between two vectors, including a normalization term. We extend this technique to a different optimization problem with the corresponding theoretical justification (App. C).

Scale ambiguity and adaptive optimizers. Although Adam and LARS reduce sensitivity to uniform gradient rescaling, the normalization issue arises in the loss-weight optimization objective, independently of the backbone optimizer.

Without normalization, weight increments are proportional to the dot products between individual pretraining gradients and the downstream gradient. Thus, objectives with larger gradient norms may receive disproportionately large weights despite weaker directional alignment. Conventional weight-space constraints do not resolve this directional bias (Fig. 2). Our normalization directly targets the best attainable composite-gradient alignment.

Regarding Q2, our event-sequence experiments use gradient clipping with a threshold of 1, while gradient norms before clipping are typically 4–15. Thus, baseline gradient magnitudes are also controlled. We agree that fixed weights with identical normalization would provide an additional control.

Theoretical guarantees. Theorem C.1 provides a one-sided bound justifying the efficient approximation rather than guaranteeing identical solutions in both spaces. We acknowledge short-horizon bias and will discuss this limitation.

W2. Downstream labels and empirical comparisons (Q3).

We respectfully disagree that auxiliary training alone is the most appropriate comparison. We distinguish three levels of downstream supervision:

  • A. Direct backbone supervision (auxiliary training and SFT).
  • B. Hyperparameter selection (GraP and Optuna).
  • C. No downstream supervision (task-agnostic MTL).

GraP belongs to B by design: downstream gradients are detached from the backbone, and labels influence pretraining through gradually evolving loss weights. Limited sensitivity to the labeled fraction (Tab. 8) and additional gains from SFT support this distinction, although neither GraP nor Optuna provides formal guarantees against label leakage.

Our No norm. variant (Tab. 9) is the closest pretraining adaptation of OL-AUX. Under identical supervision, normalization improves average performance from 77.48 to 78.05. GraP also outperforms auxiliary training (77.80), while GraP + SFT achieves the best average result (78.20 vs. 78.00 for Equal Weights + SFT). Moreover, GraP avoids multiple backbone backward passes required by OL-AUX.

Du et al.'s original formulation belongs to A, as it directly includes the main-task gradient in the backbone update.

Q4. MIMIC-III. We hypothesize that appropriate loss weighting is particularly beneficial on MIMIC-III due to its relatively clean and informative features. MIMIC-III and AlfaBattle also exhibit relatively low variance across seeds. While excluding MIMIC-III reduces the advantage over GradNorm, robustness on datasets where conventional weighting methods perform poorly is itself a valuable property.

Q5. Small-budget HPO. The effectiveness of a 5–10-trial search strongly depends on the prior distributions, particularly for objectives with many components. Our Optuna comparison demonstrates competitive performance without such a search.

Q6. Numerical discrepancies. Some ablations were conducted independently, resulting in minor discrepancies. We have updated the tables. Standard deviations of 0.00 reflect rounding of values below 0.005.

Q7. Partially labeled minibatches. Thank you for identifying this subtlety. Unlabeled embeddings contribute to the denominator but not the numerator. Assuming identical data distributions, the expected squared gradient norm scales approximately with the number of samples. We agree that restricting the normalization term to labeled samples during weight optimization would provide a more consistent directional comparison.

We hope these clarifications address the reviewer's concerns and would be grateful if the reviewer would reconsider their score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.