acceptodds
Under review as a conference paper at ICLR 2027

Loss Smoothing for Stable Adaptation Under Distribution Shift

Abstract

Over their life cycles, neural networks are often adapted under distribution shift. Standard adaptation methods typically optimize the target objective directly, inducing an abrupt change from the source training objective. This abrupt change can hurt optimization or distort learned representations, including features that may still be useful for the new task. We investigate how softening this transition can improve adaptation. Specifically, we formulate *loss smoothing* as a simple approach that linearly interpolates between the source and target objectives. We show that retaining the source supervision can improve target adaptation even when preserving source-task performance is not a goal. A controlled toy study shows how smoothing sustains learning of features shared between source and target tasks. In pretrained vision and language models, smoothing limits the loss of source information during adaptation, with retention advantages that persist after source supervision ends. Across a broad variety of adaptation settings, including pretrained vision model adaptation, continual supervised learning, fine-tuning "overtrained" large language models (LLMs), and offline-to-online reinforcement learning, loss smoothing improves performance, suggesting that smoother objective transitions are a broadly useful tool for model adaptation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Adaptation, Transfer Learning, Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sfXKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes loss smoothing, a method that linearly interpolates between source and target objectives during neural network adaptation. The core claim is that retaining source supervision improves target adaptation even when source-task performance is not a goal. A toy study isolates the gradient reweighting mechanism, and experiments span four settings: continual supervised learning, pretrained vision adaptation, LLM fine-tuning, and offline-to-online RL. The paper systematically rules out simpler explanations and shows retention advantages persist after source supervision ends

Critical strengths

The primary strength is the breadth of empirical coverage. Loss smoothing is a single, simple recipe tested across four genuinely distinct adaptation settings (vision, language, RL, and continual learning) with consistent directional improvements. This cross-domain validation is rare and supports the "broadly useful" framing more convincingly than a deep dive into one setting would. The systematic ablation controls (Section 5.3) ruling out KL regularization, EWC, LP-FT, and coefficient-only schedules as alternative explanations are well-designed and strengthen the claim that the source signal itself (not slower learning or initialization preservation) drives the benefit.

Critical weaknesses

"Conflicting features" lack a general definition. The shared vs. conflicting feature distinction is only formalized within the toy study, where it is separable by construction. No general mathematical definition is provided for arbitrary networks. The paper acknowledges the two "cannot be disentangled" in practice, yet the mechanistic narrative is built on this distinction.

Limited model scale undermines generalizability claims. The paper claims loss smoothing is "a broadly useful tool for model adaptation" but the largest trained model is OLMo-1B. Vision experiments use ViT-Base (~86M params), RL experiments use small MLPs. No experiment exceeds 1B parameters. The paper does not discuss scalability to larger models nor acknowledge model scale as a limitation. Domain breadth is a strength, but scale coverage is insufficient to fully support the generalizability claim.

Inconsistent and insufficient statistical evidence. No formal significance tests (paired t-tests, Wilcoxon, etc.) are reported anywhere in Section 4. Statistical reporting varies across experiments: RL results use 95% bootstrap intervals over 5 seeds (adequate), continual CIFAR uses SEM over 10 seeds (adequate), but OLMo LLM results rely on only 3 seeds with SEM values not numerically stated in the main text making it impossible to verify whether the 1.9–2.3 pp improvement margins exceed variance. The paper relies entirely on visual inspection of error bars with no formal hypothesis testing, which is insufficient to support the breadth of claims made.

Compute cost acknowledged but not quantified. The limitations section mentions smoothing "approximately doubles compute" but provides no concrete measurements in the core paper. Appendix H contains some useful data on token exposure and image overhead that should be in the core paper, not buried in supplementary material.

No code released. The paper provides no code repository, anonymized link, or supplementary code. A text-only Reproducibility Statement points to Appendix A for hyperparameters and training details, which are reasonably thorough. However, given four distinct experimental settings (vision, LLM, RL, continual learning), each with its own codebase, dependencies, and implementation subtleties, text-based descriptions alone significantly raise the reproduction barrier.

Minor: Undefined shorthand at lines 74–75. The contribution bullet references benefits coming "not from simpler alternatives" without specifying what these are. The reader only learns the paper means KL regularization, EWC, LP-FT, and coefficient-only control much later. These should be briefly characterized inline.

Minor: Core evidence placed in appendix. Tables 6 and 7 are in the appendix but directly cited in Section 4 as primary evidence.

Minor: Missing related work coverage. The paper does not discuss or compare against adversarial adaptation methods or assessor-guided/meta-learning sample-weighting approaches: both relevant to the problem of managing task transitions under distribution shift.

Questions for authors

Can you provide formal significance tests (e.g., paired bootstrap or Wilcoxon) where improvement margins appear comparable to variance?

How do you expect loss smoothing to behave at model scales beyond 1B parameters? Is there any theoretical or empirical reason to believe the gradient reweighting mechanism scales, or could the shared/conflicting feature dynamics change qualitatively at larger scale?

Can you provide concrete cost guidance?

2 comments

We thank the reviewer for their feedback and are glad they found the breadth of our evaluation valuable and our controls effective at isolating the role of the source signal. We address their concerns and questions below.

Definition of shared feature
Defining shared and conflicting features in general is difficult. Unlike in the toy example, we cannot say that everything the model computes from one part of the input is shared and everything computed from the other part is conflicting. However, our analysis of the toy example is mostly based on a gradient decomposition, which we think can be recreated in the general case. For example, given the gradients gtg_t and gsg_s of the target and source losses, the shared part can be defined as the part along the gt∣gt∣+gs∣gs∣\frac{g_t}{|g_t|} + \frac{g_s}{|g_s|} direction, which improves both losses, and a conflicting part as the one along the gt∣gt∣−gs∣gs∣\frac{g_t}{|g_t|} - \frac{g_s}{|g_s|} direction which trades one loss for another. Loss smoothing reweights these two parts: its update αgt+(1−α)gs\alpha g_t + (1−\alpha) g_s gives the shared part the weight α∣gt∣+(1−α)∣gs∣2\frac{\alpha|g_t| + (1−\alpha)|g_s|}{2} and the conflicting part the weight α∣gt∣−(1−α)∣gs∣2\frac{α|g_t| − (1−α)|g_s|}{2}. Applied to the toy example, this decomposition places the conflicting part mostly in the MNIST branch and the shared part mostly in the CIFAR branch. We think this analysis can be generalized to the more practical settings and will incorporate it into the final version of the paper.

Loss Smoothing Scale
While we appreciate the need to demonstrate scale, as you mentioned our experiments have already covered several orders of magnitude of scale, starting with small MLPs for RL/continual learning (300k params), to standard Vision Transformer architectures (~86M params), to LLMs (1B params). We limited our experiments to 1B for LLMs both because of practical compute limitations. We will run some additional experiments testing loss smoothing with 7B parameter LLMs to analyze the transferability to larger scales.

Significance Tests
We follow the reporting practice recommended by Agarwal et al. (2021) for small numbers of runs: interval estimates of the effect rather than binary significance tests. Results are reported as 95% bootstrap intervals for RL and DomainNet, and as SEM for CIFAR and OLMo. The key comparisons are not close calls:

  • RL: the overall IQM intervals of smoothing and the strongest baseline do not overlap.
  • DomainNet: every smoothing duration’s interval 25% and above lies above the hard switch’s (Table 6).
  • Continual CIFAR: formal tests give the same conclusions. Welch’s t-test over 10 seeds gives p < 10⁻⁵ for shuffled labels and p = 0.01 for permuted inputs.

For OLMo at the final checkpoint (smoothing vs hard switch):

  • Anthropic-HH: 61.8 ± 0.3 vs 59.5 ± 0.2 (SEM) (paired t-test p = 0.015).
  • Tulu-v1: 56.1 ± 1.4 vs 54.2 ± 0.8 (paired t-test p = 0.46).

We have also added the specific SEM numbers as a table in Appendix C.3.

Cost Guidance
The “approximately doubles” statement applies only within the smoothing window. Over a full run, the overhead scales with the window length. On DomainNet, measured training time rises from 1.86 to 2.14 hours (+15%) at our default 25% window, comparable to the KL baseline (2.25 hours), and only 1.91 hours (+3%) for a 5% window, which retains approximately half of the target gain. For OLMo, the 128-update window adds 26.5% tokens.

We also add a new ablation (Appendix H.3) where we examine the effect of using fewer resources when smoothing. Mixing by sampling, where source examples fill part of a fixed-size batch, removes the extra forward/backward pass entirely and matches loss mixing on DomainNet target accuracy (76.81 vs 76.77 IQM at 25%, see Appendix H). Only 1,024 source images are needed to keep almost all of the target gain (76.69). We have added this discussion and the new sample mixing results to Appendix H.

Code Release
We do plan on releasing the full codebase upon final release of the paper.

Other minor comments
Thank you for pointing out the other small issues with the manuscript. We have incorporated the feedback into the manuscript.

We hope this addresses your concerns. If it does, we would appreciate your reconsidering the score. We are happy to answer any remaining questions.

[1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021): 29304-29320.

I thank the authors for the rebuttal and the additional results, but I am keeping my rating, as the new information does not substantially change my overall assessment of the paper.

Reviewer_gwjVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. The paper evaluates a simple adaptation principle across four substantially different settings, including vision, language models, continual learning, and reinforcement learning.

  2. The controlled toy experiment provides a plausible explanation for how source supervision can prevent gradient starvation of shared features. Additional controls compare smoothing with replay, learning-rate warmup, EWC, and KL regularization.

  3. The study includes analyses of source retention, adaptation duration, and sensitivity to parameter updates. The authors also report cases where the gains are limited, making the empirical conclusions more informative.

Critical strengths
  1. The paper evaluates a simple adaptation principle across four substantially different settings, including vision, language models, continual learning, and reinforcement learning.

  2. The controlled toy experiment provides a plausible explanation for how source supervision can prevent gradient starvation of shared features. Additional controls compare smoothing with replay, learning-rate warmup, EWC, and KL regularization.

  3. The study includes analyses of source retention, adaptation duration, and sensitivity to parameter updates. The authors also report cases where the gains are limited, making the empirical conclusions more informative.

Critical weaknesses
  1. Source replay and gradual annealing of source supervision have been studied previously. The paper needs to establish more clearly what is fundamentally new beyond unifying these ideas under a common objective-interpolation formulation.

  2. The shared-feature explanation is demonstrated in a deliberately constructed toy setting. It remains uncertain how well this mechanism explains the improvements in large vision models, LLMs, and reinforcement learning.

  3. The method requires access to source data or a usable estimate of the source objective. This increases data storage, computational cost, and deployment constraints, limiting applicability when only pretrained checkpoints are available.

  4. The optimal smoothing duration varies across datasets and tasks. The paper does not provide a general procedure for selecting the schedule without additional experiments or validation.

Questions for authors
  1. What is the principal methodological contribution beyond existing source replay and annealing methods, and under what conditions does loss interpolation provide benefits that these methods cannot?

  2. Can the authors provide evidence that shared-feature retention is the mechanism behind improvements in the larger models, rather than a correlated effect of source supervision?

  3. How sensitive are the LLM results to the choice of evaluation judge, prompt sample, and training seed?

  4. Can the smoothing schedule be selected automatically without target-test evaluation or extensive hyperparameter tuning?

  5. How should the method be applied when source data are unavailable and the original source objective cannot be reconstructed from the pretrained checkpoint?

3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for their feedback and are glad they found our evaluation broad, our controlled toy experiment useful for explaining our proposed mechanism, and our analyses informative. We address their concerns and questions below.

Methodological Contribution
We agree that mixing in source supervision is not new in itself. Our contribution lies in formulating a clean, simple way to smooth the move from source to target across very different types of shift. Section 3 describes adaptation as a change in inputs, learning signals, or the loss itself. Because smoothing interpolates objectives rather than data, the same recipe covers:

  • input-distribution shifts (permuted CIFAR);
  • label and learning-signal shifts (shuffled CIFAR, ImageNet → DomainNet);
  • narrowing to a new data distribution (OLMo pretraining → instruction tuning);
  • changes in the loss function and its scale (offline-to-online RL, TD3+BC → TD3). Applying it needs only a weighted sum of the two losses and a schedule, with no shift-specific machinery. Data replay alone cannot express the RL case, where the objective itself changes.

Furthermore, while prior work shows that source supervision helps (mainly against forgetting), our work extends this with an analysis of why it helps (and specifically related to target learning). This analysis is made simpler with the specific formulation of loss smoothing. With one intervention we can study:

  • how smoothing reweights shared and conflicting learning (Sec. 5.1)
  • whether the resulting retention persists after source supervision ends (Sec 5.2)
  • whether the benefit comes from the source signal rather than from reducing the target-loss coefficient, KL or EWC penalties, or head preparation (Sec 5.3).

Shared feature retention driving target gains
While we provide detailed mechanistic results only for the toy example, the comparisons in Section 5.3 give indirect evidence for practical settings. There we show that loss smoothing outperforms other methods that also try to preserve the pre-trained model (EWC, KL regularization, LP-FT). This suggests that the gains come not just from retaining the source knowledge, but from retaining a specific part of it. To show more precisely how these methods differ from loss smoothing mechanistically, we recreated these baselines in the toy setting: KL regularization uses only target data and tries to preserve the pre-trained model's predictions on it. It penalizes any change that moves these predictions, whether it comes from flipping the conflicting features or from strengthening the shared ones. In our toy example, KL slows down the CIFAR and the MNIST branches equally, and the shared feature is learned no better than under the hard switch. EWC reweights parameter directions, but by a different criterion than loss smoothing: it tries to preserve all features used by the pre-trained model, whether shared or conflicting. In the toy setting from the paper, the pre-trained model relies on the conflicting MNIST feature, so both EWC and loss smoothing downweight the gradients of the MNIST branch. If we swap the roles so that MNIST is shared and CIFAR is conflicting, loss smoothing downweights the CIFAR branch instead, while EWC still downweights the MNIST branch. Even in the original setting, EWC does not reproduce the effect of loss smoothing and behaves similarly to hard switch. Its Fisher estimate is local, so the model can still learn the new MNIST mapping along directions that look flat for the source loss at the pre-trained model. At the same time, learning of the CIFAR feature is also slowed down, because the penalty is symmetric and penalizes changes even when they improve the source loss. LP-FT preserves the penultimate-layer representations, which can contain both shared and conflicting features. The LP phase flips the head weights of the conflicting features to the target mapping, so the model can use them for the target task from the start of fine-tuning. Loss smoothing does the opposite and slows down their use. As a result, LP-FT behaves like a hard switch.

These results in combination with the results in Section 5.3 point to the idea that the loss smoothing gains are driven by the feature reweighting explanation. To obtain direct evidence in the larger models, we believe we can extend the gradient decomposition based analysis we did for the toy setting to practical settings (see Definition of the shared feature in our reply to Reviewer sfXK) and will incorporate it into the final version of the paper.

AuthorsOpenReview ↗

Rebuttal (2/2)

Storage costs, compute costs, etc
In order to allow for easier analysis, we deliberately wanted to create a simple framework that allows for explicit mixing of objectives without other potential confounders. The costs (both storage and compute) can thus be traded off by slightly altering the setup: storage costs by using fewer samples stored from the source distribution; compute costs by estimating the mixed objective by sampling, where source examples replace part of a fixed-size batch, removing the extra forward/backward pass of loss smoothing entirely (labeled as sampling mixing below). We ran both ablations and summarize the results in the table below. We can see that overall, while the more efficient variants do result in a slight dip in ImageNet (source) performance, the target performance is fairly close to the original loss mixing, with sampling mixing actually doing slightly better on target performance. We add this and more extended results in the appendix H.3. Note, for this experiment and the rest of our work, we assume we are in the (fairly standard) regime of fine-tuning where we do have access to at least some amount of data from the source distribution.

MethodSource imagesImageNet probeDomainNet target
Hard switch—76.776.45
Loss mixingAll81.676.77
Sampling mixingAll80.776.81
Loss mixing1,02477.876.69

Smoothing Schedule Hyperparameter
While we do not provide an automatic procedure for the smoothing schedule, smoothing adds a single interpretable hyperparameter, the duration as a fraction of the adaptation budget. This can be tuned on validation data like other hyperparameters such as warmup length or learning rate. Performance is also not sensitive to the exact choice:

  • On DomainNet, every duration from 5% to 100% of training improves target accuracy over the hard switch (Table 7).
  • In RL, using one horizon (50k steps) for all suites changes the overall IQM from 85.7 to 84.4, and every horizon we tested outperforms all baselines overall. Choosing the horizon on held-out tasks also selects 50k.
  • For OLMo, selecting the window and learning rate together on separate validation prompts reproduces the reported final-checkpoint results. As a practical default, smoothing over 10–25% of the adaptation budget works across our settings. The best duration does vary in some cases (MuJoCo, OLMo), and we will state this in the limitations. We believe the discovery of automatic smoothing schedules is also an interesting future direction.

LLM results sensitivity to seed, judges, prompts
Prompt The reported results are consistent across prompt samples. We redrew the validation/evaluation split of the 805 prompts 1,000 times, re-selecting each method’s learning rate on each new validation half and evaluating on the other half. In 7 of 8 dataset–checkpoint comparisons, the median difference between smoothing and hard switch across redraws is within 0.75 points of the reported value, and the sign matches in at least 85% of redraws.

Training seed. On Anthropic-HH, the result is consistent across seeds. At each of the four checkpoints, all three seed-paired comparisons agree in sign, and the seeds differ from each other by at most 2.4 points (a spread no larger than evaluation noise alone would produce). At the final (most overtrained) checkpoint, all three favor smoothing, by +1.8 to +2.7 points. Tulu-v1 varies a bit more. At intermediate (non-final) checkpoints, roughly half of the seed-to-seed spread is attributable to evaluation noise from the finite prompt set, with the remainder reflecting the training seed. At the final (most over-trained) checkpoint, two of three seeds favor smoothing (−1.1, +0.9, +5.9) meaning that across both datasets, 5 of 6 seed-paired comparisons favor smoothing.

Judge: For the judge, we follow standard practice and randomize answer order against a fixed reference to limit position bias. Also, our LLM retention and source-feedback analyses use held-out NLL and do not depend on a judge.

We hope this addresses your concerns. If it does, we would appreciate your reconsidering the score. We are happy to answer any remaining questions.

Thank you for the detailed response. The additional results have addressed my concerns. I encourage the authors to incorporate these experiments and clarifications into the revised manuscript. I will maintain my original rating.

Reviewer_bJMaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This work proposes loss smoothing, a method for adapting pretrained models in which the training objective is a linear interpolation between the source and target losses. The interpolation coefficient is annealed toward the target over a fixed window, after which ordinary target training resumes. The authors evaluate the method on four settings including continual supervised learning on CIFAR-10, fine-tuning an ImageNet ViT on DomainNet, instruction tuning OLMo-1B from overtrained checkpoints, and offline-to-online RL on D4RL.

Critical strengths
  1. The paper evaluates one simple recipe across four quite different adaptation regimes. The design of experiments is clean.
  2. The toy analysis is clean. The decomposition of the smoothed update into shared and conflicting gradient weights is easy to follow, and the ablation separating the coefficient schedule from the reweighting gives useful intuition.
Critical weaknesses
  1. In pretrained models, the authors measure source retention but do not show that the retained features drive target gains as shown in the toy experiment.
  2. The advantage of linear smoothing over simpler alternatives is not very clear. Smoothing outperforms temporary replay on DomainNet, but constant replay remains competitive in controlled OLMo experiments, and warmup matches smoothing on permuted CIFAR.
  3. The initialization panels in Fig. 8 compare update directions on the same model, not sensitivity changes induced by smoothing. These results do not consistently support the claim that smoothing "significantly mitigates" sensitivity.
  4. The RL experiments do not fully isolate loss smoothing. In TD3+BC-to-TD3, the method also anneals the actor-objective scale, which may independently stabilize adaptation. These differences complicate attribution of the gains to loss interpolation.
  5. The RL smoothing horizon is selected on the evaluated tasks while baseline settings remain fixed which favors smoothing. It is also unclear how the 128-update OLMo window shown in Fig. 3 was chosen.
Questions for authors
  1. Do retained source features contribute to target performance in pretrained models? Since smoothing's gain largely disappears with LP-FT, could it primarily mitigate random-head mismatch?
  2. Does reduced sensitivity persist until the end of training and is the result robust across perturbation settings?
  3. How would RL results change with smoothing horizons selected on held-out tasks? How was the 128-update OLMo window selected?
2 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for their feedback and are glad that they found our experimental design clean and thorough. We address their concerns and questions below:

Connection between feature retention and target gains
While we do see that the source retention of loss smoothing seems to benefit the target task learning (Figure 7b), this only holds up to a certain point as too much retention can prevent the model from learning the new task. Plain retention itself is not the complete mechanism behind the target gain; we simply use it as an easy to compute proxy in practical settings. We argue that the gains of loss smoothing come from retaining specific parts of the source knowledge: the features shared with the target task. Methods that try to preserve the pre-trained model without this selectivity (KL regularization, EWC, LP-FT) do not reproduce the gains (Section 5.3). We do a further investigation of this in the toy setting which shows how each of them treat shared and conflicting features differently than loss smoothing (see the Shared feature retention driving target gains part of the reply to Reviewer gwjV). In the toy example, this decomposition places the conflicting part mostly in the MNIST branch and the shared part mostly in the CIFAR branch. We think this analysis can be generalized to the more practical settings by decomposing the gradient into parts aligned with the shared direction between source task and target task (gt∣gt∣+gs∣gs∣\frac{g_t}{|g_t|} + \frac{g_s}{|g_s|}) and parts aligned with the conflicting directions (gt∣gt∣−gs∣gs∣\frac{g_t}{|g_t|} - \frac{g_s}{|g_s|}) and will incorporate it into the final version of the paper.

Regarding the comparison to LP-FT, we agree that in DomainNet, part of the effect of loss smoothing comes from mitigating the random-head mismatch, as the LP phase does in LP-FT. However, while LP freezes the backbone, loss smoothing only slows down the updates that conflict with the source task. Changing the backbone early is important here: both the hard switch and loss smoothing achieve higher target accuracy than their LP-FT versions (Figure 7a), and loss smoothing makes these changes while preserving relevant source knowledge. After LP, the target loss starts low, so there is little left for smoothing to reweight. The random-head mismatch is also specific to the DomainNet setup: in all the other settings in the paper, the model keeps its output layer, and loss smoothing still improves target performance.

Advantage of linear smoothing over temporary replay
Smoothing actually does outperform replay for most of the controlled OLMo setup (see Figure 3), especially on the more “overtrained” checkpoints. While different alternatives do exist that match smoothing on some setups, smoothing is the only one that consistently wins across the different setups.

Sensitivity changes induced by smoothing
We show two different types of comparisons in Figure 8. The left and middle plots measure the sensitivity of the model in the target (corresponding to hard switch training) and mixed (corresponding to smoothed training) directions at initialization. We see that at initialization, smoothing generally chooses directions that are less sensitive, hence mitigating the effect of overtraining induced sensitivity at the start of the fine tuning process.
The right plot in Figure 8 shows a comparison with different models obtained after 128 steps of training with either hard switch or smoothing (128 steps corresponds to 26.5% of the full training and the end of the smoothing period). We see that smoothed training takes the model to a region where the target update direction is less sensitive for the source distribution, i.e. doing a target-only update changes the smoothed model’s source predictions less. We also ran a new, similar ablation on DomainNet showing a similar trend on that setup (Appendix G.3). At matched target progress, smoothed models are significantly less sensitive.

RL experiments scale annealing
The scale annealing is only used for the Mujoco experiments. For AntMaze and Adroit, there is no scale annealing used and thus the effect of loss smoothing is directly visible. We ran additional ablations for the Mujoco environments of “hard switch + scale annealing” and “smoothing - scale annealing”. The results are in the table below:

ConditionMuJoCo IQM [95% CI]
Hard switch68.4 [63.8, 73.4]
Hard switch + anneal68.2 [61.6, 74.4]
Smoothing - anneal64.2 [60.4, 66.8]
Smoothing (paper, includes anneal)76.2 [68.3, 82.8]

We’d also argue that scale annealing, while not part of actual loss smoothing, fits into the spirit of the method by creating smooth transitions between training objectives. We have added these ablations to the paper, thank you for the suggestion.

AuthorsOpenReview ↗

Rebuttal (2/2)

Smoothing horizon selection
For RL, we reselected the horizon for each task based on the performance on other tasks in the suite. For every task in the MuJoCo and Adroit suites, the horizon selected stays at 50k (what is reported in the main result). On AntMaze, the horizon selected changes from the 250k reported to 250k for 1 task, 100k for 2 tasks, and approximately tied between the two for 1 task. Overall, 50k seems to be a good default that works well for all settings, including AntMaze (lowering the IQM from 94.3 to 93.0, which still outperforms the other methods).

For OLMo, since we wanted to study the “overtrained” effect, we selected the smoothing ramp based on the validation performance of the runs starting from the most “overtrained” checkpoint, i.e. the longest pretrained checkpoint (3.09T).

We hope this addresses your concerns. If it does, we would appreciate your reconsidering the score. We are happy to answer any remaining questions.

Comments on the paper

1 comment
Area_Chair_Bau6OpenReview ↗

Reminder: author–reviewer discussion deadline approaching

Dear Reviewers and Authors,

Thank you to the reviewers for your time, effort, and dedication in reviewing this paper. This AC also appreciates the authors' detailed rebuttal, which addresses the reviewers' comments and includes additional clarifications and experimental results.

To the reviewers: As the author–reviewer discussion deadline approaches, please carefully review the authors' responses and assess whether they adequately address your concerns. If any concerns have not been fully addressed, please let the authors know what further clarification or evidence would be helpful.

Thank you again for your valuable contributions and constructive discussion.

Best,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.