acceptodds
Under review as a conference paper at ICLR 2027

Additivity and Selection Value in Depth-Indexed Model Updates

Abstract

Selecting multi-step losses for a shared dynamics model requires predicting the value of training on a set of objectives. Does this prediction need interactions between losses? We study the question through finite-update probes that fit additive and quadratic set predictors to a fixed teacher-relative utility. For fixed-length SGD blocks, we derive orders , , and for raw singleton, pair, and triple effects at step size . The leading singleton term is initial-gradient alignment. Experiments on two continuous-control tasks with feedforward and recurrent dynamics exhibit this hierarchy. Ridge fitting nevertheless assigns substantial coefficients to pair features: removing the penalty shifts most larger-set prediction comparisons in favor of the quadratic model but changes no choice within the three-loss family. Additive and quadratic choices agree at 178 of 180 common states. In an optimizer stress test, ten-step Adam blocks from fresh optimizer state produce larger higher-order effects, and both predictors select negative-gain updates despite positive mean oracle gain. Closed-loop controls distinguish useful composition from useful interaction modelling. Selected triples improve lower-tail return over All-five updates at matched total loss weight, but increasing All-five's total weight reverses the ordering; continued inexpensive training also outperforms quadratic selection under the recorded cost budgets. These results identify three requirements for interaction modelling to improve training: nonadditive update effects, decision-relevant prediction, and benefits that justify acquiring the labels. no choice within the three-loss family. Additive and quadratic choices agree at 178 of 180 common states. This agreement is not universal: Ten-step Adam blocks from fresh optimizer state produce larger higher-order effects, and both predictors select negative-gain updates despite positive mean oracle gain. Closed-loop controls distinguish useful composition from useful interaction modelling. Selected triples improve lower-tail return over All-five updates at matched total loss weight, but increasing All-five's total weight reverses the ordering; continued inexpensive training also outperforms quadratic selection under the recorded cost budgets. These results identify three requirements for interaction modelling to improve training: nonadditive update effects, decision-relevant prediction, and benefits that justify acquiring the labels.

Primary area:
reinforcement learning
Keywords:
model-based reinforcement learning, world models, loss selection, update valuation, interaction modeling, regularization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VsR1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether interactions between different loss functions need to be considered when training a shared model, and whether more complex models of these interactions lead to practical benefits. The authors study this question using dynamics models in continuous-control tasks, with five training losses corresponding to different prediction depths. They compare two approaches for predicting the benefit of combining losses: an additive model that sums the contributions of individual losses, and a quadratic model that also considers pairwise interactions. The authors estimate the effects of different loss combinations through short training updates and use these estimates to select losses for subsequent training.

On the theoretical side, the paper analyzes the effects of different loss combinations under small-step SGD, showing that individual loss contributions dominate at first order, while interactions appear at higher orders. Experiments on two continuous-control tasks with feedforward and recurrent models show that, although modeling interactions can sometimes improve prediction accuracy, the additive and quadratic models make the same selection in 178 out of 180 cases. The more complex model also fails to provide consistent improvements in training outcomes. Further experiments with different optimizers, loss combinations, and computational budgets show that using all losses or spending the additional computation on continued training can sometimes be more effective than complex loss selection.

Overall, the paper suggests that the presence of interactions between losses does not necessarily justify modeling them explicitly. More complex approaches are useful only when they lead to better decisions and the benefits outweigh the additional computational cost.

Critical strengths

S1. The paper addresses a meaningful research question. It examines whether modeling interactions between different loss functions is necessary, considering not only prediction accuracy but also its impact on training decisions and computational costs. This helps clarify when more complex loss selection methods may be useful.

S2. An interesting theoretical analysis is provided. The authors derive different orders for individual loss effects, pairwise interactions, and higher-order interactions under small-step SGD, offering some explanation for why a simple additive model may be sufficient.

S3. The experimental evaluation covers several important aspects. Beyond comparing the prediction accuracy of additive and quadratic models, the authors examine their effects on loss selection and subsequent training performance. Additional experiments with different optimizers, regularization settings, loss weights, and computational budgets help assess the benefits and limitations of modeling interactions.

Critical weaknesses

W1. Limited experimental settings and generalizability. The experiments cover only two continuous-control tasks (Walker-walk and Cheetah-run), using feedforward and recurrent models with five losses at different prediction depths. Since these losses come from the same dynamics model and may be highly correlated, it remains unclear whether the findings generalize to other tasks, model architectures, or more diverse loss combinations.

W2. Limited scope of the theoretical analysis. The main theoretical results rely on fixed-step SGD with small learning rates, while adaptive optimizers commonly used in practice may behave differently. Although the authors include experiments with Adam, these use a fresh optimizer state rather than preserving the optimizer state during continuous training. This limits how well the theoretical findings apply to more general training settings.

W3. Unclear practical value of interaction modeling. The additive and quadratic models make the same selection in 178 out of 180 cases, and the more complex quadratic model does not consistently improve downstream performance. While these negative results align with the paper's goal, the experiments do not clearly show when interaction modeling would be beneficial. As a result, the three proposed conditions mainly summarize the observations, and their practical guidance for training strategy design remains limited.

Questions for authors

Q1. Could the authors extend the evaluation to more diverse tasks, model architectures, or loss combinations to better assess the generalizability of the findings?

Q2. How would the results change when using Adam with a continuously maintained optimizer state rather than a fresh state?

Q3. Can the authors provide concrete examples where modeling loss interactions leads to better training decisions and downstream performance?

Reviewer_jdrpOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses a specific problem of model-based reinforcement learning problem represented by Walker-walk and Cheetah-run scenarios modeled with an MLP or a GRU dynamics model. In this problem, a notable approach is to choose losses with respect to different prediction depth. This, however, mixes the effect of different joints that may contribute to the overall dynamics in an identifiably distinct nonlinear manners, and the choice of loss requires how should we aggregate these contributions. The authors devise a set of constrained experiments to answer if the disentanglement of the interference between the loss contributors for different-step loss is necessary, resulting in an answer: no.

Critical strengths
  • The paper has stated what previous works they follow and what challenge they had found. The effort in conducting experiments is real.
  • The main research problem I read from the paper is to question the necessity of identifying the cause of different-step losses in using an aggregated multi-step loss to train a dynamics model in a model-based reinforcement learning scenario. This, I believe, is a valid and meaningful research question.
Critical weaknesses
  • Although the main research question I summarized in Strength 2 is a valid and meaningful question worth exploring, the set of experiments to deliver the answer has a limited scope on specific scenarios of Cheetah-run and Walker-run with two dynamics model architectures. Even 2019 paper of Hafner et al. (Learning Latent Dynamics for Planning from Pixels) uses six scenarios to demonstrate. The paper requires broader evidence to hold its claim.
  • I am pro with writing with AI, but this is too much. For example, although Hafner’s 2019 paper on multi-step losses are phenomenal work in this field and so does the work of Chua et al., these are specific methods designed to solve a particular problem in a particular way: to maximize the utility of dynamics models in model-based reinforcement learning. Then, as a follow up paper, I would expect from the authors a minimal act of kindness to spare a paragraph or two for this broader context. But, current introduction nor the main text give a hint on what actual problem the paper wishes to tackle. It would be much helpful to the readers and to the authors as well to have a fully rewritten introduction aiming to guide a newcomer in this topic.
  • Random numbers flooded in the main text making it even harder to read. A few numbers can be helpful to make the paper appear rigorous and sound. But having a number on almost every line of text in Section 5 hides the actual message the authors try to deliver. If the main claim is determined, then all these numbers are just supporting materials. We don’t need “3.18-3.78 times raw pair-effect RMS” (line 308) if we can say “more than three times the raw pair-effect RMS”. Same message, much clearer delivery. I mean, what does “Quadratic exceeds random triples by 0.01195–0.01804 and fixed {2, 4, 6} by 0.01713–0.02379” mean? What about “Increasing All0five’s total weight gives 0.02225–0.02893, larger than the 0.01077–0.01631 composition contrast at weight 0.6”? I would like to listen to the author’s own interpretations on these numbers rather than the numbers themselves.
Questions for authors

Can the authors explain on what practical problem they wish to address in this paper, what were the noteworthy previous approaches that this work follow, and what specific challenges the paper tries to handle that has not been addressed fully in the previous literature? I wish to hear this in a raw, terse format in sentence or two in plain English.

Reviewer_8wLVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether modelling interactions between depth-indexed prediction losses improves the selection of finite updates to a shared dynamics model. It compares additive and quadratic predictors fitted to singleton and pair update gains, using a fixed teacher-relative utility. The analysis establishes a local hierarchy of singleton, pair, and triple effects for fixed-length SGD blocks and shows that ridge regularization can produce fitted interaction coefficients even for additive labels. Experiments on two continuous-control tasks with MLP and GRU dynamics find that additive and quadratic predictors almost always select the same subsets. Additional controls examine loss weighting, probe costs, and failure under cold-start Adam.

Critical strengths
  1. Useful separation of prediction and decision quality. The paper distinguishes raw update interactions, fitted interaction coefficients, and the value of the resulting selections. The observation that improved prediction need not change the selected subset is relevant to evaluating learned selectors.

  2. Careful controls within the studied setting. Common-state comparisons, independent audit batches, matched-total-weight controls, and continued-training baselines help distinguish the effects of composition, correction strength, and label acquisition cost.

  3. Informative regularization analysis. The closed-form ridge analysis provides a concrete explanation for why fitted pair coefficients should not be directly interpreted as evidence of genuine complementarity. The paper also reports negative results and acknowledges important limitations.

Critical weaknesses
  1. The practical motivation for the specific selection problem is insufficiently established. The paper motivates multi-step supervision, but the need to select at most three of five depth losses is less clear. The cardinality constraint is justified primarily as the smallest setting that tests extrapolation beyond pair labels, rather than as an operational constraint. Since the losses share the same eight-step rollout, cardinality does not directly represent computational cost. Consequently, it remains unclear when a practitioner should use this selection procedure instead of training with all depths and tuning their weights.

  2. The conceptual novelty of the main theoretical result appears limited. The hierarchy of first-, second-, and third-order effects is consistent with a local Taylor expansion under small-step updates. Extending this analysis to fixed-length SGD blocks is useful, but the manuscript does not sufficiently establish what consequential, previously unknown behaviour follows from it beyond the controlled local setting. The ridge analysis is more specific, although its broader implications beyond the particular fitting design remain underexplored.

  3. The empirical scope is narrow relative to the practical implications. The evidence comes from two tasks, two relatively small architectures, and short correction phases following pretraining. The optimizer stress test uses freshly initialized Adam states rather than the persistent states typical of ongoing training. Although the manuscript appropriately acknowledges these restrictions, the results provide limited guidance about loss selection during longer training runs or with standard stateful optimizers.

  4. The negative result has limited demonstrated reach beyond the constructed selector. The finding that pair-label acquisition is less productive than continued training is informative. However, the paper does not clearly establish which existing, practically relevant interaction-based approach is challenged by this finding. The experiments primarily demonstrate that the particular quadratic probing procedure is unnecessary in the tested regime; this is a narrower contribution than a general conclusion about the value of interaction modelling.

  5. The connection between the selection utility and downstream performance needs stronger evidence. The selection target measures agreement with a frozen teacher over fixed action menus, with an emphasis on a fixed tail of cases. Closed-loop evaluation provides an important downstream check, but it remains unclear how reliably this local proxy identifies updates that improve control across training states. This matters because both the measured interaction structure and the chosen subsets depend on the utility definition.

Questions for authors
  1. What practical constraint motivates selecting at most three depths when all losses share an eight-step rollout? Can the authors identify a realistic setting in which this cardinality restriction is necessary or beneficial?
  2. How would selected triples perform at total correction weight 1.01.0, compared with All-five at the same total weight? This missing comparison would help clarify whether the reported reversal is primarily a consequence of correction strength.
  3. Does the near-additivity and selection agreement persist with warm-state Adam and longer correction trajectories? If not, can the analysis provide a useful diagnostic for identifying when additive selection becomes unreliable?
  4. How strongly do teacher-relative update gains predict subsequent control improvement? Can the authors provide evidence beyond aggregate endpoint comparisons, or demonstrate robustness to a materially different utility target?
  5. Which existing interaction-based selection method or established training practice should change in light of these results? A more explicit answer would help clarify the paper’s significance.

Overall, the paper offers a careful analysis of a controlled loss-selection setting, with useful distinctions between interactions, prediction accuracy, selection quality, and acquisition cost. My main concern is significance: the practical necessity of the particular selection problem is not sufficiently established, the main local theoretical result is relatively expected, and the empirical evidence is too restricted to support a broader change in training practice. A stronger connection to an established training workflow, or evidence identifying when interaction modelling becomes useful, would substantially strengthen the contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.