acceptodds
Withdrawn from ICLR 2027

Training Contracts: Learning Validity Boundaries for Reusing Training Interventions in Embodied Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Embodied reinforcement learning accumulates experience from training interventions, which may be transferred to future learning processes. However, the same intervention can have different effects under different learning contexts, making it difficult to determine when an intervention remains valid for reuse. We introduce Training Contracts, which learn an explicit validity boundary for each training intervention by comparing training processes with and without each intervention in the source domain. Each contract records supported, contradicted, and unresolved regions, and validates the boundary on independent source checkpoints. For target tasks, the trainer uses the validity boundary to decide whether to reuse an intervention, perform a probe, or abstain. All target policies are trained from scratch, and only external intervention knowledge is transferred. Across twelve target configurations from six task families, Training Contracts improve average performance from 0.59 to 0.64 (8.5% relative gain) over the strongest conditional-effect prediction baseline. Beyond the primary benchmark, including Meta-World and real-robot experiments, Training Contracts improve success rates by 14.5% and 15.1% over the average baseline performance, respectively.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Reinforcement Learning, Training Intervention Transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_X47uOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the reuse of training interventions in embodied reinforcement learning by transferring trainer-side knowledge about the contexts in which specific interventions have previously helped or harmed learning. For each intervention, the proposed Training Contracts framework collects paired training comparisons and computes supported and contradicted regions. Then the selected regions are evaluated on independent source checkpoints. During training on target tasks, the trainer uses the training contracts to decide whether to reuse an intervention, conduct a budgeted paired probe, or abstain from applying it. The paper evaluates the method on twelve target configurations across six manipulation task families, with additional experiments involving Meta-World and a physical SO-101 robot. Training Contracts reports an average anytime score of 0.640, compared with 0.590 for the uncertainty-aware conditional-effect baseline CE-U. The corresponding scores are 0.615 and 0.563.

Critical strengths
  1. The paper studies a useful problem. The same training intervention may help in some situations but hurt in others. Learning when an intervention should or should not be reused makes sense.

  2. The paper compares with several baselines and includes ablation studies, Meta-World, and real robot experiments. The results are generally good. Training Contracts improves the anytime score from 0.590 to 0.640 compared with CE-U. It also accepts fewer harmful interventions (25 vs. 49) at the same 45% coverage.

Critical weaknesses
  1. The novelty of the method is not very clear. The method combines paired comparisons, rule learning, independent confirmation, and decisions about whether to reuse an intervention. These ideas are already quite familiar. Table 2 suggests that most of the improvement comes from the learned regions. It would be helpful to explain what the other parts of Training Contracts add.

  2. The experiments do not clearly support some parts of the discovery objective. Removing the rule complexity penalty (τ=0\tau=0) improves the anytime score from 0.64 to 0.66. Also, γ=1\gamma=1 gives 0.65, compared with 0.64 for the default γ=2\gamma=2. The paper does not show whether the complexity penalty actually produces simpler rules, or whether a larger γ\gamma reduces harmful reuse decisions.

Questions for authors

Q1. Contribution of the full method. In Table 2, CE-U achieves 0.637 when given the learned validity regions, compared with 0.640 for Training Contracts. What additional benefit does the full method provide beyond these learned regions?

Q2. Discovery objective. Removing the rule complexity penalty improves performance from 0.64 to 0.66, and using a smaller γ\gamma also gives a better result. Could the authors explain why these penalties are needed?

Q3. Partially confirmed contracts. What happens if only the supported or contradicted region passes independent confirmation? Does the method keep the confirmed region and discard the other, or reject the whole contract?

Reviewer_f629OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

this paper shows when experience with a training intervention, such as modifying an auxiliary reward or an optimization setting, can be reused in a new embodied RL problem. Source checkpoints are branched into paired intervention/control continuations. The resulting evidence is used to discover compact predicates identifying supported, contradicted, and unresolved contexts, with independent source confirmation before direct reuse. During target training, the contract selects reuse, a budgeted paired probe, or abstention. Only external intervention knowledge transfers; target policies and learner states are initialized afresh.

Critical strengths
  1. this paper give a clear formulation of conditional training-experience reuse, with unusually good experimental hygiene for this genre. Paired continuations from common checkpoints, split roles assigned before training with descendants confined to their split, retained counterevidence, lineage-level replication with the raw per-lineage contrasts reported, and failure-inclusive denominators. This is consequential because it makes the primary contrast interpretable and in principle reproducible, which is rare for trainer-memory papers.

  2. discriminating controls that localise where the gain comes from. Fixed 12% probe allocation, disabled probes (0.575 vs 0.535), predicate permutation (0.525), a threshold-free matched-coverage audit (25 vs 49 known harms among 432 accepted cases), and the bridge control (CE-U given the learned regions reaches 0.637). together these rule out "more probing" as the explanation and show that the learned validity regions carry the benefit. This matters for interpreting what the method actually contributes.

Critical weaknesses

the comparison does not fully isolate the contribution of contract learning from existing partition-based policy learning, and the uncertainty targets may differ. CE-U with imported contract regions reaches 0.637 versus 0.640 for the full method, making the region-learning procedure central. An honest tree or decision-list policy with matched complexity, independent confirmation, and the same reuse/probe/abstain interface would be a more direct control than only generic regression and nearest-neighbor predictors. Furthermore, Appendix E.3 constructs a confidence bound for a stratum’s mean effect, whereas F.2 calibrates CE-U using residuals against noisy root-level realized effects. The latter may include irreducible outcome variation that is not penalized in the same way by a confidence bound on a mean. This is a possible explanation requiring a matched-estimand control, not an established explanation of the reported gain.

Questions for authors
  1. does calibration against realized root effects make CE-U conservative for outcome variation while the contract bound targets only uncertainty in a stratum mean?

  2. what is the exact source confirmation continuation horizon, and how does it relate to target probes and the final anytime objective? How often do intervention-effect signs change with the horizon?

Reviewer_6quROpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The concept of training contracts refers to transferring knowledge about when to apply training interventions across robotic RL tasks. This paper focuses on learning interpretable supported, contradicted, and unresolved regions from paired source experiments, then validates the rules on independent checkpoints, and selects reuse, probing, or abstentio during target training. The experiment claims to improve the primary budget-weighted anytime score from 0.59 to 0.64.

Critical strengths
  • Careful experimental ablations.
Critical weaknesses
  • Problem not well-motivated: The scope of this paper doesn't convince me the importance of this problem.
  • Unclear methodological novelty: The approach resembles rule learning with independent validation and abstention. A baseline using the same rule space and confirmation procedure would better establish its contribution.
  • Restricted generality: Transfer relies on manually defined diagnostics and compatible semantic adapters. Unobserved changes can invalidate previously confirmed rules.
  • Limited physical evidence: Two real-world tasks, without reported run-level confidence intervals or an explicit absolute training budget, provide limited evidence of practical robustness.
  • Abstract presentation: Essential examples and implementation details appear mainly in the appendix, making the core method unnecessarily difficult to understand.
Questions for authors

Please refer to the weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.