acceptodds
Under review as a conference paper at ICLR 2027

Severity Targets Can Reward Corrupted Labels: Diagnosing Sample Weights for Control

Abstract

A difficult trajectory is not necessarily a reliable one. We show that severity-based sample weighting can conflate these properties and reward corrupted action labels. With 20% of Push-T training episodes corrupted, applying a severity target directly gives corrupted windows 1.89× the clean-window weight; a learned weighting network retains the same preference. A direct-target control and a feature × target ablation locate the failure in the target rather than the network. Replacing severity-only supervision with a consistency-aware target changes the corrupted-versus-clean ordering, while an amplitude-matched control that preserves the original ordering does not recover performance. A jerk- and path-matched negative control confirms the consistency-aware target preserves genuinely hard but correct maneuvers while collapsing a matched corrupted twin, so it reads label reliability rather than dynamical roughness. Across four tested synthetic corruption modes, the effect follows the target components: corruptions that inflate action jerk or path length are up-weighted, whereas a constant offset that leaves them intact is nearly neutral. Five-seed Push-T experiments further show that target correlation and closed-loop utility need not agree. USV experiments provide complementary calibration diagnostics and the same separation between target agreement and task utility. We therefore recommend evaluating weight ordering, dispersion, corruption preference, and matched task outcomes separately.

Primary area:
applications to robotics, autonomy, planning
Keywords:
sample reweighting, imitation learning, label reliability, corrupted labels, importance weighting, diffusion policy, robot learning, evaluation diagnostics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jEXWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

It explores the limitations of severity-based sample weighting in control learning, particularly when training data contains corrupted action labels. The paper shows that commonly used difficulty measures, such as action jerk and path length, can incorrectly assign higher importance to corrupted samples. To address this, it explores a learned residual weighting model and introduces a consistency-aware target that considers the agreement between actions and observed trajectories. The approach is evaluated on USV control and Push-T manipulation tasks, with comparisons against uniform weighting, Meta-Weight-Net, L2RW, and several controlled variants. The experiments show that severity-based weighting can favour corrupted samples, while consistency-aware weighting helps correct this behaviour. Additional ablations examine the effects of weighting targets, input features, and weight magnitudes. However, the presentation and writing require significant improvement, and the corrected weighting does not consistently outperform uniform training in closed-loop task performance.

Critical strengths
  • It highlights an important limitation of severity-based weighting, where corrupted action labels can be mistaken for difficult training examples and assigned higher importance.

  • It examines the effects of weighting targets, model features, and weight magnitude through controlled experiments. The comparison between genuinely difficult maneuvers and corrupted examples is particularly useful, as it shows that severity measures alone may not distinguish valid hard examples from unreliable labels.

Critical weaknesses
  • The paper is poorly organized and difficult to follow. The motivation, proposed methodology, and experimental analysis are not presented clearly, making it challenging to understand the main contributions and how the different components fit together.

  • The distinction between difficult samples and corrupted supervision has already been explored in noisy-label learning and imperfect demonstration learning. The paper does not adequately differentiate its contribution from this existing literature.

Questions for authors
  • The paper is difficult to follow, particularly the problem definition, motivation, research gap, and the description of the weighting mechanism and experimental setup.

  • Several closely related studies on imperfect demonstrations and data quality appear to be missing: (1) Wang et al., Learning to Weight Imperfect Demonstrations (ICML 2021), (2) Belkhale et al., Data Quality in Imitation Learning (NeurIPS 2023), and (3) Yue et al., How to Leverage Diverse Demonstrations in Offline Imitation Learning (ICML 2024). Could you clarify how their findings differ from these studies, and what genuinely new insight is provided beyond the known distinction between difficult and unreliable training examples?

  • The consistency-aware target reduces the importance assigned to corrupted samples; the reported closed-loop performance does not consistently outperform uniform weighting. What practical advantage does the proposed approach offer over uniform weights? Do the existing experiments provide convincing evidence that the correction consistently improves control performance?

  • The conclusion is largely based on a specific severity definition and limited corruption settings. How can you establish that the observed failure is a broader problem with severity-based weighting rather than a consequence of particular target formulation? Do the reported results provide sufficient evidence across different corruption types, tasks, and random seeds?

Reviewer_yhmWOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how severity-based weighting of datapoints affects behaviour cloning from offline control datasets. The authors define a hand-coded severity calibration target, which is used to train the weight network, and then compare its effect on the action prediction MSE and task success rate of the final policy. The authors also evaluate meta-learning-based approaches that use validation loss to update the weights via meta-gradients. Finally, the authors introduce a consistency-aware weighting scheme.

Through experimentation on Push-T and USV, the authors show that a) training the weights to correlate with severity does not necessarily improve task performance, b) severity weights tend to weight corrupted data higher, and c) consistency-aware weighting mitigates this.

Critical strengths

The authors raise an interesting question about how to weight data points during behaviour cloning. The consistency-aware weighting shows promising results, supported by the jerk-matched ablation.

Critical weaknesses

The writing exposition is poor; the paper misses key details

  1. The abstract and introduction focus too deeply on experiment details and results without introducing the relevant background and, more importantly, the problem setting. This makes it very hard to comprehend the two sections. Examples: What is the task? What does severity mean in this context? What does USV stand for? Etc.

  2. Section 3: How is the decision problem modelled? Is it an MDP? What is a “wave”, a “sea-state”, etc.?

  3. In the intro, Push-T is said to be the “primary testbed”, but the subsequent sections (3-6) discuss the USV task?

Lack of sufficient motivation of the problem setting

  1. What is the downstream task objective? Is it the final return of the control policy? This is not mentioned in Section 3.

  2. What is the motivation behind weighting training samples? More importantly, why must the weights have a separate training objective disjoint from the final return?

  3. Why should we focus on severity weighting as a research problem? Is it the current standard for weighting data points in established control problems? How does it compare to return-based weighting such as Advantage-Weighted Regression [1], or expert-guided offline imitation methods such as ORIL [2] and DWBC [3]?

  4. Can you validate that the proposed targets, which are averages of normalised observation features, capture “severity”? Why does it capture task difficulty?

  5. If one already has access to the severity targets or weights, what is the motivation for training a separate network instead of using the targets directly?

Experimental results/insights are poor

  1. The paper’s main contribution is that “severity-based weighting”, which is a hand-coded heuristic that does not consider the actual return of the control policy, has no positive effect on the final return of the trained policy — isn’t this expected?

  2. Many results in the paper use a single training seed, making it difficult to assess the significance of the reported results.

  3. The experimental scope is very narrow.

    1. In the USV experiments, the final return of the control policy in the simulator is not reported. Isn’t that the main objective?

    2. Push-T is the only environment that reports these scores. Does the observed behaviour generalise to other control settings?

  4. The paper does not report sufficient details for reproducibility. Examples: How is the USV dataset created? How are the residual priors in Table 1 generated?

  5. All of the weighting methods seem to perform worse than the uniform weighting case. This raises the question of whether we need to be weighting the datapoints differently and whether there is a need for the proposed consistency-aware weighting.

References

  1. Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning. arXiv:1910.00177, 2019.

  2. Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Aytar, Misha Denil, Nando de Freitas, and Scott Reed. Offline Learning from Demonstrations and Unlabeled Experience. NeurIPS Offline Reinforcement Learning Workshop, 2020.

  3. Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations. ICML, PMLR 162:24725–24742, 2022.

Questions for authors

Please refer to the Weaknesses.

Reviewer_G72ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether severity-based sample weighting in offline control can inadvertently assign higher importance to corrupted action labels. Using Push-T as the primary testbed, the authors introduce synthetic action-label corruption and examine how different weighting targets affect sample priorities and closed-loop performance. Through direct-target controls, feature–target ablations, and component-level analyses, the paper shows that the up-weighting of corrupted samples is mainly driven by action jerk and path length in the severity target, rather than the weighting network itself. A consistency-aware target is further examined as a potential correction, while complementary USV experiments investigate weight calibration and its relationship with task performance. The results provide useful evidence that agreement with a predefined weighting target does not necessarily translate into better control performance.

Critical strengths
  1. The paper provides useful experimental evidence that corrupted-sample up-weighting can originate from the severity target rather than the weighting network. In particular, the direct-target experiments, feature–target ablations, and component-level analyses identify how action jerk and path length contribute to the observed preference for corrupted samples. These controls make the failure-mechanism analysis more convincing than a simple comparison of weighting methods.

  2. The distinction between target alignment and actual task utility is meaningful. The Push-T and USV experiments show that stronger agreement with a prescribed weighting target does not necessarily lead to better control performance. The additional corruption modes and matched hard-maneuver control also help clarify the conditions under which the observed weighting failure occurs.

Critical weaknesses
  1. The proposed consistency-aware target relies on the cosine similarity between the first commanded action offset and the agent's recent displacement (Appendix D.8). However, directional consistency does not necessarily indicate label correctness. Legitimate control behaviors may involve sharp turns, direction reversals, or corrective maneuvers that temporarily disagree with the preceding motion. Although the matched experiment in Figure 17 shows that the proposed target preserves the weights of the selected clean-hard samples, the corrupted counterparts are constructed by directly reversing the action directions, which also reverses the proposed consistency signal by construction. This experiment is useful for distinguishing the correction from a simple smoothness penalty, but it does not establish whether the measure can reliably separate corrupted labels from valid but dynamically complex behaviors in more general settings.

  2. The practical benefits of the proposed correction remain limited. In Table 5, QAlign achieves a closed-loop success rate of 0.181, compared with 0.191 for uniform weighting, with no statistically significant difference (p = 0.710, Table 18). The direct consistency-aware target does improve performance relative to the direct severity target, but its advantage over the amplitude-matched control is only marginal (p = 0.069). Moreover, matching the within-bucket weight standard deviation does not necessarily control for other differences in the weight distributions, making it difficult to attribute the performance change exclusively to weight ordering. The primary experiment also relies on a fixed corruption fraction and one policy backbone, while comparisons with dedicated noisy-label learning methods are absent. These limitations weaken the evidence for the practical effectiveness and broader applicability of the proposed correction.

Questions for authors
  1. Could the authors explain how the proposed cosine-based consistency measure distinguishes corrupted labels from legitimate control behaviors involving direction reversals, sudden corrective actions, or temporary motion against the previous direction? The negative control in Figure 17 is informative, but its construction directly changes the consistency signal being evaluated. Additional analysis of naturally occurring low-consistency but valid maneuvers would materially strengthen this claim.

  2. The paper attributes the improvement of the consistency-aware target to changes in sample-weight ordering rather than weight magnitude. However, the amplitude-matched control only matches within-bucket weight dispersion, and the performance difference between direct_qual and direct_shrink is marginal. Could the authors provide additional evidence that ordering, rather than other differences in the weight distribution, is responsible for the observed improvement?

Comments on the paper

1 comment

The paper has been revised substantially, including the framing, organisation, experimental evidence, and interpretation of its central results. I appreciate the authors’ significant efforts to do this, but I believe this calls for a fresh review, which is outside the scope of this rebuttal. It also indicates that the original version of the paper, with reasonable (minor) modifications, is not suitable for publication.

The authors also admit that many of the central claims of the previous version are not sufficiently justified and reframe the paper, changing its central claims. This includes: a) their proposed severity target is not an established weighting method; it is unclear whether this corresponds to difficulty; b) neural network training of the weights is redundant.

Some further questions remain:

a) Corrupted labels are difficult to fit: Isn’t it obvious that prediction errors (difficulty) would be high on corrupted data? This is already well established and motivates a lot of literature on data curation/cleaning. What new insight do these experiments establish beyond demonstrating this known failure mode in Push-T?

b) Reward-free intrinsic-difficulty setting: Difficulty is not clearly defined, and the setting itself is not well motivated. For example, task difficulty requires rewards to be well defined. Prediction difficulty does not. If task success rate is the objective, why can’t we use rewards to come up with the weighting?

c) Limited generalisation: The paper’s main claims rest on experiments in a single environment; how do we know that the results are not specific to the environment/dataset?

For all the above reasons, I maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.