acceptodds
Withdrawn from ICLR 2027

DCRL: DIRECT CONTRASTIVE REWARD LEARNING FROM CARDINAL TRAJECTORY SCORES

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Trajectory-level scores evaluate complete behaviors but not individual decisions. We study learning a standalone dense Markov reward from fixed, possibly suboptimal trajectories with cardinal scores, without transition rewards, pairwise preference labels, or additional environment interaction. We present Direct Contrastive Reward Learning (DCRL), which matches similar states across trajectories to derive two local reward constraints. Contrastive ordering compares different actions using trajectory-score differences; smoothness assigns similar rewards to similar state-action pairs. DCRL combines these constraints with trajectory-score fitting, then freezes the learned reward for any downstream reinforcement-learning algorithm that accepts scalar rewards. The ordering is assumed only as a statistical tendency across state-matched comparisons, allowing individual violations. Using 10 suboptimal demonstrations per task on several Gymnasium MuJoCo benchmarks, DCRL outperforms T-REX, R4, and Frozen RRD (a fixed-data version of RRD) in final mean SAC return on every task. It achieves 3.6–10.1 times the return of the best supplied demonstration and reaches 80.0–90.5% of true-reward SAC performance on three tasks. Ablations show that contrastive ordering improves on trajectory-score fitting alone and that smoothness provides a further gain.

Primary area:
reinforcement learning
Keywords:
Inverse RL, learning dense reward, demonstration trajectories, contrastive constraints

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DsCjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper aims to learn a reward function for state-action pairs from trajectory-level scores. The authors propose two local reward constraints: a contrastive ordering constraint that directly compares actions at similar states based on their trajectory-level scores, and a smoothness constraint that assigns similar rewards to similar state-action pairs. The proposed method falls into the field of inverse reinforcement learning (IRL). Accordingly, in the experimental section, the authors compare their method with state-of-the-art IRL methods.

Critical strengths
  1. This paper observes that a higher-scoring trajectory is more likely to contain a higher-quality action than a lower-scoring trajectory when the states are similar. The authors use this observation to learn reward functions in several IRL tasks and achieve improved performance. Although this observation does not always hold, the authors empirically demonstrate its effectiveness on the benchmarks reported in the paper.

  2. I think the proposed method can be easily integrated into many RL algorithms, and this plug-and-play property offers some potential for extending the method to more benchmarks or real-world tasks where only trajectory-level scores are available.

Critical weaknesses
  1. My major concern is the motivation of this paper, which is also summarized in Assumption 1. The authors assume that a higher-scoring trajectory is more likely to contain a higher-quality action when the states are similar. However, practical scenarios may severely violate this assumption, and I do not think it generally holds in real-world tasks. As we all know, the trajectory-level score is influenced by all state-action pairs along the trajectory. Thus, I doubt whether the proposed method can be successfully extended to more practical tasks beyond the benchmarks considered in this paper, even though it is straightforward to apply the algorithm to other tasks. If the authors can thoroughly address this major concern, I would be willing to raise my score.

  2. In the experimental section, the authors only consider three baseline methods, among which only one (R4) is relatively recent. I think it would be better to include some additional baselines to further demonstrate the strengths of the proposed method.

Questions for authors

Could the authors provide some failure cases where the proposed method does not work well? I think such cases would be useful for exploring the limitations of the method and clarifying the applicability of Assumption 1.

Reviewer_ndk7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DCRL, which learns a standalone dense Markov reward from a fixed set of possibly suboptimal trajectories, each labeled with one cardinal score, and then freezes the reward for downstream RL. The overall pipeline enforces three requirements on the learned reward. First, it should respect each demonstration’s cardinal score, which is done by fitting the average predicted reward of each trajectory to its score. Since this alone leaves the reward of individual transitions within a trajectory ambiguous, DCRL adds two local requirements. Second, similar transitions should receive similar rewards (a smoothness loss). Third, for different actions taken in similar states, the action from the higher-scoring trajectory should tend to receive higher reward (a Bradley–Terry contrastive loss). To decide which cross-trajectory pairs these constraints apply to, and how strongly, the authors design heuristic similarity normalizers: density-adaptive state radii in a PCA-reduced space, a global quantile-based action radius, and a tanh-bounded score-gap weight. With 10 suboptimal demonstrations per task on four Gymnasium MuJoCo tasks, SAC trained on DCRL’s reward outperforms T-REX, R4, and Frozen RRD, reaching 3.6–10.1 times the return of the best demonstration. Ablations show that both local constraints contribute.

Critical strengths
  1. The paper is clean, and the notation is clear and consistent; Table 1 and Appendix B are helpful. The authors are cautious about why each design choice is made and about their assumptions.
  2. The problem setup is practical. In many settings, scoring a complete behavior is easier than providing reliable per-step feedback. A standalone reward that can be plugged into a downstream RL algorithm is also a flexible output.
Critical weaknesses
  1. The empirical evaluation is limited for an empirical work.

a. Only one downstream learner is evaluated. The paper claims the learned reward can be used with any RL algorithm that accepts scalar rewards, but all results come from SAC. Prior work [1] has shown, through its ReBRAC experiments, that reward design can become much less significant under a strong learner. Whether DCRL’s advantage over the baselines persists with a stronger online learner, such as FlashSAC [2], remains unclear.

b. All four tasks are Gymnasium MuJoCo locomotion. More challenging environments (e.g., Humanoid) and other task types (e.g., manipulation) are needed to support the generality of the method.


  1. The few-demonstration and cardinal-score settings are each tested at a single point.

a. All experiments use exactly 10 demonstrations per task. Learning rewards from one or only a few trajectories has attracted broad interest [3, 4, 5], and DCRL is motivated by the same scarce-data regime. Yet how DCRL performs with fewer or more demonstrations is never tested. This matters for the method itself, since pair construction requires differently scored trajectories, and the scales δ\delta and QQ are estimated from only NN scores.

b. All demonstrations come from early SAC checkpoints, and every score is the true undiscounted return (Section 5). Using cardinal score gaps seems the key distinction from T-REX and R4 (Section 2), but this ingredient is never isolated. How does DCRL behave when scores are noisy, monotonically transformed, or not equal to the environment return, or when demonstrations come from a different source? If performance is unchanged when the score gaps are distorted, the method may effectively be relying on ordinal information only.


  1. The presentation of the method could be improved. Sections 4.2-4.3 introduce the matching representation, the local radii, and the action radius before the reader knows how mm, κ\kappa, and ww enter the objective. I could not fully understand the role of each component until I read Section 4.4. I suggest bringing the learning objective (Eq. 10) earlier, followed by how the transition pairs and weights are constructed, so the reader has a top-down path to DCRL.

References:

[1] Minimal Ingredients for Reward Assignment from Expert Demonstrations (Dong et al., 2025)

[2] FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control (Kim et al., 2026)

[3] Optimal Transport for Offline Imitation Learning (Luo et al., 2023)

[4] Robot Policy Learning with Temporal Optimal Transport Reward (Fu et al., 2024)

[5] Imitation Learning from a Single Temporally Misaligned Video (Huey et al., 2025)

Questions for authors
  1. Why can DCRL lead to policies far beyond the demonstrations? As the policy improves, SAC visits transitions well outside the demonstrations’ support. What rewards does DCRL assign to those new transitions? Table 4 shows episode-level rank agreement, but not how the learned reward behaves per transition outside the support. Could the authors provide insights here? In these tasks, one possibility is that the forward reward is the torso displacement per step, and the observation includes the torso’s x-velocity. Is the extrapolation driven by fϕf_\phi tracking the observed forward velocity?
  2. How were the reward-learning hyperparameters in Table 6 determined? Were they tuned per task or shared, and on which tasks? Are they transferable to other domains, e.g., higher-dimensional state spaces where PCA and nearest-neighbor matching may behave differently? In particular, k=2k=2 retains very few candidates per anchor and also sets the neighbor rank to q=1q=1, so each local radius is just the nearest-neighbor distance. The described method is more complex than what is used in evaluation.
  3. I could not fully understand why the trajectory-score fitting objective works to produce such a good proxy reward. In Section 5, each score ziz_i is the total undiscounted return of the trajectory, while Eq. (1) and Eq. (6) fit the average predicted reward to it. Could the authors explain the motivation for matching the average predicted reward to a total return, and why this is a sensible way to anchor the learned reward? Relatedly, I had trouble reconciling the scale. If the average per-step prediction matches ziz_i, I would expect the learned return over an episode to be much larger than the true return, but Table 4 reports R^/R\hat{R}/R between 0.48 and 1.23. Any intuition on this?
Reviewer_vmTXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies learning a standalone dense Markov reward from a fixed set of suboptimal trajectories, each labeled with a single cardinal score, without transition-level feedback, pairwise preferences, or additional environment interaction. The proposed method, DCRL, matches transitions across trajectories by state similarity and derives two local constraints: a contrastive ordering loss that uses trajectory-score gaps to order rewards of dissimilar actions at matched states, and a smoothness loss that ties together predictions for similar state-action pairs. These are combined with a trajectory-score fitting term, and the learned reward is frozen for downstream RL. Experiments on MuJoCo locomotion tasks compare against ranking-, rating-, and return-based reward learning baselines, with ablations on the two local constraints.

Critical strengths
  • If cardinal scores can be provided by human labelers with consistency, this framework will benefit with only few suboptimal trajectories and their scores.
  • The procedure is specified concretely enough to be replicable with full algorithm description and implementation details, and the authors are made adaptations to baselines for fair comparisons.
  • The objective ablation in Figure 4 does isolate the contribution of the contrastive term over trajectory-score fitting alone within mujoco benchmark.
Critical weaknesses

W1. The problem setting is weakly motivated.

  • The preference-based reward learning literature adopted ordinal pairwise comparisons precisely because absolute scalar judgments are difficult for humans to provide consistently. At scale, maintaining calibration of cardinal scores "on a common scale" (Sec. 3.1) across annotators is close to impossible, and the paper never addresses this.
  • No realistic source of cardinal trajectory scores is identified anywhere in the paper: not for robot learning, nor for any other domain.
  • In the experiments, each trajectory "is scored by its total undiscounted episode return" (Sec. 5), i.e., the ground-truth return. This makes the setting circular: having exact cardinal scores with no noise doesn't make sense.
  • There is no experiment with noisy, miscalibrated, or monotonically transformed scores, so robustness to the score imperfections that would arise in any practical instantiation is unknown.

W2. The empirical validation is too weak to support the claims.

  • The evaluation uses only four MuJoCo Gym locomotion tasks, which are saturated and far removed from practical robot learning. There is no evaluation on manipulation benchmarks, or real robot data. The reviewer recommends adding experiments on at least Meta-world.
  • The learned reward's scale disagreement with the true return (Table 4) is acknowledged but not analyzed; the Spearman correlations are computed on binned episodes and do not substitute for showing the reward is well-behaved off the demonstration distribution.

W3. Limited novelty and no limitations discussion.

  • The method is largely a recombination of existing methods: Bradley–Terry preference loss, return regression (as in RRD), and a smoothness regularizer.
  • The related work omits the rating-based RL (White et al., AAAI 2024), which learns rewards from absolute human evaluations of individual trajectories and is directly relevant to the cardinal-score premise.

Minor comment

  • Having limitations section is highly recommended.
Questions for authors
  • In what realistic robot-learning scenario would cardinal scores on a common scale be available, while per-transition rewards and pairwise preferences are not? How does DCRL degrade under rater noise or per-rater miscalibration (e.g., different monotone transforms applied to subsets of scores)?
  • Using the true MuJoCo rewards, what fraction of retained pairs in P satisfy the ordering of Assumption 1, and how does the violation rate vary?
  • How sensitive are results to k (why 2?), αa, λc, and λs?
  • Why restrict evaluation to MuJoCo Gym locomotion? Results on D4RL datasets or a manipulation suite would be needed to support any claim of practical relevance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.