acceptodds
Under review as a conference paper at ICLR 2027

Learning Optimal Robot Rewards from Imperfect Trajectories with Quasimetrics

Abstract

Video reward models for robot learning are typically trained to predict temporal progress, the relative position of each frame within a demonstration. However, this target measures the demonstrator’s value rather than the optimal one. Pauses and detours therefore distort the target, and failed attempts yield no target at all, since the time to success is never observed. We propose Quasimetric Reward Model (QRM), which targets the optimal goal-reaching value by learning from bounds rather than regression targets. Each observed video segment is a feasible path whose return provides a lower bound on the optimal value, even when the trajectory is inefficient or failed. A quasimetric model combines these bounds through the triangle inequality to tighten its value estimates. Such a value is defined for a single goal state, whereas a language instruction admits many successful outcomes and provides no goal image for a new task. QRM therefore distills the value of reaching the nearest successful training outcome into a compact model conditioned on visual history and language. Its dense reward complements the sparse success signal during reinforcement learning. On MetaWorld tasks unseen during reward model training, QRM improves its policy success rate with imperfect videos that degrade temporal-progress models, outperforming vision-language reward models 50× its size. For fine-tuning a pretrained robot policy, QRM yields 1.8× the success rate of the strongest baseline on held-out LIBERO-90 tasks and matches or exceeds every baseline on real-robot manipulation tasks learned from few demonstrations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Reward model, Robot learning, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_HR4MOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

QRM learns a quasimetric distance between visual states from upper bounds on segment costs, so inefficient and failed videos still provide valid supervision. It then distills the distance to the nearest successful training endpoint into a language-conditioned model that requires no goal image, and uses that distance as a potential-based shaping term alongside a sparse success reward. The method achieves the top IQM on 8 unseen MetaWorld tasks, reaches 0.910 versus 0.507 for the best baseline on 8 held-out LIBERO-90 tasks with GR00T N1.6 + DSRL, and matches or beats baselines on 3 real-robot tasks.

Critical strengths
  1. Clean and well-motivated framing: temporal progress estimates the demonstrator's value, segment costs bound the optimal value, and quasimetric structure combines bounds. 2.Failed trajectories become useful without outcome labels or preference annotation.
  2. Evaluation is rigorous: evaluate in both simulation and 3 real world tasks
Critical weaknesses
  1. MAD already learns quasimetrics from arbitrary-quality, action-free trajectories. QRM adds language-conditioned nearest-goal distillation. Compare against a goal-image QRL/MAD oracle on identical data to isolate its value and cost.
  2. In Fig. 5a, 1× failed data hurts performance; 50× adds only 6.7 points with overlapping CIs. This weakens the bound-tightening claim.
  3. Eq. 10 requires simulator success checks or human labels for every real-robot episode. QRM shapes an existing reward; learned-reward-only training is untested, and PBRS invariance relies on the sparse term.
  4. QRM rises from 0.797 to 0.927 with PBRS (Fig. 6c), exceeding most reward-model gaps. Add sparse-reward-only baselines to Figs. 2–3 to isolate shaping’s benefit.
Questions for authors
  1. What happens without the sparse success reward, training on the shaping term alone? Is QRM meant to replace the task reward or only to densify it?
  2. Why does 1× failed data hurt in Fig. 5a, and does the quasimetric's distance error actually decrease with more failed data?
  3. How were the 8 MetaWorld and 8 LIBERO-90 held-out tasks chosen?
  4. How sensitive are results to ε_QRL, H, α, and λ₀?
3 comments
AuthorsOpenReview ↗

Response to reviewer HR4M

We thank the reviewer for the careful reading of our paper and the insightful feedback.

[W1] Comparison with Goal-Image QRL/MAD Oracle.

We acknowledge that QRL (Wang et al., 2023) and MAD (Steccanella et al., 2026) already learns quasimetrics from imperfect, action-free trajectories. However, QRL and MAD require a specified goal state to evaluate progress, whereas our setting provides only language instructions that may correspond to multiple valid outcomes. QRM addresses this gap through two stages. Stage 1 learns goal-conditioned quasimetric distances from trajectory segments’ cost bounds, and Stage 2 distills the distance to the nearest successful outcome into a language-conditioned reward model. To isolate the contribution of Stage 2, we compare QRM against a goal image oracle using the same frozen quasimetric learned in Stage 1.

(1) Goal-image oracle comparison. We compare QRM against a goal-conditioned oracle that directly evaluates the same frozen Stage-1 quasimetric. We generate oracle goal images using a scripted policy in a simulation environment.

MethodIQM95% CI
QRM0.927[0.820, 0.967]
QRM (Oracle Goal)0.923[0.767, 0.983]

Across eight unseen MetaWorld (Yu et al., 2020) tasks, QRM achieves a comparable IQM to the goal-image oracle. This shows that eliminating the goal image does not produce a reduction in downstream policy performance.

(2) Distillation fidelity. To assess how faithfully Stage 2 reproduces its goal-conditioned teacher, we measured the mean absolute error (MAE) on 194 held-out trajectories. QRM achieves a MAE of 3.96% of the horizon, with 3.60% on successful trajectories and 4.32% on failed trajectories. These results show that QRM can approximate the nearest-goal teacher targets using only visual observations and language instructions at inference, without requiring goal images.

(3) Computational and deployment cost. QRM eliminates goal acquisition overhead while providing low-latency reward evaluation. QRM provides language-conditioned rewards with a median inference latency of 19.40 ms per query, without goal image generation or goal bank retrieval. This advantage is particularly relevant to real-world manipulation, where obtaining valid goal images for novel tasks requires additional successful demonstrations or computationally expensive image generative models. Nearest-goal distillation removes this dependency, enabling low-latency reward evaluation directly from observations and language instructions.

Overall, our contribution beyond QRL/MAD is not quasimetric learning itself, but its conversion into a practical language-conditioned reward model. The additional experiments show that nearest-goal distillation closely approximates its teacher and enables goal-image-free downstream RL with comparable performance, while removing goal acquisition overhead at deployment.

[W2 & Q2] Effect of Failed Trajectories and Distance Accuracy

(1) Why does performance decrease at 1×? In Figure 5(a), the decrease from 0.800 to 0.747 IQM is within the overlapping 95% confidence intervals, so it does not establish a systematic negative effect. Importantly, Eq. (6) guarantees that adding feasible paths cannot loosen the ideal quasimetric upper bound, while finite-sample optimization and downstream RL may exhibit fluctuations. Thus, the downstream RL results are compatible with our bound-tightening claim.

(2) Does failed data actually improve distance accuracy? To test this, we construct a controlled experiment with exact shortest-path references. We compare four conditions: (1) successful demonstrations following only a longer route, (2) adding an incomplete failed trajectory that connects to a shorter route without reaching the goal, (3) using the same videos as (2) but removing training pairs encoding this connection, and (4) adding a complete successful shortcut as a reference.

Training conditionDistance MAE ↓ [95% CI]
Successful trajectories only24.74 [23.46, 25.85]
+ Incomplete failed connector6.25 [4.55, 7.95]
Same videos, connecting training pairs removed7.82 [6.65, 9.42]
+ Complete successful shortcut3.20 [2.25, 4.26]

Adding the incomplete failed trajectory reduces distance MAE by 74.7%. Furthermore, the failed connector outperforms the same videos with connecting training pairs removed, with a paired MAE difference of −1.57 [−2.31, −0.93]. This shows that the improvement comes in part from useful path connectivity, rather than simply observing additional frames.

Together, these results strengthen our central claim: failed trajectories provide valid path constraints that can improve optimal distance estimation even without reaching the goal. The new experiment directly demonstrates improved distance accuracy through path connectivity, while Figure 5(a) suggests potential downstream RL benefits as failed data scales.

AuthorsOpenReview ↗

Response to reviewer HR4M

[Q1 & W3 & W4] Role of sparse success reward and densification of QRM

We clarify that QRM is designed to densify an existing sparse task reward, rather than replace it. As described in Appendix A.1.4, PBRS (Ng et al., 1999) provides intermediate feedback without changing the total discounted reward. Without the sparse success reward, successful and failed trajectories starting from the same state receive the same total discounted reward. Thus, the sparse reward defines the task objective, while QRM provides dense feedback to accelerate learning.

To isolate QRM's contribution, we additionally evaluate a sparse-reward-only baseline (α=0\alpha=0 in Eq. 10) under the same MetaWorld RL protocol as Figure 2, using eight held-out tasks and three policy seeds.

MethodIQM95% CI
Sparse-only (α=0\alpha=0)0.633[0.473, 0.780]
QRM (full, PBRS)0.927[0.820, 0.967]

QRM improves success IQM by 29.4 percentage points over sparse-reward-only learning under the same interaction budget. This directly demonstrates the empirical benefit of QRM's dense shaping signal beyond the sparse task reward.

[Q3] Choice for MetaWorld and LIBERO tasks

For MetaWorld, we follow the eight held-out tasks used in the ReWiND’s evaluation protocol (Zhang et al., 2025), enabling direct comparison with prior work. For LIBERO-90 (Liu et al., 2023), we focus on Kitchen Scenes 1–3 to evaluate generalization across diverse manipulation skills within a consistent environment domain. The eight held-out tasks are chosen to span object placement, stacking, drawer opening, and appliance operation, covering both object-centric and articulated-object interactions. This setting allows us to assess transfer to diverse unseen tasks.

[Q4] Hyperparameters sensitivity of QRM

To check how much QRM is sensitive to the hyperparameters, we conducted experiments on eight held-out MetaWorld tasks, varying each hyperparameter independently while keeping the others fixed. As reported in the table below, QRM demonstrates robust performance across a broad range of hyperparameter settings. These results suggest that QRM is not highly sensitive to its hyperparameter choices, with all achieving comparable IQM to original QRM.

Configurationε_QRLHαλ₀Final IQM [95% CI]
QRM (original)0.2548110.927 [0.820, 0.967]
ε_QRL = 0.1250.12548110.913 [0.787, 0.960]
ε_QRL = 0.50.548110.873 [0.753, 0.943]
H = 240.2524110.920 [0.823, 0.950]
H = 960.2596110.930 [0.807, 0.990]
α = 0.50.25480.510.927 [0.887, 0.957]
α = 50.2548510.960 [0.873, 0.983]
λ₀ = 0.50.254810.50.947 [0.903, 0.970]
λ₀ = 50.2548150.903 [0.770, 0.973]

The authors have addressed my concerns. I maintain my score of 3.

Reviewer_3ASiOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to learn robot rewards from imperfect demonstration videos. Many existing methods treat a frame's temporal position within a demonstration as task progress, but pauses, detours, and failed attempts can make this signal misleading. QRM instead treats each observed trajectory segment as one feasible way to reach its endpoint, so its cost provides an upper bound on the optimal cost between those states.

Using this idea, the authors learn a distance model from mixed-quality videos and distill it into a language-conditioned reward model that does not require a goal image during policy learning. They evaluate the reward on unseen MetaWorld tasks, LIBERO-90 fine-tuning, and three real-robot tasks, and study whether suboptimal or failed videos can still improve reward learning.

Critical strengths
  1. S1. A fresh and well-motivated view of video rewards. The paper identifies an important gap between temporal progress and optimal task progress that prior methods largely overlook. Treating observed segments as bounds rather than equality targets is a principled way to avoid learning the demonstrator's pauses and detours.

  2. S2. The experiments support the central motivation. Controlled comparisons suggest that QRM can benefit from mixed-quality and failed trajectories where temporal-progress training degrades. The compact reward model is practically appealing, and the real-robot results provide preliminary evidence of applicability.

Critical weaknesses

W1. The real-robot comparison uses different reward scales without a sensitivity analysis. QRM uses α = 100 on the robot, whereas ReWiND and RoboMeter directly use predicted progress, with all methods sharing the same success bonus and SAC settings. Since reward magnitude can affect finite-budget SAC training, the paper should report per-chunk reward magnitudes or α/rescaling sensitivity before attributing the advantage specifically to reward quality.

Questions for authors
  1. Reward scaling on the real robot. What per-chunk reward magnitudes does each method produce, and how sensitive are the results to QRM's α and to rescaling the baseline rewards?
  2. Reward-training robustness. Can you report multiple reward-model training seeds for the main MetaWorld comparison and the UB/TE ablation?
2 comments
AuthorsOpenReview ↗

Response to reviewer 3ASi

We thank the reviewer for the careful reading of our paper and the insightful feedback.

[W1 & Q1] Real-robot reward scaling

To identify the potential confounding effect of reward magnitude, we conducted an additional analysis of the actual per-chunk rewards and designed a reward scale sensitivity experiment.

(1) Reward magnitude analysis. We analyze the rewards of all three real-world tasks, removing the shared success bonus of 64. The following table reports the root mean square (RMS) of dense rewards. The reward RMS of QRM is approximately 3.5-6.0x that of ReWiND (Zhang et al., 2025) and RoboMeter (Liang et al., 2026), and RoboMeter (LoRA).

TaskQRM (α=100\alpha=100)ReWiNDRoboMeterRoboMeter (LoRA)
Cube1.950.560.590.40
Stapler2.490.740.670.34
Towel2.200.450.630.37
Overall2.220.600.630.37

(2) Reward scale sensitivity. To test whether QRM's advantage depends on this larger reward scale, we additionally evaluate QRM (full) with α=25\alpha=25. This setting approximately matches QRM's dense reward RMS to ReWiND and RoboMeter. We retain the same SAC settings, training budget, and success bonus.

MethodCubeStaplerTowel
ReWiND35%20%25%
RoboMeter40%30%30%
RoboMeter (LoRA)55%30%35%
QRM (full), α=100\alpha=10055%75%65%
QRM (full), α=25\alpha=2555%70%75%

QRM maintains its performance advantage after matching its reward scale. At α=25\alpha=25, QRM achieves an average success rate of 66.7%, comparable to 65.0% at α=100\alpha=100, and continues to match or outperform every baseline on all three tasks. These results suggest that QRM provides informative reward signals that effectively guide policy learning, even when its reward magnitude is matched to those of the baselines.

[Q2] Multi-seed results for MetaWorld

We extended our experiments to 3 independent reward-model training seeds for the main MetaWorld (Yu et al., 2020) comparison and 5 seeds for the upper-bound (UB) versus temporal-equality (TE) ablation.

(1) Main MetaWorld comparison (3 reward-training seeds).

MethodIQM95% CI
ReWiND (DINOv3)0.726[0.573, 0.847]
ReWiND (full)0.664[0.506, 0.801]
QRM (expert)0.876[0.779, 0.942]
QRM (full)0.914[0.814, 0.954]

The multi-seed results preserve the performance ordering observed in Figure 2. QRM (expert) outperforms ReWiND (DINOv3) by 15.0 percentage points, while QRM (full) outperforms ReWiND (full) by 25.0 percentage points. Moreover, adding mixed-quality trajectories improves QRM's IQM from 0.876 to 0.914, whereas ReWiND's IQM decreases from 0.726 to 0.664. These results provide additional evidence that QRM benefits from imperfect training trajectories.

(2) UB versus TE ablation (5 reward-training seeds).

MethodIQM95% CI
QRM (Temporal Equality)0.745[0.645, 0.825]
QRM (Upper Bound)0.801[0.724, 0.874]

UB achieves 5.6 percentage points higher IQM than TE, consistent with the original results in Figure 6(a). These additional results with more reward model training seeds support that upper bound supervision can provide more useful supervision than the temporal equality supervision.

Thank you for the response and the additional experiments. You have solved my concerns. The results suggest that QRM's performance is robust to reward scaling and remains consistent across different training seeds. I will maintain my score.

Reviewer_RQrQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

What's the problem: The paper focuses on learning dense visual RM for language-conditioned robot manipulation tasks. The demontration videos usually have unequal quality and can be imperfect, e.g., with pauses, failures etc. Video RMs often predict temporal progress, but because of the imperfections, the remaining time may not be an optimal indicator.

On the other hand, goal-conditioned quasimetric RL methods can learn optimal shortest-path distances from imperfect data, but they require a target image g. And when a task is only specified by language, this is not available.

What this paper does: The authors tackle this problem by proposing the Quasimetric Reward Model (QRM), turning a quasimetric distance into a language-conditioned reward.

It proceeds in 2 stages:

(1) Learn a visual quasimetric distance across video states, by maximizing the within trajectory distances subject to observed cost upper bounds. It still needs a goal image.

(2) This distance model is frozen, and the distance to the nearest successful training ending of the same task is distilled into a cost-to-go model (called J in the paper) that takes only the current video observations and language instruction. So there's no need of a goal image anymore. Finally during RL, this cost-to-go model is deployed via potential-based reward shaping with a sparse success bonus.

QRM is validated on MetaWorld, LIBERO-90 and 3 real robot tasks.

Critical strengths
  1. The paper proposes a method converting goal-conditioned quasimetrics into a language reward model that doesn't need a goal image anymore.

The stage 2 (cf. the summary above) compresses the distances to multiple valid goal states into a lightweight model - it doesn't need to query large VLM during online RL anymore, making learning more practical.

  1. QRM achieves strong downstream RL performance. It also uses fewer parameters compared to VLM reward models.
Critical weaknesses
  1. The reward used (in section 4.3) in the experiments always uses a shaping term > 0, so we cannot clearly tell how much QRM's dense reward contributes. This question can be answered with an ablation experiment with exactly the same RL setup but with the shaping term set to 0.

  2. A central claim of the paper is that using one sided upper bounds instead of temporal regression recovers the optimal remaining cost from imperfect videos. But there's currently a lack of comparison between both, and Fig 6(a) that compares both show a very small difference with large confidence interval overlap.

Questions for authors

For the main points, please refer to the section on weaknesses.

Suggestions

It would be good to improve the paper's clarity.

For instance, for Figure 1: the caption should be improved and the caption should either be (preferably) more self-contained, or explicitly refer to sections that correspond to different parts of the Figure. Currently the reader needs to jump back and forth to really understand the figure and that degrades the paper's clarity. The figure can either be a high-level description (without all the notations) or a detailed walkthrough, but the level of detail between the figure and its caption should match.

5 comments
AuthorsOpenReview ↗

Response to reviewer RQrQ

We thank the reviewer for the careful reading of our paper and the insightful feedback.

[Suggestion] Improvement of Figure 1.

We have revised the figure to emphasize the high-level workflow, removing detailed equations. The revised figure and caption are intended to provide a self-contained overview without requiring readers to navigate between sections. These changes are reflected in the revised paper.

[W1] Contribution of QRM's dense reward.

We added a sparse-only baseline (α=0\alpha=0) to the MetaWorld (Yu et al., 2020) experiments, removing QRM's dense shaping term from Eq. (10) while retaining the sparse success reward. We keep the RL algorithm, policy initialization, training data, interaction budget, and all other hyperparameters unchanged.

MethodIQM95% CI
Sparse-only (α=0\alpha=0)0.633[0.473, 0.780]
QRM (full, PBRS)0.927[0.820, 0.967]

QRM improves the final success IQM by 29.4 percentage points over sparse-only learning on eight held-out MetaWorld tasks. This shows that QRM's learned shaping reward provides substantial additional learning benefits beyond the sparse success signal alone.

AuthorsOpenReview ↗

Response to reviewer RQrQ

[W2] Upper-bound supervision versus temporal regression.

Upper-bound supervision enables QRM to infer shorter feasible paths from imperfect demonstrations, whereas temporal regression tends to reproduce the demonstrator's timing. We conducted additional experiments to verify this. A direct path-composition comparison shows that QRM exploits connections across demonstrations that temporal regression baselines do not, while a five-seed ablation provides evidence favoring upper-bound supervision.

(1) Direct comparison with temporal regression on path composition.

QRM reduces remaining-cost prediction error by 19.30 percentage points when a connecting trajectory is available, whereas two temporal regression baselines show virtually no improvement. This directly tests the proposed advantage of upper-bound supervision.

Setup. We extend the path-composition experiment in Appendix A.4 using a MetaWorld (Yu et al., 2020) reaching task with two conditions.

  • Connected: A long demonstration S→X→Y→G takes 130 actions due to a detour through Y. An auxiliary trajectory U→X→G provides a shorter continuation from X. Combining the observed segments yields a 70-action route S→X→G that is never demonstrated in full.
  • X-bypass: We replace the auxiliary trajectory with U→Z→G, preserving its length and endpoint while removing the connection at X.

We compare QRM against a RynnValue-style (Huang et al., 2026) remaining-time regressor (T−t) and a ReWiND-style (Zhang et al., 2025) normalized-progress regressor (t/T). All methods use the same training trajectories across five scenes and three training seeds. We evaluate remaining cost predictions against the recorded 70-action short route, reporting mean relative error (%, lower is better).

MethodX-bypass error ↓Connected error ↓Change
Remaining-time regression (T−t)95.38%95.72%+0.34 pp
Normalized progress regression (t/T)50.17%49.91%−0.26 pp
QRM (UB)36.63%17.33%−19.30 pp

QRM benefits substantially from the connecting trajectory, whereas neither temporal regression baselines exhibit a comparable improvement. These results provide evidence that QRM can combine segments from different demonstrations to infer a shorter feasible route that was never demonstrated in full. In contrast, the temporal regression baselines remain tied to the demonstrated timing and do not exploit the connecting trajectory.

(2) 5 seeds upper-bound versus temporal-equality ablation.

To address the uncertainty in Figure 6(a), we extend the experiment to five reward model training seeds. This comparison isolates the supervision objective within QRM, replacing the one-sided upper-bound (UB) constraint with temporal equality (TE), while holding the architecture, training data, and remaining procedure fixed.

MethodIQM95% CI
QRM (Temporal Equality)0.745[0.645, 0.825]
QRM (Upper Bound)0.801[0.724, 0.874]

UB achieves 5.6 percent points higher IQM than TE across five reward-model training seeds. Although some confidence intervals overlap, the higher IQM with more seeds supports the benefit of upper bound supervision. The path-composition experiment above provides complementary evidence about why this supervision differs from temporal regression.

(3) Downstream comparisons with temporal-progress reward models.

QRM also consistently outperforms temporal-progress reward models in downstream RL. The first comparison uses QRM (full) and ReWiND (full), the second trains both models on the same inefficient demonstrations, and the third compares QRM (expert) with ReWiND (PBRS). These results show that QRM produces more effective rewards for downstream RL, including when demonstrations are imperfect and when the reward shaping method is matched.

Experimental settingQRM IQMReWiND IQM
Mixed-quality data, matched DINOv3 backbone (Fig. 2)0.9270.587
Successful but inefficient demonstrations (Fig. 5c)0.6730.410
Matched training data and PBRS (Fig. 6c)0.8530.787

In summary, the new path-composition experiment demonstrates QRM's ability to compose imperfect trajectories’ segments, unlike the temporal regression baselines. The five-seed UB/TE ablation provides additional evidence of improved downstream performance. Together, these results support learning shortest path cost estimates from upper bounds rather than fitting the timing of imperfect demonstrations.

Dear Reviewer RQrQ,

Thank you again for your thoughtful feedback. We have revised the manuscript and conducted additional controlled experiments to address your concerns.

We would greatly appreciate hearing whether these additions address your concerns. Please let us know if you have any further questions or concerns, and we would be happy to address them.

Thank you for your time and consideration.

Comments on the paper

1 comment

Dear Reviewers,

Thank you again for your insightful and constructive feedback. We have posted detailed responses along with additional experiments addressing your concerns, including a sparse-only baseline, real-robot reward-scale comparisons, and evaluations with additional training seeds.

We would greatly appreciate your feedback on whether these new results address your concerns and whether any questions remain. We are happy to provide further clarification, analyses, or experiments during the remaining discussion period.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.