- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
QRM learns a quasimetric distance between visual states from upper bounds on segment costs, so inefficient and failed videos still provide valid supervision. It then distills the distance to the nearest successful training endpoint into a language-conditioned model that requires no goal image, and uses that distance as a potential-based shaping term alongside a sparse success reward. The method achieves the top IQM on 8 unseen MetaWorld tasks, reaches 0.910 versus 0.507 for the best baseline on 8 held-out LIBERO-90 tasks with GR00T N1.6 + DSRL, and matches or beats baselines on 3 real-robot tasks.
Critical strengths
- Clean and well-motivated framing: temporal progress estimates the demonstrator's value, segment costs bound the optimal value, and quasimetric structure combines bounds. 2.Failed trajectories become useful without outcome labels or preference annotation.
- Evaluation is rigorous: evaluate in both simulation and 3 real world tasks
Critical weaknesses
- MAD already learns quasimetrics from arbitrary-quality, action-free trajectories. QRM adds language-conditioned nearest-goal distillation. Compare against a goal-image QRL/MAD oracle on identical data to isolate its value and cost.
- In Fig. 5a, 1× failed data hurts performance; 50× adds only 6.7 points with overlapping CIs. This weakens the bound-tightening claim.
- Eq. 10 requires simulator success checks or human labels for every real-robot episode. QRM shapes an existing reward; learned-reward-only training is untested, and PBRS invariance relies on the sparse term.
- QRM rises from 0.797 to 0.927 with PBRS (Fig. 6c), exceeding most reward-model gaps. Add sparse-reward-only baselines to Figs. 2–3 to isolate shaping’s benefit.
Questions for authors
- What happens without the sparse success reward, training on the shaping term alone? Is QRM meant to replace the task reward or only to densify it?
- Why does 1× failed data hurt in Fig. 5a, and does the quasimetric's distance error actually decrease with more failed data?
- How were the 8 MetaWorld and 8 LIBERO-90 held-out tasks chosen?
- How sensitive are results to ε_QRL, H, α, and λ₀?
3 comments
Response to reviewer HR4M
[Q1 & W3 & W4] Role of sparse success reward and densification of QRM
We clarify that QRM is designed to densify an existing sparse task reward, rather than replace it. As described in Appendix A.1.4, PBRS (Ng et al., 1999) provides intermediate feedback without changing the total discounted reward. Without the sparse success reward, successful and failed trajectories starting from the same state receive the same total discounted reward. Thus, the sparse reward defines the task objective, while QRM provides dense feedback to accelerate learning.
To isolate QRM's contribution, we additionally evaluate a sparse-reward-only baseline ( in Eq. 10) under the same MetaWorld RL protocol as Figure 2, using eight held-out tasks and three policy seeds.
| Method | IQM | 95% CI |
|---|---|---|
| Sparse-only () | 0.633 | [0.473, 0.780] |
| QRM (full, PBRS) | 0.927 | [0.820, 0.967] |
QRM improves success IQM by 29.4 percentage points over sparse-reward-only learning under the same interaction budget. This directly demonstrates the empirical benefit of QRM's dense shaping signal beyond the sparse task reward.
[Q3] Choice for MetaWorld and LIBERO tasks
For MetaWorld, we follow the eight held-out tasks used in the ReWiND’s evaluation protocol (Zhang et al., 2025), enabling direct comparison with prior work. For LIBERO-90 (Liu et al., 2023), we focus on Kitchen Scenes 1–3 to evaluate generalization across diverse manipulation skills within a consistent environment domain. The eight held-out tasks are chosen to span object placement, stacking, drawer opening, and appliance operation, covering both object-centric and articulated-object interactions. This setting allows us to assess transfer to diverse unseen tasks.
[Q4] Hyperparameters sensitivity of QRM
To check how much QRM is sensitive to the hyperparameters, we conducted experiments on eight held-out MetaWorld tasks, varying each hyperparameter independently while keeping the others fixed. As reported in the table below, QRM demonstrates robust performance across a broad range of hyperparameter settings. These results suggest that QRM is not highly sensitive to its hyperparameter choices, with all achieving comparable IQM to original QRM.
| Configuration | ε_QRL | H | α | λ₀ | Final IQM [95% CI] |
|---|---|---|---|---|---|
| QRM (original) | 0.25 | 48 | 1 | 1 | 0.927 [0.820, 0.967] |
| ε_QRL = 0.125 | 0.125 | 48 | 1 | 1 | 0.913 [0.787, 0.960] |
| ε_QRL = 0.5 | 0.5 | 48 | 1 | 1 | 0.873 [0.753, 0.943] |
| H = 24 | 0.25 | 24 | 1 | 1 | 0.920 [0.823, 0.950] |
| H = 96 | 0.25 | 96 | 1 | 1 | 0.930 [0.807, 0.990] |
| α = 0.5 | 0.25 | 48 | 0.5 | 1 | 0.927 [0.887, 0.957] |
| α = 5 | 0.25 | 48 | 5 | 1 | 0.960 [0.873, 0.983] |
| λ₀ = 0.5 | 0.25 | 48 | 1 | 0.5 | 0.947 [0.903, 0.970] |
| λ₀ = 5 | 0.25 | 48 | 1 | 5 | 0.903 [0.770, 0.973] |
The authors have addressed my concerns. I maintain my score of 3.
Response to reviewer HR4M
We thank the reviewer for the careful reading of our paper and the insightful feedback.
[W1] Comparison with Goal-Image QRL/MAD Oracle.
We acknowledge that QRL (Wang et al., 2023) and MAD (Steccanella et al., 2026) already learns quasimetrics from imperfect, action-free trajectories. However, QRL and MAD require a specified goal state to evaluate progress, whereas our setting provides only language instructions that may correspond to multiple valid outcomes. QRM addresses this gap through two stages. Stage 1 learns goal-conditioned quasimetric distances from trajectory segments’ cost bounds, and Stage 2 distills the distance to the nearest successful outcome into a language-conditioned reward model. To isolate the contribution of Stage 2, we compare QRM against a goal image oracle using the same frozen quasimetric learned in Stage 1.
(1) Goal-image oracle comparison. We compare QRM against a goal-conditioned oracle that directly evaluates the same frozen Stage-1 quasimetric. We generate oracle goal images using a scripted policy in a simulation environment.
Across eight unseen MetaWorld (Yu et al., 2020) tasks, QRM achieves a comparable IQM to the goal-image oracle. This shows that eliminating the goal image does not produce a reduction in downstream policy performance.
(2) Distillation fidelity. To assess how faithfully Stage 2 reproduces its goal-conditioned teacher, we measured the mean absolute error (MAE) on 194 held-out trajectories. QRM achieves a MAE of 3.96% of the horizon, with 3.60% on successful trajectories and 4.32% on failed trajectories. These results show that QRM can approximate the nearest-goal teacher targets using only visual observations and language instructions at inference, without requiring goal images.
(3) Computational and deployment cost. QRM eliminates goal acquisition overhead while providing low-latency reward evaluation. QRM provides language-conditioned rewards with a median inference latency of 19.40 ms per query, without goal image generation or goal bank retrieval. This advantage is particularly relevant to real-world manipulation, where obtaining valid goal images for novel tasks requires additional successful demonstrations or computationally expensive image generative models. Nearest-goal distillation removes this dependency, enabling low-latency reward evaluation directly from observations and language instructions.
Overall, our contribution beyond QRL/MAD is not quasimetric learning itself, but its conversion into a practical language-conditioned reward model. The additional experiments show that nearest-goal distillation closely approximates its teacher and enables goal-image-free downstream RL with comparable performance, while removing goal acquisition overhead at deployment.
[W2 & Q2] Effect of Failed Trajectories and Distance Accuracy
(1) Why does performance decrease at 1×? In Figure 5(a), the decrease from 0.800 to 0.747 IQM is within the overlapping 95% confidence intervals, so it does not establish a systematic negative effect. Importantly, Eq. (6) guarantees that adding feasible paths cannot loosen the ideal quasimetric upper bound, while finite-sample optimization and downstream RL may exhibit fluctuations. Thus, the downstream RL results are compatible with our bound-tightening claim.
(2) Does failed data actually improve distance accuracy? To test this, we construct a controlled experiment with exact shortest-path references. We compare four conditions: (1) successful demonstrations following only a longer route, (2) adding an incomplete failed trajectory that connects to a shorter route without reaching the goal, (3) using the same videos as (2) but removing training pairs encoding this connection, and (4) adding a complete successful shortcut as a reference.
Adding the incomplete failed trajectory reduces distance MAE by 74.7%. Furthermore, the failed connector outperforms the same videos with connecting training pairs removed, with a paired MAE difference of −1.57 [−2.31, −0.93]. This shows that the improvement comes in part from useful path connectivity, rather than simply observing additional frames.
Together, these results strengthen our central claim: failed trajectories provide valid path constraints that can improve optimal distance estimation even without reaching the goal. The new experiment directly demonstrates improved distance accuracy through path connectivity, while Figure 5(a) suggests potential downstream RL benefits as failed data scales.