- Rating
- 2(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper learns visual reward and progress models from physics-enabled 3DGS copies of real workspaces, plus a few successful real demonstrations. The paper tests the models on eight robot tasks and uses them to improve policies through offline and online learning.
Critical strengths
- The eight real-robot tasks cover two platforms and include deformable objects. Offline policy success rises from 32% to 49%, showing that the critics help policy learning as well as reward prediction.
- The renderer comparison keeps trajectories and labels fixed. This gives a useful test of how realistic scene appearance affects transfer to real images. Comparing sparse and dense labels also helps explain the results.
Critical weaknesses
- Online learning still needs some human involvement. Appendix C.2 uses human labels for calibration and allows an operator to reject success declarations. The detector statistics are measured on the calibration data. Held-out false-positive rates and counts of operator overrides are needed to judge how reliably it can stop rollouts on its own.
- The online comparisons give the models different camera inputs and polling rates. Sim2Critic uses multiple views, the generalist detectors use the scene view, and RoboReward runs less often. These differences can change when rollouts end and what data are collected, making the effect of critic quality hard to separate.
- VOC checks whether scores rise over time in successful rollouts; it does not directly measure task progress. Appendix D uses outcome metrics and example traces to check failures and lost progress. Quantitative tests of interrupted or reversed actions would give stronger evidence that the dense reward tracks the actual state.
- A small real-demonstration count does not capture all the setup work. The method also needs scene reconstruction, registration, physics setup, scripted experts and progress rules (Section 4 and Appendix B). Reporting this effort would make the cost of replacing manual reward labels easier to judge.
Questions for authors
- What are detector precision and recall on held-out data, and how many success declarations did operators override? Without those overrides, how often would rollouts have ended incorrectly?
- Does the advantage remain with the same camera inputs and polling rules, or with a shared success detector while only the dense reward changes?
2 comments
Thank you again for the careful review. We noticed that your rating changed after our response, but we could not find an updated comment, so we would like to understand which concerns remain open so that we can address them in the discussion. To make that easier, here is a short summary of where each point stands.
W1/Q1 (detectors, overrides). The held-out audit you asked for shows that no detector, ours or the generalists', can yet end rollouts on its own, so the veto remains a safety net for every arm (now stated in Sections 5.3, E); the operator never sets a reward. Within that shared loop ours are the more reliable: they caught 71–100% of successes, against 0–31% for the generalist detectors, and on Hang-towel would have ended 9–10 of 32 rollouts wrongly, against 13 of 32 for Robometer (F.s.). The other three generalist arms needed no overrides only because their detectors never fired, so their successful episodes ran to the time cap and were treated like time-outs, with no success signal.
W2/Q2 (matched inputs, shared detector). With our Dense detector shared across arms on Hang-towel (your suggestion), the generalist rewards reached 48% (Robometer (F.s.)) and 56% (RoboReward (F.s.)), against 48% each with their own detectors and 76% for Dense. A shared detector does not close the gap. Camera inputs and polling do not explain it either: in matched replays, scene-only vs. multi-view input and 0.5 vs. 4 Hz polling were not separated, and our scene-only critics still lead both generalists on outcome AUROC (0.960, vs. 0.912 for Robometer (F.s.) and 0.850 for RoboReward (F.s.)).
W3 (progress). On the reversed actions you asked for, our critics return to their starting value, while both generalists keep credit for undone progress (median +0.146 and +0.750). Under interruption, ours and Robometer (F.s.) both stay flat.
W4 (effort). Per task, 13–28 min of human time and 2.5–198 L40S GPU-hours, beyond a one-time ≈ 30 min workspace setup (Section D.11).
Where the reward is ultimately used, ours improves the policy more than either generalist reward. Offline RL over eight tasks, which involves no detector, polling or operator, gives a 49.0% success rate for Dense, against 37.5% for Robometer (F.s.) () and 34.5% for RoboReward (F.s.), from 32.0% for the base policy. Online, Dense reaches 80% on Wire-rope and 76% on Hang-towel, against 48% for either generalist on both tasks (pooled ).
Which of these points, or any other, still falls short for you, and what evidence would most change your assessment? We will do our best to provide it within the discussion period.
Thank you for the careful review; revision changes are in blue in the updated paper.
W1, Q1. Held-out detector statistics and operator overrides. On the 32 learning-on rollouts of each arm, none were used for calibration (Table 13, Section D.5):
Each rollout is classified once, on the detector's first fire: TP is a success whose first fire falls in its final 10 s, EARLY a success that fired earlier, FP a failure that fired. Precision = TP / (TP + EARLY + FP); recall = TP / successes.
Held-out precision is low. Without the operator, 9–10 of 32 Hang-towel rollouts and 3–5 of 32 Wire-rope rollouts under our critics would have ended on a wrong declaration, so the veto is a necessary safety net, not a formality; now stated in Sections 5.3 and E. The operator never sets a reward: each step's reward is a −1 step cost plus the critic's value, and a veto only withholds the success terminal. Success rates in Fig. 6 are operator labels on separate evaluation rollouts, not detector outputs. The three generalist arms with 0 wrong ends never fired (recall 0%), so their successes ran to the time cap.
W2, Q2. Matched cameras and polling; shared detector. The RL agent's own observation is identical across arms; only the reward server differs.
W3. Beyond VOC. We agree, and added three quantitative tests, including the reversed and interrupted actions you suggest (Sections 5.1, D.8, D.9):
In short, our critics score the current state rather than episode history, and on real progress loss they are above chance and comparable to the best generalist. In RL, where the reward is ultimately used, ours improves the policy more than either generalist reward, offline and online (Fig. 6).
W4. Setup effort. Now reported in Section D.11 (Tables 15, 16), with human time and compute separated:
In total, each task takes about 13–28 min of human time and 2.5–198 GPU-hours beyond the workspace setup. Section E discusses future directions for reducing this cost.