acceptodds
Under review as a conference paper at ICLR 2027

Sim2Critic: Learning Reward Critics from Gaussian Splatting Simulation for Real-World Robot Manipulation

Abstract

Simulation offers a key advantage for robot learning: privileged access to the underlying state enables accurate supervision of task success and progress. On real robots, however, this state is hidden behind visual observations, leaving post-training of generalist policies dependent on sparse, human-labeled outcomes. We introduce Sim2Critic, a vision-based critic trained with privileged supervision in a 3D Gaussian Splatting (3DGS) digital twin of the deployment scene and transferred to real-world observations. We study two variants that differ only in their supervision: Sim2Critic (Sparse) learns from sparse success signals, whereas Sim2Critic (Dense) learns from dense progress signals derived from the same simulator state. Across eight real-world manipulation tasks on two robot platforms, both variants transfer to real rollouts, requiring as few as ten real demonstrations per task. It outperforms existing generalist reward models in reward alignment and outcome detection, while using substantially fewer parameters and achieving lower inference latency. Beyond evaluation, the transferred critic supports policy improvement without per-episode reward annotation. It defines returns for offline reinforcement learning on a policy’s own real-world rollouts and provides both rewards and success detection for online reinforcement learning on the robot. Through these two applications, we enhance the success rate of deployed policies by up to 2.2 times. Together, these results show that Sim2Critic turns a scanned workspace into a practical source of per-step reward for real-world robot learning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot manipulation, learned reward model, physical digital twin

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Xrk3OpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper learns visual reward and progress models from physics-enabled 3DGS copies of real workspaces, plus a few successful real demonstrations. The paper tests the models on eight robot tasks and uses them to improve policies through offline and online learning.

Critical strengths
  1. The eight real-robot tasks cover two platforms and include deformable objects. Offline policy success rises from 32% to 49%, showing that the critics help policy learning as well as reward prediction.
  2. The renderer comparison keeps trajectories and labels fixed. This gives a useful test of how realistic scene appearance affects transfer to real images. Comparing sparse and dense labels also helps explain the results.
Critical weaknesses
  1. Online learning still needs some human involvement. Appendix C.2 uses human labels for calibration and allows an operator to reject success declarations. The detector statistics are measured on the calibration data. Held-out false-positive rates and counts of operator overrides are needed to judge how reliably it can stop rollouts on its own.
  2. The online comparisons give the models different camera inputs and polling rates. Sim2Critic uses multiple views, the generalist detectors use the scene view, and RoboReward runs less often. These differences can change when rollouts end and what data are collected, making the effect of critic quality hard to separate.
  3. VOC checks whether scores rise over time in successful rollouts; it does not directly measure task progress. Appendix D uses outcome metrics and example traces to check failures and lost progress. Quantitative tests of interrupted or reversed actions would give stronger evidence that the dense reward tracks the actual state.
  4. A small real-demonstration count does not capture all the setup work. The method also needs scene reconstruction, registration, physics setup, scripted experts and progress rules (Section 4 and Appendix B). Reporting this effort would make the cost of replacing manual reward labels easier to judge.
Questions for authors
  1. What are detector precision and recall on held-out data, and how many success declarations did operators override? Without those overrides, how often would rollouts have ended incorrectly?
  2. Does the advantage remain with the same camera inputs and polling rules, or with a shared success detector while only the dense reward changes?
2 comments

Thank you for the careful review; revision changes are in blue in the updated paper.

W1, Q1. Held-out detector statistics and operator overrides. On the 32 learning-on rollouts of each arm, none were used for calibration (Table 13, Section D.5):

ArmPrecisionRecallOverridesWrong ends w/o operator
Towel, Ours (Sparse)33%71%1710
Towel, Ours (Dense)53%83%119
Towel, RoboReward (F.s.)–0%00
Towel, Robometer (F.s.)24%31%1613
Rope, Ours (Sparse)29%100%65
Rope, Ours (Dense)57%80%43
Rope, RoboReward (F.s.)–0%00
Rope, Robometer (F.s.)–0%00

Each rollout is classified once, on the detector's first fire: TP is a success whose first fire falls in its final 10 s, EARLY a success that fired earlier, FP a failure that fired. Precision = TP / (TP + EARLY + FP); recall = TP / successes.

Held-out precision is low. Without the operator, 9–10 of 32 Hang-towel rollouts and 3–5 of 32 Wire-rope rollouts under our critics would have ended on a wrong declaration, so the veto is a necessary safety net, not a formality; now stated in Sections 5.3 and E. The operator never sets a reward: each step's reward is a −1 step cost plus the critic's value, and a veto only withholds the success terminal. Success rates in Fig. 6 are operator labels on separate evaluation rollouts, not detector outputs. The three generalist arms with 0 wrong ends never fired (recall 0%), so their successes ran to the time cap.

W2, Q2. Matched cameras and polling; shared detector. The RL agent's own observation is identical across arms; only the reward server differs.

  • Shared detector (your suggestion; Section D.7). On Hang-towel we re-ran both generalist arms with our Dense detector at its operating point (4 Hz), so only the dense reward differs. Evaluation success: 56% RoboReward (F.s.) and 48% Robometer (F.s.), vs. 48% each with their own detectors, and 76% for the Dense arm re-evaluated on the same days (its reported rate). A shared detector does not close the gap.
  • Matched cameras and polling (Section D.6). Scene-view-only Sim2Critic critics for both tasks detect outcomes as well as our multi-view critics on replayed sessions (AUROC not separated on 4/4 task × target pairs), and 0.5 Hz polling matches 4 Hz (4/4). On the evaluation banks, scene-only input keeps AUROC (0.971 → 0.960, vs. 0.912 Robometer (F.s.) and 0.850 RoboReward (F.s.) on these two tasks) but lowers VOC (0.922 → 0.886, on par with RoboReward (F.s.) at 0.898): the wrist views mainly help progress tracking.
  • Offline RL has no live detection or polling. All models score the same recorded rollouts through one formula: Dense reaches 98 successes of 200, vs. 75 for Robometer (F.s.) and 69 for RoboReward (F.s.) (Table 12).

W3. Beyond VOC. We agree, and added three quantitative tests, including the reversed and interrupted actions you suggest (Sections 5.1, D.8, D.9):

  • Reversal. 80 successful episodes played forward, then backward. The median same-frame gap |forward − backward| is 0 for ours (by construction), 0.093 for Robometer (F.s.) and 0.398 for RoboReward (F.s.); 95% CIs of both differences from ours exclude 0. At the end of the undone episode both generalists keep progress credit (median end − start 0.146 and 0.750; ours 0).
  • Interruption. On a frame held for 3 s, the median drift is 0 for ours and 0.029 for Robometer (F.s.), so neither drifts; RoboReward (F.s.) places no query inside the hold.
  • Real progress-loss events. 22 drops, falls, knock-overs and towel slips in the evaluation banks, timed by an AI agent and checked by a human annotator, both blind to all model values. AUROC of a value drop at the event vs. matched control windows: Robometer (F.s.) 0.809, Dense 0.778, Sparse 0.775, RoboReward (F.s.) 0.379.

In short, our critics score the current state rather than episode history, and on real progress loss they are above chance and comparable to the best generalist. In RL, where the reward is ultimately used, ours improves the policy more than either generalist reward, offline and online (Fig. 6).

W4. Setup effort. Now reported in Section D.11 (Tables 15, 16), with human time and compute separated:

Human timeCompute (L40S)
Per workspace, one-time≈ 30 min (scan 3, crop 10, scene-camera calibration 15, registration 2)≈ 4 min, plus ≈ 0.6 GPU-h wrist-camera refinement (once, both robots)
Per task, simulation≈ 10 min reviewing rendered successes and failures. Predicate, scripted expert, failure templates and progress ladder (Dense only) are AI-assisted; physics uses Genesis defaults, untuned2.5–27 GPU-h on six tasks; ≈ 118 on Hang-towel and ≈ 198 on Wire-rope, where the cloth and rope solvers dominate
Per task, real3–18 min teleoperation (10 demos; 16 on Hang-mug)none

In total, each task takes about 13–28 min of human time and 2.5–198 GPU-hours beyond the workspace setup. Section E discusses future directions for reducing this cost.

Thank you again for the careful review. We noticed that your rating changed after our response, but we could not find an updated comment, so we would like to understand which concerns remain open so that we can address them in the discussion. To make that easier, here is a short summary of where each point stands.

W1/Q1 (detectors, overrides). The held-out audit you asked for shows that no detector, ours or the generalists', can yet end rollouts on its own, so the veto remains a safety net for every arm (now stated in Sections 5.3, E); the operator never sets a reward. Within that shared loop ours are the more reliable: they caught 71–100% of successes, against 0–31% for the generalist detectors, and on Hang-towel would have ended 9–10 of 32 rollouts wrongly, against 13 of 32 for Robometer (F.s.). The other three generalist arms needed no overrides only because their detectors never fired, so their successful episodes ran to the time cap and were treated like time-outs, with no success signal.

W2/Q2 (matched inputs, shared detector). With our Dense detector shared across arms on Hang-towel (your suggestion), the generalist rewards reached 48% (Robometer (F.s.)) and 56% (RoboReward (F.s.)), against 48% each with their own detectors and 76% for Dense. A shared detector does not close the gap. Camera inputs and polling do not explain it either: in matched replays, scene-only vs. multi-view input and 0.5 vs. 4 Hz polling were not separated, and our scene-only critics still lead both generalists on outcome AUROC (0.960, vs. 0.912 for Robometer (F.s.) and 0.850 for RoboReward (F.s.)).

W3 (progress). On the reversed actions you asked for, our critics return to their starting value, while both generalists keep credit for undone progress (median +0.146 and +0.750). Under interruption, ours and Robometer (F.s.) both stay flat.

W4 (effort). Per task, 13–28 min of human time and 2.5–198 L40S GPU-hours, beyond a one-time ≈ 30 min workspace setup (Section D.11).

Where the reward is ultimately used, ours improves the policy more than either generalist reward. Offline RL over eight tasks, which involves no detector, polling or operator, gives a 49.0% success rate for Dense, against 37.5% for Robometer (F.s.) (p=0.03p=0.03) and 34.5% for RoboReward (F.s.), from 32.0% for the base policy. Online, Dense reaches 80% on Wire-rope and 76% on Hang-towel, against 48% for either generalist on both tasks (pooled p=0.004p=0.004).

Which of these points, or any other, still falls short for you, and what evidence would most change your assessment? We will do our best to provide it within the discussion period.

Reviewer_bnB7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Sim2Critic, an approach to reward learning (estimating task progress) that constructs and uses a digital twin for data generation that can then be used to train a critic. To bridge the visual sim2real gap the authors propose to use gaussian splatting and MV-SAM3d for scene and object reconstruction i nsim (with data captured from a short video capture of the workspace). While building a digital twin requires additional data and a few extra steps in the pipeline, it has the key benefit of being able to train a critic on large amounts of sim data as well as using dense supervision from privileged info that is only available in simulation. The proposed method Sim2Critic is validated on 8 tasks across 2 different robot setups and is compared against off-the-shelf reward models such a RoboReward and Robometer. Results show that the proposed method improves with number of sim trajectories and improves offline and online RL policy improvement of pi 0.5 more than baselines.

Critical strengths

Overall, the paper is easy to read, the proposed method is well motivated and intuitive, and the tasks are reasonably complex given the type of model (pi 0.5) and baselines that they authors compare against. I especially appreciate that the authors include offline and online RL results since those are key use cases of reward models like the one proposed here.

Critical weaknesses

I have a few moderately severe concerns that the authors will have to address before I feel comfortable providing a higher score:

  • Given that the ablations in fig 5b seem to indicate that more simulation trajectories improves performance across most tasks, I wonder why the authors choose to only generate 1k trajectories. It seems rather feasible to collect an order of magnitude more trajectories and just train the critic longer even if model size is kept the same, iit would be interesting to see whether the trend continues or it plateaus at some point (and if so, why does it plateau)
  • The proposed method seems highly related to TwinRL. Can the authors please clarify how the approach differs? It's a Feb 2026 paper so not concurrent work, but it's possible that I'm missing some of the details.
  • The paper generally follows the evaluation framework of Robometer (which is pretty well accepted in recent literature) but seemingly makes some rather arbitrary changes to the specific metrics used (e.g. AUROC instead of kendall) without explanation. It would be helpful if the authors could either change their metric to be the same as in Robometer or at least be explicit about why the evaluation differs.
  • One of the papers main claims is that the privileged supervision from the digital twin is what makes this method attractive. A simple way to validate this claim would be to simply finetune Robometer on the generated sim data; if the finetuned Robometer still performs worse than the proposed method then it's reasonable to conclude that the supervision is key, if the sim-finetuned Roboeter improves over the proposed method then perhaps having large amounts of task-specific data is the most important factor. This is never really tested in the experiments.
  • A number of highly related work is (from what I can tell) never discussed or cited in the paper, including TwinRL mentioned above, asymmetric actor-critic, VIP, LIV, Video-Language Critic, or SIMPLER (not reward learning but relevant nonetheless). Several of the baselines build on these earlier works / is the same line of work so they are worth engaging with in some capacity.
  • I am also concerned about statistical significance for the offline and online RL results. It seems likely that the 46 vs 49% offline RL improvement between sim2critic sparse and dense could be just noise. Regardless, this is a very small difference considering that the dense supervision is a main selling point.
Questions for authors

I would appreciate if the authors can please respond to my comments in the "weaknesses" section above.

1 comment

Thank you for your insightful comments and suggestions; we ran the requested experiments and discussed them below.

W1. Scaling simulation data. We extended Fig. 5(b) to 2,048 episodes. It plateaus: from 1K to 2K, VOC and AUROC are flat on all four tasks (changes within about 0.01), so we stopped at 2K. We hypothesize that beyond about 1K the error is dominated by the residual real-vs-twin gap, which more episodes from the same twin cannot shrink. A 10K buffer would also need about 1–6 TB of storage and 10× the collection compute per task.

W2. TwinRL. TwinRL uses a phone-scanned twin to add fine-tuning trajectories and to find failure-prone configurations that steer human-in-the-loop real-robot RL; real-robot reward and success still come from people. Sim2Critic uses the twin to train the evaluator that replaces those human labels. The two are complementary: our critic could supply TwinRL's real-world reward.

W3. Metrics. We now report Robometer's exact suite for every model (Table 9) and explain each choice (Sections 5.1, D.1):

  • Kendall τ\tau → AUROC. Our ground truth is a binary operator outcome, for which τa=2nsnfn(n−1)(2 AUROC−1)\tau_a=\frac{2n_sn_f}{n(n-1)}(2\,\mathrm{AUROC}-1): the within-task ranking is identical, but AUROC is comparable across tasks with different success rates. Under τa\tau_a our critics stay first and second (0.346 / 0.345 vs. 0.322 for the best baseline).
  • Unnormalized gap → dd. The gap depends on each model's output scale, so we standardize it. Under the unnormalized gap we also rank first and second (0.342 / 0.334 vs. 0.301).
  • VOC, Pearson → Spearman (following GVL). Under Pearson, RoboReward (Few-shot) ranks first (0.907 vs. 0.858 Dense, 0.854 Sparse), because our targets are monotone but non-linear in time. VOC only measures monotonicity on successful rollouts; on real progress-loss events (Sections 5.1, D.8), RoboReward (Few-shot) is below chance (AUROC 0.379) against 0.778 / 0.775 for ours.

W4. Robometer fine-tuned on twin data. We ran this on all eight tasks: Robometer-4B fine-tuned with its own code for 1,000 steps on our twin split plus the same real demonstrations, twin failures labelled by the simulator's success predicate (Sections 5.1, D.2). Eight-task means, same real banks:

ModelDataVOCAUROCdd
Robometer (Few-shot)real demos0.7940.9121.85
Robometer + twintwin + demos0.8680.9382.34
Sim2Critic (Sparse)twin + demos0.9260.9412.45
Sim2Critic (Dense)twin + demos0.9320.9432.50
  • Twin data lifts Robometer on all three metrics: VOC +0.075 [+0.053, +0.099], AUROC +0.026 [+0.009, +0.035], dd +0.49 [+0.38, +0.62] (paired episode-bootstrap 95% CIs).
  • It stays below our critic on VOC (−0.064 [−0.082, −0.050] vs. Dense) and is not separated on AUROC or dd, at about 50× our parameters and 7× our latency (Table 11).
  • Twin data also helps Robometer in offline RL:
TaskRobometer (Few-shot)Robometer + twinOurs (Sparse)Ours (Dense)
Sort-blocks48%56%68%64%
Place-flower28%36%40%36%

Of your two readings: task-specific twin data with simulator-verified outcomes is the main lever, and it helps a generalist too; privileged labels with our compact critic do at least as well at about 1/50 the model size. We revised the contribution statement accordingly (Section 1).

W5. Related work. All added to Section 2 (TwinRL: see W2):

  • Asymmetric actor–critic: privileged state is the critic's input during policy training in simulation, and the critic is discarded at deployment. We use it only as the label of a critic deployed on real images.
  • VIP, LIV, Video-Language Critic: reward or value learned from demonstration and video corpora, the line our generalist baselines come from. Our supervision is simulator-state labels on scene-specific twin rollouts, including failures.
  • SIMPLER (with GSWorld, PolaRiS): simulation used to evaluate policies. We train, in the twin, an evaluator that runs on real observations.

W6. Statistical significance. We agree, and separate two questions (pp-values: two-sided Fisher exact test):

  • Dense vs. Sparse is not resolved by RL. Offline: 98 vs. 92 successes of 200, p=0.62p=0.62 (task-stratified CMH p=0.60p=0.60). Online: p=0.74p=0.74 (Wire-rope), 0.750.75 (Hang-towel). We present them as two target forms of one contribution, twin-grounded critic supervision, not Dense as the selling point. Dense's measurable edge is VOC on the Hang-mug, Wire-rope, and Hang-towel tasks (Table 8).
  • Sim2Critic vs. baselines, the main claim. Offline, 200 trials each: Dense 98 vs. base 64 (p<0.001p<0.001) and vs. the best generalist reward, Robometer (Few-shot), 75 (p=0.03p=0.03); Sparse 92 vs. base (p=0.006p=0.006) and vs. Robometer (Few-shot) (p=0.10p=0.10). Online, pooled over both tasks, 50 trials each: Dense 39 and Sparse 35 vs. 24 for either generalist reward (p=0.004p=0.004, p=0.04p=0.04).
Reviewer_gUMQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper proposes to train a vision-based critic using privileged supervision in simulation and only image-based data in real-life tasks. Notably, the simulation uses 3D Gaussian Splatting which has better visual fidelity than the original simulator's mesh rasterizer. The paper shows that the proposed method, Sim2Critic, has better value-order correlation, AUROC, and success-failure separation than compared methods. Furthermore, the reward generated by Sim2Critic appears to enhance RL fine-tuning results when compared to other approaches.

Critical strengths

None.

Critical weaknesses
  1. There is a whole literature on training asymmetric actor/critic where the model uses privileged information when possible during training. This paper should include this to the related work and hopefully differentiates how itself from them.

References

  • [1]: Ebi, Daniel, et al. "Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access." arXiv preprint arXiv:2509.26000 (2025).
  • [2]: Degrave, Jonas, et al. "Magnetic control of tokamak plasmas through deep reinforcement learning." Nature 602.7897 (2022): 414-419.
  • [3]: Kaufmann, Elia, et al. "Champion-level drone racing using deep reinforcement learning." Nature 620.7976 (2023): 982-987.
  • [4]: Pinto, Lerrel, et al. "Asymmetric actor critic for image-based robot learning." arXiv preprint arXiv:1710.06542 (2017).
Questions for authors
  1. What are the axes in the plots under Figure 4?
  2. In the analysis between Sim2Critic and other reward models, do other methods also use the "categorical-style" value? If not, is this an important implementation detail?
  3. If Sim2Critic estimates the value, while other approaches estimates the reward, then is it not unfair to compare between them for the RL fine-tuning? My understanding here is that the privileged information and the photorealism are the key to this approach but not the "value estimation.
5 comments

Thank you for reviewing our work. Revision changes are in blue in the updated paper.

W1. Asymmetric actor–critic. We added a paragraph on this literature, citing [1–4] (Section 2, "Privileged information in RL"). The difference:

Asymmetric actor–critic [1–4]Sim2Critic
Privileged stateinput to the criticlabel only (g(st)g(s_t) or h(st)h(s_t)); the critic sees RGB + proprioception in training and deployment
Critic at deploymentdiscarded; never sees real imagesthe deliverable: scores real rollouts of policies it never saw (four generalist policies in Table 1; π0.5\pi_{0.5} in RL)
Trained in simulationthe policy, by RLonly the critic, by supervised regression; our method trains no policy in the twin

Because our method never runs RL in the twin, about 10310^3 scripted episodes per task suffice (vs. up to 8×1048\times10^4 episodes in [4] and 2×1062\times10^6 environment steps in [1]), which matters with our costly deformable-object solvers. A critic that only judges outcomes from frames is also less exposed to the dynamics gap than a sim-trained actor, whose actions depend on it directly. As a single-task probe of the alternative, offline RL on 158 simulated Place-coin-and-ball episodes alone (privileged dense reward) gave 20% real-robot success, no gain over the base policy (20%). Offline RL on the same number of real rollouts (158, Table 5), rewarded by our Dense critic, reaches 28% (Table 12).

What Sim2Critic adds to that literature: privileged simulator state can supervise a critic that is itself deployed on real images, and that critic improves a real generalist policy (32% → 49% offline; up to 2.2× online).

Q1. Axes in Fig. 4. x: episode time in seconds. y: each model's output on its native scale, [0,1][0,1] for all models (RoboReward's five rubric levels mapped linearly) except VLAC, [−1,1][-1,1]. Now stated in the caption.

Q2. Categorical value. Partly shared, and not important. Robometer's progress head is also categorical (C51-style bins on [0,1][0,1], read as the expectation); RoboReward outputs one of five rubric levels. Replacing our 101-bin HL-Gauss head with a scalar MSE head, all else fixed (new Section D.10, Table 14), changes VOC by +0.02 to +0.08 and AUROC by −0.04 to +0.03 on the four ablation tasks (MSE − HL-Gauss, six-seed means); the one cost is Place-toy's dd (about 2.0 → 1.0). The categorical head is not the source of the critic's quality; we keep it as the submitted design, which follows Robometer's.

Q3. Value vs. reward in RL. We agree with your reading, and the comparison is like-for-like:

  • Same slot for every model. All models enter offline RL through one formula (Eq. 4), rk=c X(fk)+o~e 1[k=Ke]r_k = c\,X(f_k) + \tilde o_e\,\mathbf 1[k=K_e], with XX the model's per-frame progress score and oeo_e its end-of-episode score; the advantage baseline is fit identically. The baselines also output progress, not per-step reward, and Robometer's own offline-RL recipe uses progress as the dense reward, which we follow. Our critic is never used as a baseline or bootstrap target.
  • The value form is not what helps. Only Sim2Critic (Sparse) has a value-form target (discounted success return). Sim2Critic (Dense) regresses the progress ladder h(st)h(s_t), with no discounting or bootstrapping, and matches or exceeds Sparse in aggregate (Table 1, Fig. 6).
  • What does matter is what you name: rendering (a mesh rasterizer instead of 3DGS drops Hang-mug AUROC from 0.987 to 0.442, Fig. 5a) and twin supervision (fine-tuning Robometer on our twin data lifts it on all three metrics; new paragraph in Section 5.1).

Thank you for including the asymmetric actor-critic literature. While I find the result promising, I do agree with reviewer bnB7 that this work has quite an overlap with TwinRL. More importantly, the response indicates that what matters is the rendering and the twin supervision but not the type of values. To that end I am somewhat confused by what the actual contribution of this paper is---is it a combination of just the former two?

We sincerely appreciate your continued engagement. We are glad that you find the result promising, and thank you for acknowledging the added discussion of the asymmetric actor–critic literature. Below, we answer your question on the contribution and the overlap with TwinRL.

Is it a combination of just the former two? The contribution is what those two are used for: the twin trains the evaluator, not the policy. Privileged simulator state serves only as the label, never as an input, for a compact critic that runs on real images and supplies the reward for real-robot RL, without per-episode reward annotation. The twin-based work discussed in Section 2 of the revised paper, TwinRL included, uses the twin to train policies, generate policy data, or evaluate policies inside the simulator, not to train an evaluator for real rollouts. Rendering and twin supervision are what make our use work, and the paper measures each. We agree that the type of value does not matter, whether value vs. reward or categorical vs. scalar head, and we claim neither; Sparse and Dense are two target forms of a single method.

What our paper shows:

  1. Rendering is what lets the labels transfer. With trajectories and labels fixed, 3DGS beats a mesh rasterizer on both metrics on all four ablation tasks; on Hang-mug, the rasterizer drops AUROC from 0.987 to 0.442 (Fig. 5a).
  2. Twin supervision is the lever, beyond our architecture. Fine-tuning Robometer on our twin data plus the same real demonstrations lifts it from 0.794 to 0.868 VOC and from 0.912 to 0.938 AUROC (Sections 5.1, D.2). The twin also supplies every failure example the critic trains on, and its controlled failures improve both metrics on all four ablation tasks (Fig. 5c).
  3. The resulting evaluator is small and improves a real policy. An 87M critic trained on about 1K scripted twin episodes and as few as 10 real demonstrations per task outperforms zero-shot and few-shot generalist reward models over 50× its size on all three metrics, averaged over eight real tasks (Table 1). As the RL reward, it raises π0.5\pi_{0.5} from 32% to 49% offline, against 37.5% for the best generalist reward (Table 12).

TwinRL. The overlap is the phone-scanned 3DGS twin itself; what is learned from it differs. TwinRL uses the twin on the policy side: to expand fine-tuning trajectories, to run RL inside the twin as a warm-up that fills the replay buffer, and to select configurations for human-in-the-loop real-robot rollouts. It describes no learned reward or success model. Sim2Critic trains no policy in the twin; it trains the model that scores real rollouts and provides the RL reward. Our paper includes one policy-side use of the same twin as a baseline: Co-training (imitation learning with 500 additional successful twin demonstrations per task) reaches 34.5% over eight tasks, against 32.0% for the base policy and 49.0% for offline RL with our Dense critic (Table 12). The two uses are complementary: our critic could provide the reward in a TwinRL-style loop.

We hope this clarifies the contribution and how it differs from TwinRL.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We sincerely thank all reviewers for their constructive feedback. Changes in the revised paper are marked in blue.

Contribution. Sim2Critic turns a phone scan of the deployment workspace into a physics-enabled 3DGS twin and uses the twin's privileged state only as labels to train a compact vision critic that runs on real images. On eight real-world tasks across two robots, it outperforms zero-shot and few-shot generalist reward models with over 50× fewer parameters. Used as the reward for RL, without per-episode reward annotation, it raises π0.5\pi_{0.5} from 32% to 49% success offline and by up to 2.2× online.

New experiments.

  • Categorical vs. scalar value head (gUMQ): replacing our categorical head with a scalar MSE head does not lower VOC and changes AUROC by −0.04 to +0.03, so the categorical head is not the source of the critic's quality.
  • Robometer fine-tuned on our twin data, all eight tasks (bnB7): it improves on all three metrics, stays below our critic on VOC, and is not separated on AUROC or dd. Twin data with simulator-verified outcomes is the main lever, and it helps a generalist too.
  • Simulation scaling to 2K episodes (bnB7): critic quality plateaus beyond about 1K.
  • Robometer's exact metric suite (bnB7): our critics rank first and second under Kendall τa\tau_a and the unnormalized gap.
  • Tests beyond VOC (Xrk3): real progress-loss events, synthetic reversal, and interruption.
  • Online-RL controls (Xrk3): held-out detector precision, recall and operator overrides; arms with a shared success detector; matched camera and polling inputs.

Paper updates.

  • Section 1: contribution statement revised to reflect the Robometer + twin result.
  • Section 2: new paragraph on privileged information in RL (asymmetric actor–critic); TwinRL, VIP, LIV, Video-Language Critic and SIMPLER added.
  • Section 5: metric choices explained; new paragraphs on lost progress and on Robometer with twin data; Fig. 5(b) extended; the operator veto in online RL stated as a required safety net.
  • Appendix: new Sections D.2 and D.5–D.11 with Tables 9, 10 and 13–16, including a setup-effort breakdown; per-task cost and detector statistics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.