- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SARM2 combines shared action-primitive estimation with a primitive-conditioned mixture-of-experts head to predict task progress. SPIRAL uses progress differences and sparse episode outcomes for offline-to-online policy improvement from demonstrations and physical robot rollouts, following one-time reward-model calibration. Evaluation covers reward prediction on ten tasks and policy improvement on two.
Critical strengths
- Reusable primitives enable shared stage estimation across tasks. Reward prediction improves over dense and general-purpose baselines, with architectural ablations supporting the design at comparable active head compute.
- Real-robot folding and whiteboard-cleaning results show substantial gains over sparse rewards and fine-tuned Robometer. Controlled reward-source comparisons and improvements with both DICE-RL and AWR support practical usefulness.
Critical weaknesses
- The downstream benefit of expert routing remains unresolved. Dense SARM shares the primitive estimator but uses a single MLP head; it and the stage/routing ablations are absent from policy-learning comparisons. Lower prediction error does not establish better control than these simpler alternatives. Robometer's budget-limited LoRA adaptation and the shared SARM2-trained offline initialization further limit conclusions about independently optimized complete pipelines.
- Evidence for broader reliability is limited. Policy gains cover only two tasks, with 30 evaluation trials per condition and no reported variation across independent training runs. Eight of ten reward-model benchmark tasks occur in the primitive source corpus, while value training and rollout calibration remain task-dependent. Transfer to substantially different tasks is therefore weakly tested.
Questions for authors
N/A
4 comments
W2. Breadth of the evidence.
(a) Two policy tasks and single training runs. We respectfully believe that the two policy tasks are representative of the challenging settings that general-purpose robot policies face today: Folding Shorts is long-horizon deformable-object manipulation from both flat and crumpled initial states, and Cleaning Whiteboard is long-horizon bimanual tool use with force-sensitive, contact-rich wiping while one arm stabilizes a movable board. Each real-robot RL run on such tasks is costly, so each row comes from one training run, as stated in the revised Table 2 caption, and the uncertainty below covers execution variability. To show that the main comparisons are not within evaluation noise, the table gives success rates with 95% Wilson intervals over the 30 trials:
| DICE-RL, rollout rounds | Shorts flat SR (%) | Shorts crumpled SR (%) | Whiteboard SR (%) |
|---|---|---|---|
| Sparse | 56.7 [39.2, 72.6] | 16.7 [7.3, 33.6] | 53.3 [36.1, 69.8] |
| RMT-FT | 83.3 [66.4, 92.7] | 40.0 [24.6, 57.7] | 63.3 [45.5, 78.1] |
| SARM2 | 100.0 [88.6, 100.0] | 66.7 [48.8, 80.8] | 90.0 [74.4, 96.5] |
One-sided Fisher exact tests show that SARM2's improvement is statistically significant in every condition: p < 0.05 against RMT-FT on all three (p = 0.026, 0.035, and 0.015) and p < 0.003 against sparse rewards on all three. SARM2's gains over both baselines are therefore not explained by evaluation noise.
(b) Primitive-corpus overlap and transfer. We test the frozen estimator on two new tasks, Organize Sunglasses and Build Wood Block Tower, which lie outside the corpus and are very different types of robotic tasks from the 10 benchmark tasks (articulated-object manipulation and assembly) (details in our response to Reviewer uNTG, W2):
| Task | In primitive corpus | Stage-estimation accuracy (%) |
|---|---|---|
| Average of the 8 corpus tasks | yes | 85.08 |
| Average of the 4 unseen tasks (2 benchmark, 2 new) | no | 85.99 |
| Organize Sunglasses (new) | no | 79.62 |
| Build Wood Block Tower (new) | no | 87.80 |
Accuracy on unseen tasks is on par with tasks in the corpus, so the estimator transfers to substantially different tasks without retraining.
(c) Task-dependent value training and calibration. This is correct: SARM2 is a multitask reward model, not a zero-shot one, and the revised introduction states this explicitly. The per-task cost is small, however. We fine-tune the multitask SARM2 on Organize Sunglasses with only 1 hour of demonstrations, one fifth of the 5 hours per benchmark task, for 2k steps (about 1 hour on a single 5090 GPU; details in our response to Reviewer uNTG, W1):
| Model | Demos per task | Evaluation tasks | Demo MSE |
|---|---|---|---|
| SARM2 (Table 1) | 5 h | S1 / S2 / Overall | 0.006 / 0.031 / 0.020 |
| SARM2, fine-tuned | 1 h | Organize Sunglasses | 0.0085 |
SARM2 reaches 0.0085 MSE on this substantially different task, lower than its error on the S2 benchmark tasks (0.031), without stage labels or retraining of the primitive estimator. Calibration is also light: it is done once per task with segment-level labels on about 100 rollouts, fewer than two hours of annotation (Appendix 6.14), and later rounds need no further progress annotation.
Thank the authors for the response. Most of my concerns are addressed. I have updated my rating.
Thank you, Reviewer HoKE
Dear Reviewer HoKE,
Thank you very much for your positive feedback and for raising your score. We truly appreciate the time and care you put into reviewing our work, and your thoughtful comments have helped us improve the paper.
Best regards, The Authors
We thank Reviewer HoKE for the detailed comments on the policy-learning evidence.
W1. Downstream benefit of expert routing.
(a) Dense SARM in policy learning. We ran SPIRAL with the multitask SARM baseline (shared primitive estimator, single compute-matched dense head) under the same DICE-RL backend, calibration data, offline checkpoint, and rollout budget as SARM2 (also requested by Reviewer pdmk, W2). Each entry is from 30 trials:
(b) Robometer's offline initialization. We ran the offline DICE-RL warm-up with RMT-FT rewards on the same demonstrations. Each entry is from 30 trials:
RMT-FT's own offline checkpoint is much weaker than SARM2's and close to the sparse-reward one. Starting RMT-FT's rollout rounds from the SARM2-trained checkpoint therefore gives it a better starting point than its own pipeline would, so the shared initialization favors RMT-FT rather than limiting it, and the RMT-FT rollout result in Table 2 is likely an optimistic estimate of a fully independent RMT-FT pipeline.
(c) Robometer's adaptation budget. RMT-FT follows the official LoRA recipe. We use LoRA rather than full fine-tuning so that adaptation does not overwrite the representations Robometer learned in large-scale pretraining, which are the source of its generalization. LoRA is also already expensive here: its 2,000 steps take about the same wall-clock time as SARM2's 10,000 training steps while using a GPU with twice the memory, and the model has about 4B parameters versus about 200M for SARM2.
(d) Stage and routing ablations in policy learning. We ran w/o SE and w/o MG through SPIRAL with the same DICE-RL backend, calibration data, offline checkpoint, and rollout budget as SARM2. Each entry is from 30 trials:
Both ablations improve the policy less than SARM2 in every condition, and removing the stage estimator hurts more than removing multi-gate routing (for example, 70.0% and 73.3% vs. 90.0% whiteboard success). This ordering matches demo MSE in Table 1 (0.034 for w/o SE, 0.026 for w/o MG, 0.020 for SARM2), so both the stage signal and multi-gate routing improve downstream policy success, not only reward prediction accuracy.