acceptodds
Under review as a conference paper at ICLR 2027

SARM2: Multitask Stage Aware Reward Modeling for Self Improving Robotic Manipulation

Abstract

Fine-tuning vision-language-action (VLA) policies for long-horizon manipulation still largely depends on behavior cloning (BC), which requires costly high-quality demonstrations and limits policies to the demonstration distribution. Reward models support learning beyond demonstrations through data reweighting and dense reward signals for reinforcement learning (RL). However, existing reward models struggle to combine multitask scalability and precision: task-specific stage-aware models require per-task annotation and retraining, while general vision-language reward models often miss subtle progress in long-horizon tasks. We introduce SARM2, a multitask stage-aware reward model combining a shared action-primitive stage estimator with a multi-gate Mixture-of-Experts (MMoE) value head for accurate dense rewards. We use its dense rewards in SPIRAL, an offline-to-online pipeline for policy self-improvement from demonstrations and autonomous rollouts. On ten tasks, SARM2 reduces overall value-estimation MSE by 33.3% versus multitask SARM and 53.5% versus the best evaluated general-purpose reward model baseline. Starting from offline RL policies, SPIRAL raises success rates from 60% to 100% on Folding Flattened Shorts, 33.3% to 66.7% on Folding Crumpled Shorts, and 50% to 90.0% on Cleaning Whiteboard, supporting a robot data flywheel driven by accurate dense rewards.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Reward Modeling, Robotic Manipulation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HoKEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SARM2 combines shared action-primitive estimation with a primitive-conditioned mixture-of-experts head to predict task progress. SPIRAL uses progress differences and sparse episode outcomes for offline-to-online policy improvement from demonstrations and physical robot rollouts, following one-time reward-model calibration. Evaluation covers reward prediction on ten tasks and policy improvement on two.

Critical strengths
  • Reusable primitives enable shared stage estimation across tasks. Reward prediction improves over dense and general-purpose baselines, with architectural ablations supporting the design at comparable active head compute.
  • Real-robot folding and whiteboard-cleaning results show substantial gains over sparse rewards and fine-tuned Robometer. Controlled reward-source comparisons and improvements with both DICE-RL and AWR support practical usefulness.
Critical weaknesses
  • The downstream benefit of expert routing remains unresolved. Dense SARM shares the primitive estimator but uses a single MLP head; it and the stage/routing ablations are absent from policy-learning comparisons. Lower prediction error does not establish better control than these simpler alternatives. Robometer's budget-limited LoRA adaptation and the shared SARM2-trained offline initialization further limit conclusions about independently optimized complete pipelines.
  • Evidence for broader reliability is limited. Policy gains cover only two tasks, with 30 evaluation trials per condition and no reported variation across independent training runs. Eight of ten reward-model benchmark tasks occur in the primitive source corpus, while value training and rollout calibration remain task-dependent. Transfer to substantially different tasks is therefore weakly tested.
Questions for authors

N/A

4 comments

We thank Reviewer HoKE for the detailed comments on the policy-learning evidence.

W1. Downstream benefit of expert routing.

(a) Dense SARM in policy learning. We ran SPIRAL with the multitask SARM baseline (shared primitive estimator, single compute-matched dense head) under the same DICE-RL backend, calibration data, offline checkpoint, and rollout budget as SARM2 (also requested by Reviewer pdmk, W2). Each entry is from 30 trials:

Reward (DICE-RL, rollout rounds)Shorts flat SR (%)Shorts crumpled SR (%)Whiteboard SR (%)Whiteboard prog.
Sparse56.716.753.30.825
RMT-FT83.340.063.30.883
Multitask SARM (dense head)86.750.066.70.900
SARM2100.066.790.00.975

(b) Robometer's offline initialization. We ran the offline DICE-RL warm-up with RMT-FT rewards on the same demonstrations. Each entry is from 30 trials:

Offline DICE-RL warm-upShorts flat SR (%)Shorts crumpled SR (%)Whiteboard SR (%)Whiteboard prog.
Sparse rewards (Table 2)33.316.733.30.617
RMT-FT rewards43.316.736.70.692
SARM2 rewards (Table 2)60.033.350.00.825

RMT-FT's own offline checkpoint is much weaker than SARM2's and close to the sparse-reward one. Starting RMT-FT's rollout rounds from the SARM2-trained checkpoint therefore gives it a better starting point than its own pipeline would, so the shared initialization favors RMT-FT rather than limiting it, and the RMT-FT rollout result in Table 2 is likely an optimistic estimate of a fully independent RMT-FT pipeline.

(c) Robometer's adaptation budget. RMT-FT follows the official LoRA recipe. We use LoRA rather than full fine-tuning so that adaptation does not overwrite the representations Robometer learned in large-scale pretraining, which are the source of its generalization. LoRA is also already expensive here: its 2,000 steps take about the same wall-clock time as SARM2's 10,000 training steps while using a GPU with twice the memory, and the model has about 4B parameters versus about 200M for SARM2.

(d) Stage and routing ablations in policy learning. We ran w/o SE and w/o MG through SPIRAL with the same DICE-RL backend, calibration data, offline checkpoint, and rollout budget as SARM2. Each entry is from 30 trials:

Reward (DICE-RL, rollout rounds)Shorts flat SR (%)Shorts crumpled SR (%)Whiteboard SR (%)Whiteboard prog.
w/o SE (no stage estimator)86.746.770.00.892
w/o MG (single gate)90.056.773.30.925
SARM2100.066.790.00.975

Both ablations improve the policy less than SARM2 in every condition, and removing the stage estimator hurts more than removing multi-gate routing (for example, 70.0% and 73.3% vs. 90.0% whiteboard success). This ordering matches demo MSE in Table 1 (0.034 for w/o SE, 0.026 for w/o MG, 0.020 for SARM2), so both the stage signal and multi-gate routing improve downstream policy success, not only reward prediction accuracy.

W2. Breadth of the evidence.

(a) Two policy tasks and single training runs. We respectfully believe that the two policy tasks are representative of the challenging settings that general-purpose robot policies face today: Folding Shorts is long-horizon deformable-object manipulation from both flat and crumpled initial states, and Cleaning Whiteboard is long-horizon bimanual tool use with force-sensitive, contact-rich wiping while one arm stabilizes a movable board. Each real-robot RL run on such tasks is costly, so each row comes from one training run, as stated in the revised Table 2 caption, and the uncertainty below covers execution variability. To show that the main comparisons are not within evaluation noise, the table gives success rates with 95% Wilson intervals over the 30 trials:

DICE-RL, rollout roundsShorts flat SR (%)Shorts crumpled SR (%)Whiteboard SR (%)
Sparse56.7 [39.2, 72.6]16.7 [7.3, 33.6]53.3 [36.1, 69.8]
RMT-FT83.3 [66.4, 92.7]40.0 [24.6, 57.7]63.3 [45.5, 78.1]
SARM2100.0 [88.6, 100.0]66.7 [48.8, 80.8]90.0 [74.4, 96.5]

One-sided Fisher exact tests show that SARM2's improvement is statistically significant in every condition: p < 0.05 against RMT-FT on all three (p = 0.026, 0.035, and 0.015) and p < 0.003 against sparse rewards on all three. SARM2's gains over both baselines are therefore not explained by evaluation noise.

(b) Primitive-corpus overlap and transfer. We test the frozen estimator on two new tasks, Organize Sunglasses and Build Wood Block Tower, which lie outside the corpus and are very different types of robotic tasks from the 10 benchmark tasks (articulated-object manipulation and assembly) (details in our response to Reviewer uNTG, W2):

TaskIn primitive corpusStage-estimation accuracy (%)
Average of the 8 corpus tasksyes85.08
Average of the 4 unseen tasks (2 benchmark, 2 new)no85.99
Organize Sunglasses (new)no79.62
Build Wood Block Tower (new)no87.80

Accuracy on unseen tasks is on par with tasks in the corpus, so the estimator transfers to substantially different tasks without retraining.

(c) Task-dependent value training and calibration. This is correct: SARM2 is a multitask reward model, not a zero-shot one, and the revised introduction states this explicitly. The per-task cost is small, however. We fine-tune the multitask SARM2 on Organize Sunglasses with only 1 hour of demonstrations, one fifth of the 5 hours per benchmark task, for 2k steps (about 1 hour on a single 5090 GPU; details in our response to Reviewer uNTG, W1):

ModelDemos per taskEvaluation tasksDemo MSE
SARM2 (Table 1)5 hS1 / S2 / Overall0.006 / 0.031 / 0.020
SARM2, fine-tuned1 hOrganize Sunglasses0.0085

SARM2 reaches 0.0085 MSE on this substantially different task, lower than its error on the S2 benchmark tasks (0.031), without stage labels or retraining of the primitive estimator. Calibration is also light: it is done once per task with segment-level labels on about 100 rollouts, fewer than two hours of annotation (Appendix 6.14), and later rounds need no further progress annotation.

Thank the authors for the response. Most of my concerns are addressed. I have updated my rating.

AuthorsOpenReview ↗

Thank you, Reviewer HoKE

Dear Reviewer HoKE,

Thank you very much for your positive feedback and for raising your score. We truly appreciate the time and care you put into reviewing our work, and your thoughtful comments have helped us improve the paper.

Best regards, The Authors

Reviewer_pdmkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a multitask reward model (SARM2) that uses a mixture-of-experts value head to produce dense progress rewards. It incorporates these rewards into an offline-to-online policy improvement pipeline (SPIRAL). It demonstrates improved reward prediction across several manipulation tasks and increased policy success on two real robot tasks.

Critical strengths
  • Non-trivial gains on real robot tasks
  • Improved reward prediction across ten tasks
  • Useful ablations and evaluation with two RL backends
Critical weaknesses
  1. The paper has good results but novelty might be limited. The developed model mainly uses known techniques like rewinding or MoE routing.
  2. Some ablations that would be useful are missing. For example a comparison against multi-task SARM in the policy learning case. Multitask SARM could be used in SPIRAL with the same DICE-RL backend, calibration data, starting policy, and rollout budget as SARM2. This would clarify whether MoE routing improves downstream policy success beyond improving reward-prediction accuracy. The correlation is clear but this would be a cleaner result.
  3. While SARM2 outperforms some baselines in terms of reward prediction, I don't think it is as scalable as those. This is because it is supervision-dependent. It requires hours of labeled primitive data, in-domain value training, and several annotated rollouts per downstream task. This is already acknowledged in the paper where it says SARM2 is not a zero-shot method, but I think it would be better to clarify this earlier in the paper.
  4. It is weird that only S_1 variant works almost as good as the model that was trained on the full dataset. I think this requires more discussion.
  5. The paper uses the difference between progress values as a reward signal. While this is intuitive and has precedent, Tan et al.'s RoboDopamine, which is cited by the paper, had a different recipe and they compared their recipe with the differencing approach, concluding their recipe was giving better results. This is not necessarily a major concern, but I would love this paper to try that approach as well in case it performs better.
  6. Finally, Sontakke et al.'s RoboCLIP paper seems to be missing from the citations even though the paper discusses reward models that were built on CLIP.
Questions for authors

The weaknesses #2, #4 and #5 are also questions.

2 comments

We thank Reviewer pdmk for the positive assessment and the useful suggestions. Our responses to W2, W4, and W5 also answer the corresponding questions.

W1. Novelty. We use rewinding (from ReWiND) and multi-gate MoE as components and cite them as such. MoE is well established in language models and has been explored for scaling robot policies, but its use in reward models is still underdeveloped: to our knowledge, SARM2 is the first multitask robotics reward model built on MoE. Our contribution is how two designs work together:

  • Reusable primitive stages. A task-agnostic vocabulary of 21 action primitives (plus a null class) lets different tasks share stage information, so one stage estimator serves all tasks without per-task stage definitions or annotations. SARM, the closest prior work, needs both for every task.
  • Primitive-conditioned multi-gate routing. SARM2 uses this shared stage information to organize expert specialization: each primitive group has its own gate over a shared expert pool, and the predicted primitive selects the active gate, so experts specialize along primitive groups while related primitives still share experts.

The new downstream ablations show that both designs improve real policy success as well as prediction error, under the same DICE-RL backend, calibration data, offline checkpoint, and rollout budget (30 trials per entry):

Reward (DICE-RL, rollout rounds)Overall demo MSEShorts flat SR (%)Shorts crumpled SR (%)Whiteboard SR (%)Whiteboard prog.
No stage estimator (w/o SE)0.03486.746.770.00.892
Single gate (w/o MG)0.02690.056.773.30.925
Dense head (multitask SARM)0.03086.750.066.70.900
SARM20.020100.066.790.00.975

Removing either design raises prediction error and lowers success in every condition, so the gains in reward accuracy carry over to downstream control. SPIRAL then turns these rewards into real-robot self-improvement on two long-horizon tasks (Table 2).

W2. Multitask SARM in policy learning. We agree. We ran SPIRAL with the multitask SARM baseline (same primitive estimator, compute-matched dense value head) on both tasks, with the same DICE-RL backend, calibration rollouts and annotations, offline starting checkpoint, and rollout budget as SARM2. Each entry is from 30 trials:

Reward (DICE-RL, rollout rounds)Shorts flat SR (%)Shorts crumpled SR (%)Whiteboard SR (%)Whiteboard prog.
Sparse56.716.753.30.825
RMT-FT83.340.063.30.883
Multitask SARM (dense head)86.750.066.70.900
SARM2100.066.790.00.975

Because the two reward models share the primitive estimator and differ only in the value head, the gap comes from MMoE routing: SARM2 outperforms multitask SARM in every condition (100.0% vs. 86.7%, 66.7% vs. 50.0%, and 90.0% vs. 66.7% success; whiteboard progress 0.975 vs. 0.900), while multitask SARM is only slightly better than RMT-FT. The lower reward prediction error of MMoE routing therefore carries over to downstream policy success.

W3. Supervision dependence. We agree. SARM2 is a multitask reward model, not a zero-shot general reward model like Robometer or Robo-Dopamine: each downstream task needs in-domain demonstrations for the value head and, for policy improvement. The revised introduction now states this multitask scope, in addition to the limitations. In exchange, SARM2 is substantially more accurate: even after LoRA adaptation, Robometer's overall MSE (0.043) is more than twice that of SARM2 (0.020).

Primitive data, however, is not needed for each downstream task. The 66-hour primitive dataset is a one-time cost, and the shared stage estimator transfers to new tasks without primitive annotation or retraining. We test it on two new tasks, Organize Sunglasses and Build Wood Block Tower, which are very different types of robotic tasks from the 10 benchmark tasks (articulated-object manipulation and assembly) and were not seen when training the stage estimator (details in our response to Reviewer uNTG, W2):

TaskIn primitive corpusStage-estimation accuracy (%)
Average of the 8 corpus tasksyes85.08
Average of the 4 unseen tasks (2 benchmark, 2 new)no85.99
Organize Sunglasses (new)no79.62
Build Wood Block Tower (new)no87.80

Accuracy on unseen tasks is on par with tasks in the corpus, so a new task reuses the estimator as is.

To quantify how much effort a new task takes, we fine-tune the multitask SARM2 on Organize Sunglasses with only 1 hour of demonstrations (33 episodes), one fifth of the 5 hours per benchmark task, for 2k steps (about 1 hour on a single 5090 GPU; details in our response to Reviewer uNTG, W1):

ModelDemos per taskEvaluation tasksDemo MSE
SARM2 (Table 1)5 hS1 / S2 / Overall0.006 / 0.031 / 0.020
SARM2, fine-tuned1 hOrganize Sunglasses0.0085

SARM2 reaches 0.0085 MSE on the new task, lower than its error on the S2 benchmark tasks (0.031), with no stage labels and no retraining of the primitive estimator. Adding a task therefore costs about one hour of demonstrations and a short fine-tune. The one-time rollout calibration adds segment-level labels on about 100 rollouts, fewer than two hours per task, and no dense stage annotation is needed at any point.

W4. Why the S1-only variant is close to the full model. The S1 variant is trained and evaluated only on the five S1 tasks, so the fair comparison is on those tasks, where SARM2 has 40% lower demo MSE (0.006 vs. 0.010). We read this ablation as showing that adding the five harder S2 tasks does not degrade the S1 tasks and reduces their error, as intended for a multitask model that is extended with new tasks. A likely reason is that S2 tasks share primitives with S1 (e.g., pick, place, fold), so the shared experts see more varied examples of each primitive. The revised Section 4.2 includes this shared-primitive explanation.

W5. Robo-Dopamine's reward recipe. Our dense reward is the change in predicted progress over an action chunk, combined with the sparse episode outcome (a Monte Carlo term in the DICE-RL critic and a mixed advantage in AWR) to keep the policy anchored to task success. Robo-Dopamine instead applies policy-invariant, potential-based reward shaping on top of the task reward. Because SARM2 outputs a value-like progress estimate, it can serve directly as the potential in that recipe. We ran this comparison with the same DICE-RL backend and rollout budget. Each entry is from 30 trials:

SARM2 reward recipe (DICE-RL)Shorts flat SR (%)Shorts crumpled SR (%)Whiteboard SR (%)Whiteboard prog.
Progress difference + sparse MC (ours)100.066.790.00.975
Robo-Dopamine shaping96.770.093.30.975

Robo-Dopamine shaping is slightly better overall (higher success on crumpled shorts and the whiteboard, slightly lower on flat shorts, and the same whiteboard progress), but the two recipes stay within 5 percentage points in every condition. We have added this comparison to the revised appendix (Appendix 6.22, Table 14).

W6. RoboCLIP citation. We will cite RoboCLIP (Sontakke et al., 2023) in the related work on reward models built from pretrained vision-language models. SARM2 uses a frozen SigLIP-2 encoder rather than CLIP, but RoboCLIP's use of video-language similarity as a reward is directly relevant.

Reviewer_uNTGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces SARM2, a multitask, stage-aware reward model that uses a shared action-primitive classifier to identify the robot’s current action and a multi-gate mixture-of-experts network to estimate task progress. Differences in predicted progress are then converted into dense rewards. The accompanying SPIRAL pipeline combines learning from demonstrations, a one-time human calibration step, and autonomous execution to support continual policy improvement. In experiments, SARM2 reduces reward prediction error by 33.3% compared with multitask SARM. Success rates for folding flattened shorts and wiping a whiteboard increase from 60% and 50% at the offline stage to 100% and 90%, respectively.

Critical strengths

This work defines task stages using action primitives such as grasping, placing, and folding, allowing the stage classifier to be reused across tasks and reducing the effort required to define stages and train separate models for each task. Although the primitive training set does not include the shorts-folding task, the classifier still achieves 97.52% accuracy on it, providing direct evidence for this cross-task reuse.

Using the change in predicted progress before and after executing an action chunk as the reward preserves useful actions within failed trajectories. For example, a robot may successfully grasp an object and complete part of a wiping task but fail during the final steps. These earlier, effective actions can still receive positive feedback. This approach is better suited to long-horizon tasks and makes autonomous execution data more useful for learning.

Sharing experts across primitive groups allows the model to draw on similar experiences from different tasks. When training is expanded from the five classic tasks to all ten tasks, the MSE on the classic tasks decreases from 0.010 to 0.006. Under this experimental setting, joint training therefore both broadens task coverage and improves prediction on the original tasks.

Critical weaknesses

The method still requires target-task data, so the cost of adding new tasks is not fully addressed. Although the primitive classifier can be reused, the progress model must be trained on demonstrations of the downstream task and subsequently calibrated using execution trajectories from that task. Shared action primitives therefore mainly eliminate the need to rebuild the stage classifier; they do not eliminate task-specific adaptation of the reward model. The paper explicitly acknowledges on page 9 that it presents a multitask reward model rather than a zero-shot general-purpose reward model. To further substantiate its scalability, the paper should report how much data is required to add a new task and how training and annotation costs change as the number of tasks increases.

Evidence for the primitive classifier’s generalization to unseen tasks is limited. Appendix 6.12 states that eight of the ten evaluation tasks are represented in the source data used to train the primitive classifier; only shorts folding and plate insertion into a drying rack are absent. This does not imply test-trajectory leakage, but most results are still obtained on previously seen task categories. Accuracy on the two unseen tasks is 97.52% and 79.01%, respectively. These results demonstrate some transfer capability, but they are insufficient to establish whether this capability remains reliable across a broader range of new tasks and action combinations.

Primitive recognition is unreliable on some complex tasks, yet the model directly uses the predicted class to select a gate. Table 10 reports primitive classification accuracies of only 57.74% for closing an umbrella and 70.85% for setting a table. Because the model assigns the primitive group using the highest-probability class, classification errors may propagate to expert selection and progress prediction. The paper neither compares performance using ground-truth primitive labels nor quantifies how much reward prediction error is attributable to primitive misclassification. In particular, whether hard gate selection causes abrupt changes in predicted progress at stage boundaries or during ambiguous actions remains untested.

The initial progress labels are derived from normalized time rather than actual task completion. The target in Equation (3) is the negative normalized number of remaining steps, which corresponds to t/Tt/T when expressed as progress. Consequently, pauses and ineffective adjustments in demonstrations receive increasingly higher progress labels simply as time passes. The same physical state may also receive different labels when operators execute the task at different speeds. Stage information and backtracking frames may discourage the model from relying solely on temporal cues, but they do not correct the bias in the supervision labels themselves. Accurate fitting to demonstration labels therefore does not necessarily imply accurate assessment of actual task progress.

Questions for authors

see weaknesses

5 comments

We thank Reviewer uNTG for the careful reading and the concrete suggestions. We respond to each weakness below and will add the new results to the revised appendix.

W1. Cost of adding a new task. We agree that SARM2 is not zero-shot, as stated on page 9 (lines 481–482). Our claim is that multitask pretraining makes a new task simple to adapt to. We add Organize Sunglasses, a task very different from all 10 benchmark tasks and absent from the primitive corpus (task description and snapshots in revised Appendix 6.2; adaptation details in Appendix 6.19). We fine-tune the multitask SARM2 on only 1 hour of demonstrations (33 episodes), versus 5 hours per benchmark task, for 2k steps (about 1 hour on a single 5090 GPU):

ModelDemos per taskEvaluation tasksDemo MSE
SARM2 (Table 1)5 hS1 / S2 / Overall0.006 / 0.031 / 0.020
SARM2, fine-tuned1 hOrganize Sunglasses0.0085

Adapting to a new task is therefore simple: it uses one fifth of the data and one fifth of the pretraining steps. Though this is a complex task, SARM2 with short adaptation reaches 0.0085 MSE on the new task, even lower than its error on the S2 benchmark tasks (0.031). Adaptation is a short fine-tune of the value head on raw demonstrations, with no stage labels and no retraining of the primitive estimator. It works because multitask pretraining already learned the transferable parts: the frozen primitive estimator recognizes the new task's action primitives with 79.62% accuracy without retraining and selects the matching gates, and the shared experts behind those gates learned per-primitive progress profile from all 10 tasks. Only the task-specific part remains: how the new task orders and paces familiar primitives.

Annotation cost. Adding a task requires no dense stage or primitive annotation. The demonstrations only carry coarse adjust/mistake labels, which annotators add during routine data-quality review, and the one-time rollout calibration uses segment-level labels on about 100 rollouts, which takes fewer than two hours per task (Appendix 6.14). The primitive estimator and shared experts are reused as is, so each new task costs about one hour of demonstrations, under two hours of rollout labeling, and about two hours of GPU time (one hour of adaptation and one hour of calibration fine-tuning). The total cost therefore grows only linearly with the number of tasks, at this small per-task rate.

W2. Transfer of the primitive estimator to unseen tasks. We agree that two unseen tasks give limited evidence. We add two new tasks, Organize Sunglasses and Build Wood Block Tower. Both are very different types of robotic tasks from the 10 benchmark tasks (Organize Sunglasses is an articulated-object manipulation task, and Build Wood Block Tower is an assembly task), and neither was seen when training the stage estimator (task descriptions and snapshots in revised Appendix 6.2; action-primitive prediction plots in Figure 14 of revised Appendix 6.12). We evaluate the frozen shared estimator without retraining:

TaskIn primitive corpusStage-estimation accuracy (%)
Average of the 8 corpus tasksyes85.08
Folding Shortsno97.52
Plates into Dish Rackno79.01
Organize Sunglasses (new)no79.62
Build Wood Block Tower (new)no87.80

On both new tasks, accuracy (79.62% and 87.80%) is in the range of the two unseen benchmark tasks. Across all four unseen tasks the average is 85.99%, on par with the 85.08% average on the eight benchmark tasks that occur in the source data. The estimator therefore does not depend on having seen a task: it learns the general visual and motion profile of each action primitive, which carries over to new objects and new action sequences without retraining.

W3. Effect of primitive misclassification. We ran the comparison the reviewer suggests: the same MMoE value head with ground-truth primitive labels in place of predicted ones. The gap between the two rows measures the error attributable to misclassification:

Stage input to the value headS1 MSES2 MSEOverall MSERollout ρ T1Rollout ρ T2
Predicted primitives (SARM2)0.0060.0310.0200.8330.667
Ground-truth primitives (oracle)0.0060.0230.0160.8330.778

Ground-truth primitives lower overall MSE from 0.020 to 0.016, almost entirely on the S2 tasks (0.031 to 0.023), which include the two tasks with the lowest stage-estimation accuracy, and raise T2 rollout ρ from 0.667 to 0.778. Misclassification therefore costs some accuracy on the harder tasks, but the cost is bounded: with predicted primitives, SARM2 still outperforms every baseline in Table 1.

On abrupt changes: most primitive errors occur at stage boundaries, where the estimator switches to the next primitive slightly late and seldom predicts an unrelated class (Figures 12–14 in revised Appendix 6.12). A late switch only changes which gate weights the experts: the expert pool is shared across all gates and top-2 routing mixes experts, so a wrong gate reweights experts that have also learned progress on the same tasks. Predicted progress therefore shows a brief plateau or delay without large jumps, as the progress predictions in Figure 9 (revised Appendix 6.11) show.

W4. Time-normalized progress labels. We thank the reviewer for this point. The paper does not describe our demonstration labels in enough detail. We now describe them in the main paper (Section 3.1.2, page 5), together with their benefit: SARM2 learns to separate genuine progress from hesitation and errors, so stalled behavior earns no reward and mistakes are penalized. The full procedure is in revised Appendix 6.21. The training targets are not purely time-based. In our demonstrations, pauses are labeled as adjust and wrong actions as mistake by human annotators during routine data-quality review. Adjust segments keep a constant label and mistake segments lower it, so the progress label follows actual task progress, and because these labels are task-agnostic, they apply directly to any new task.

Our dataset also has dense stage annotations, which allows a direct comparison with SARM-style stage-aware progress targets. We fine-tune SARM2 with these targets on the same 10-task data (5 hours per task):

Progress targetS1 MSES2 MSEOverall MSERollout ρ T1Rollout ρ T2
Normalized time with adjust/mistake labels (current)0.0060.0310.0200.8330.667
Stage-aware, SARM-style0.0060.0260.0170.8890.833

Stage-aware targets give lower S2 and overall MSE and higher rollout ρ on both tasks, with the same S1 MSE, confirming that more semantic supervision helps. The trade-off is cost: they require per-task stage definitions and per-episode stage annotation, including each new one, which is what SARM2 is designed to avoid. We therefore keep the adjust/mistake labels as the default and will report the stage-aware variant as an option when dense annotations are available.

Dear Reviewer uNTG,

Thank you again for your time and thoughtful review. We have posted detailed responses to your comments, along with a revised manuscript that includes the new experiments and analyses you suggested. As the discussion period is coming to a close, we would be very grateful if you could kindly take a look at our responses and let us know whether they address your concerns. We would be happy to clarify anything further before the deadline.

Best regards, The Authors

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

We sincerely thank all reviewers for their time, effort, and thoughtful feedback. Your insightful questions have been invaluable, and the new experiments and clarifications they prompted have substantially strengthened the paper. In particular, the revision now includes:

  • two new, substantially different tasks showing that the stage estimator transfers to unseen tasks and that SARM2 adapts to a new task with only one hour of demonstrations (Reviewers uNTG, pdmk, and HoKE);
  • downstream policy-learning comparisons with multitask SARM and the stage and routing ablations, showing that our design choices improve real policy success, not only prediction accuracy (Reviewers pdmk and HoKE);
  • a ground-truth-primitive analysis (Reviewer uNTG), a Robo-Dopamine reward-shaping comparison (Reviewer pdmk), and statistical tests on the policy results (Reviewer HoKE).

As the discussion period is coming to a close, we would be very grateful if you could take a moment to review our responses and let us know whether they address your concerns. We would be happy to clarify anything further. Thank you again for your generous engagement with our work.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.