- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper creates a harness for a 27b Qwen model to predict task specific rewards. Their method follows the process of proposing and then revising sub tasks, and then assigning progress scores. They compare to other progress based reward methods (such as robometer) on different reward metrics across a semantic mismatch benchmark and then use their reward for chunk level bc filtering for policy training.
Critical strengths
The authors include multiple strong current progress reward baselines for their simulation results. I also found that the semantic mismatch analysis is strong as for example a robometer assigned progress under wrong object interactions while having high temporal correlation.
Critical weaknesses
I found the pi.5 simulation results to be surprisingly low. Pi 05 fine tuned on the libero datasets as reported on the open pi repository reaches over 92% success rates. Can the authors clarify why these numbers are different, especially for the success filtered BC?
For the real world results, the other reward models are not included. Can you share results with at least success only rollout filtering and if applicable the strongest baselines? These would help establish ARS’s advantage over simple selection methods.
Additionally for each reward to be calculated, it requires a large amount of tokens through a 27b language model. The paper reports token counts, but Including runtime comparisons between robometer per action chunk / trajectory vs ARS would help clarify the practical costs and real world feasibility.
I found the harness description in the main paper hard to follow and unclear on the exact method and tools used by their base model to predict the final progress score. These details are mostly clarified in the appendix but it would be helpful to move them into the main paper.
For their experiments, the rollout collection procedure for the fixed behavior policy is unclearly specified. What are the details of your behavior policy used for the experiments, and how was it trained?
Questions for authors
Included in weaknesses section
5 comments
Reminder to respond to the authors’ rebuttal
With one day left in the rebuttal period, please read the authors’ response and leave a comment. It would help the authors to know which of your concerns they have addressed and which questions remain open, so they can make the most of the time left.
Thanks for your time. Your constructive feedback helps the authors improve their work and strengthens the ICLR community.
Best regards,
Area Chair
Thank you for the response
For W1, why do you think the performance is still very low even when you finetune pi on the much larger success filtered dataset? Reading reviewer JqCP's review, I also agree that including a conventional offline RL baseline with binary success reward such as IQL or other similar methods would be an important comparison.
This also raises another question for me regarding comparisons. The behavior policy used to collect the offline data already achieves 54.6% success, compared with 57.9% for ARS and 44.2% for vanilla BC. ARS improves over rollout success filtered BC by 13.7%, but its improvement over the behavior policy is only 3.3%, with a very large rollout budget. Why are the downstream policies not initialized from the behavior checkpoint?
I also thank the authors for including the timing comparisons, inclusion of the code, as well as real world baselines. These address my concerns on these items directly. Even though the time is much slower than other baselines, I think it is fair to include this as a limitation and assess the work without considering the slow inference speed.
Response to Reviewer Sk7k
Thank you for the helpful questions about practical costs and experimental details.
W1/W5
This is a perceptive observation. OpenPI’s >92% result [1] uses the full LIBERO demonstration dataset. Our behavior policy is obtained by fine-tuning pretrained $ \pi_{0.5} $ on three demonstrations per task from LeRobot [2], then held fixed to collect rollouts. This fixed policy achieves 54.6% success (546/1,000 trials) under the same evaluation protocol as Table 2. The collected dataset contains 53.8% successful trajectories (2,204/4,096). Limited-demonstration SFT initialization also appears in $ \pi_{\mathrm{RL}} $ [3] and RedFlow [4].
All Table 2 policies use the same pretrained π0.5 initialization and learn from the shared rollout pool, with method-specific filtering and no demonstration data. We will clarify these distinctions and the behavior-policy setup in the final version.
References:
W2
Please see Point 3 of our Overall response (2/2) for the real-robot results with success-only rollout filtering and our strongest simulation reward baseline, TOPReward-32B (Table R5).
W3
Please see Point 1 of our Overall response (1/2) for the per-trajectory runtime comparison between Robometer and ARS (Table R1).
W4
We agree that the main-text description should be clearer. Due to space constraints, we placed the detailed workflow and tool descriptions in Appendix A. In the final version, we will include a concise summary in the main text, explaining how visual inspection, event verification, and progress editing produce per-frame scores. We have also uploaded the code to help readers understand and reproduce the workflow.
Specifically, we will add the following summary to Section 3: