- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper trains a VLM critic to predict the current subtask and score action chunks for that subtask, using best-of-N TD backups. After fine-tuning on each target task, it chooses among actions proposed by a base policy. The paper tests this on four long-horizon two-arm manipulation tasks.
Critical strengths
- Learning values for one subtask at a time reduces the prediction horizon while still accounting for the chosen action.
- Shirt-hang success rises from 10/24 to 22/24, with gains on all four tasks. The subtask-conditioning and pretraining tests also help show which parts of the approach are useful.
Critical weaknesses
- The critic runs without supplied subtask labels, but the full system still needs some manual switching. Appendix A.3 says an operator advances the base policy's prompts for grocery-packing and LEGO. Fully automatic steering has therefore not been shown on those two tasks.
- Most tests of individual design choices use only shirt-hang and 24 evaluation attempts, without results from independently trained critics. Smaller gains, such as lid-sealing's 10/24 to 15/24, are harder to judge. More training runs would help show how consistent these gains are.
- The full-task baselines in Table 2 omit subtask prediction and conditioning, while SeeQ uses both. Table 4 shows these choices affect performance. The comparison therefore does not isolate the shorter return horizon. A full-task critic with the same auxiliary supervision would be a cleaner test.
- Backups stop at subtask boundaries, so the critic ignores effects on later subtasks. Appendix A.4 acknowledges that local progress can make later steps harder, but does not measure this tradeoff. Testing cases where finishing one subtask obstructs another would help judge the method's long-horizon benefit.
Questions for authors
- What are grocery-packing and LEGO success rates when prompts advance automatically, for example using the critic's predicted subtask?
- Can you report paired trial outcomes and results from independently trained critics? Does explicit subtask conditioning also help on tasks beyond shirt-hang?
2 comments
Response to Reviewer fgG2 (2/2)
Most ablations use one task and 24 trials. Can you report paired outcomes and independently trained critics? Does conditioning help beyond shirt-hang?
We have extended the pretraining and subtask-conditioning ablations to grocery-packing (Sections 4.3–4.4; Tables 3–4). Explicit conditioning improves performance from 15/24 to 22/24 on shirt-hang and from 11/24 to 17/24 on grocery-packing, holding the subtask-prediction objective present in both variants. Robot pretraining also improves performance on both tasks.
Regarding pairing, Appendix A.1.5 specifies that trial i uses the same physical setup and task instruction across methods. This provides matched evaluation conditions. However, the revision reports aggregate success counts rather than the per-trial outcome matrix, and does not include results from independently trained critic seeds. We therefore do not claim that these evaluations quantify training-seed variability.
We agree that smaller differences, including lid-sealing’s 10/24 → 15/24, should be interpreted with appropriate caution. The new ablations strengthen the evidence that the main design choices help across tasks, while additional trials and independent training runs would improve the precision and reproducibility assessment. Further, the project website provides evidence of qualitative differences in the rollout with and without steering to better demonstrate the impact of the SeeQ method.
A full-task critic with the same auxiliary supervision would be a cleaner comparison.
We agree and have added precisely this comparison in Section 4.4 and Table 4. The new task-level TD-BoN critic learns the same subtask-prediction objective and conditions action values on its decoded subtask. Its backups nevertheless cross subtask boundaries and propagate the full-task completion reward.
On shirt-hang, this critic achieves 13/24, compared with 12/24 for the original task-level baseline and 22/24 for SeeQ. Thus, the additional language supervision and conditioning do not close the gap to the subtask-level formulation.
| Method | shirt-hang success |
|---|---|
| Task-level TD-BoN | 12/24 |
| Task-level TD-BoN + subtask prediction and conditioning | 13/24 |
| Subtask-level TD-BoN + subtask prediction and conditioning (SeeQ) | 22/24 |
Stopping backups at subtask boundaries ignores effects on later subtasks. Can the authors measure this tradeoff?
We agree that this is a real limitation of the objective, and this is why we included it in the limitation section (A.4) of the paper. Equation 5 deliberately stops bootstrapping at subtask boundaries to shorten credit assignment. Consequently, an action can receive a high value for completing the current subtask even if it makes a later subtask harder. Appendix A.4 explicitly discusses this failure mode, including the example of disturbing another object while packing the current one. Ultimately, there’s no free lunch, approximating global progress with local progress will always come at a cost, but with our ablations we show that this cost is necessary to obtain high performance gains.
Our empirical claim is that this tradeoff is beneficial for steering the evaluated base policies: the metric in Table 1 is complete-task success, and SeeQ improves that metric on all four tasks.
We hope the new experiments and these clarifications address your concerns about the contribution and the scope of the evaluation.
Response to Reviewer fgG2 (1/2)
Thank you for your careful review and constructive suggestions! We have added a task-level critic with matched language supervision and extended our component ablations to grocery-packing. Changes to the paper’s draft have been marked in blue, the changes are in sub-sections 4.3, 4.4 and 4.5. We have also created new sub-sections A.2.3 and A.2.4 in the appendix. We also clarify below the distinction between the critic’s subtask inference and the base policy’s prompting protocol.
You are correct that the complete system on these two tasks uses manually advanced prompts for the base policy, as documented in Appendix A.3. But note crucially, that the SeeQ critic does not use manual prompt switching, it is the base policy that requires manual prompt switching. The choice of the base policy is orthogonal to the central contribution of our paper which is on training an effective critic. All methods including baselines on these tasks use manual switching for the policy, which makes the comparison absolutely fair.
In more detail, our claim about automatic subtask inference is actually applied to the critic: SeeQ receives the observation and full-task instruction, predicts the active subtask itself, and uses that prediction to score candidate action chunks. The operator does not supply or overwrite the critic’s subtask prediction.
The same fixed base policy and prompting protocol are used for the base-policy evaluation and critic comparisons. The reported improvements therefore measure the benefit of value-based action selection under that shared protocol: 9/24 → 17/24 on grocery-packing and 5/24 → 10/24 on LEGO-disassembly.
This distinction matters because SeeQ is designed to improve a supplied policy through action selection, without requiring a particular policy-training method or high-level controller. The reason for providing subtasks to the policy is that without it, pi-0.5 fails to achieve non-zero success rate, in other words, it fails to generalize over the space of packing/LEGO configurations. Note that this is a limitation of the policy and SeeQ is agnostic to the policy and is only concerned with the critic.