acceptodds
Under review as a conference paper at ICLR 2027

SeeQ: Training Generalist Value Functions for Long-Horizon Robotic Manipulation

10 views

Abstract

Despite rapid progress, generalist robot policies remain brittle on complex, long-horizon tasks that comprise multiple stages or require repeated attempts and deliberation on the same underlying stage before success. Q-value functions can improve these policies by ranking candidate actions or guiding policy improvement, but learning from sparse task-level rewards entails long credit-assignment horizons, difficult Bellman backups, and broad data-coverage requirements. We introduce SeeQ (Subtask-elicited Q-functions), which instead learns Q-values for the currently active subtask. This shortens the value-prediction horizon and enables effective learning with temporal-difference (TD) objectives. During training, subtask-level annotations present in offline robot data provide the decomposition and enable learning from broad, potentially suboptimal robot datasets. To eliminate the need for human annotations or modular subtask prediction systems at test time, our Q-function architecture is trained to autoregressively predict the active subtask in natural language before estimating its value. We instantiate SeeQ using a base vision-language backbone, pretrain it on diverse open-source robot manipulation data, and finetune it on downstream tasks. Across four real-world manipulation tasks on two bimanual robot platforms, the SeeQ value function substantially improves best-of-𝑁 policy steering.

Primary area:
applications to robotics, autonomy, planning
Keywords:
value functions, reinforcement learning, generalist, robot learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fgG2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper trains a VLM critic to predict the current subtask and score action chunks for that subtask, using best-of-N TD backups. After fine-tuning on each target task, it chooses among actions proposed by a base policy. The paper tests this on four long-horizon two-arm manipulation tasks.

Critical strengths
  1. Learning values for one subtask at a time reduces the prediction horizon while still accounting for the chosen action.
  2. Shirt-hang success rises from 10/24 to 22/24, with gains on all four tasks. The subtask-conditioning and pretraining tests also help show which parts of the approach are useful.
Critical weaknesses
  1. The critic runs without supplied subtask labels, but the full system still needs some manual switching. Appendix A.3 says an operator advances the base policy's prompts for grocery-packing and LEGO. Fully automatic steering has therefore not been shown on those two tasks.
  2. Most tests of individual design choices use only shirt-hang and 24 evaluation attempts, without results from independently trained critics. Smaller gains, such as lid-sealing's 10/24 to 15/24, are harder to judge. More training runs would help show how consistent these gains are.
  3. The full-task baselines in Table 2 omit subtask prediction and conditioning, while SeeQ uses both. Table 4 shows these choices affect performance. The comparison therefore does not isolate the shorter return horizon. A full-task critic with the same auxiliary supervision would be a cleaner test.
  4. Backups stop at subtask boundaries, so the critic ignores effects on later subtasks. Appendix A.4 acknowledges that local progress can make later steps harder, but does not measure this tradeoff. Testing cases where finishing one subtask obstructs another would help judge the method's long-horizon benefit.
Questions for authors
  1. What are grocery-packing and LEGO success rates when prompts advance automatically, for example using the critic's predicted subtask?
  2. Can you report paired trial outcomes and results from independently trained critics? Does explicit subtask conditioning also help on tasks beyond shirt-hang?
2 comments
AuthorsOpenReview ↗

Response to Reviewer fgG2 (1/2)

Thank you for your careful review and constructive suggestions! We have added a task-level critic with matched language supervision and extended our component ablations to grocery-packing. Changes to the paper’s draft have been marked in blue, the changes are in sub-sections 4.3, 4.4 and 4.5. We have also created new sub-sections A.2.3 and A.2.4 in the appendix. We also clarify below the distinction between the critic’s subtask inference and the base policy’s prompting protocol.

Grocery-packing and LEGO still use manual prompt switching. What happens when prompts advance automatically?

You are correct that the complete system on these two tasks uses manually advanced prompts for the base policy, as documented in Appendix A.3. But note crucially, that the SeeQ critic does not use manual prompt switching, it is the base policy that requires manual prompt switching. The choice of the base policy is orthogonal to the central contribution of our paper which is on training an effective critic. All methods including baselines on these tasks use manual switching for the policy, which makes the comparison absolutely fair.

In more detail, our claim about automatic subtask inference is actually applied to the critic: SeeQ receives the observation and full-task instruction, predicts the active subtask itself, and uses that prediction to score candidate action chunks. The operator does not supply or overwrite the critic’s subtask prediction.

The same fixed base policy and prompting protocol are used for the base-policy evaluation and critic comparisons. The reported improvements therefore measure the benefit of value-based action selection under that shared protocol: 9/24 → 17/24 on grocery-packing and 5/24 → 10/24 on LEGO-disassembly.

This distinction matters because SeeQ is designed to improve a supplied policy through action selection, without requiring a particular policy-training method or high-level controller.  The reason for providing subtasks to the policy is that without it, pi-0.5 fails to achieve non-zero success rate, in other words, it fails to generalize over the space of packing/LEGO configurations. Note that this is a limitation of the policy and SeeQ is agnostic to the policy and is only concerned with the critic.

AuthorsOpenReview ↗

Response to Reviewer fgG2 (2/2)

Most ablations use one task and 24 trials. Can you report paired outcomes and independently trained critics? Does conditioning help beyond shirt-hang?

We have extended the pretraining and subtask-conditioning ablations to grocery-packing (Sections 4.3–4.4; Tables 3–4). Explicit conditioning improves performance from 15/24 to 22/24 on shirt-hang and from 11/24 to 17/24 on grocery-packing, holding the subtask-prediction objective present in both variants. Robot pretraining also improves performance on both tasks.

Regarding pairing, Appendix A.1.5 specifies that trial i uses the same physical setup and task instruction across methods. This provides matched evaluation conditions. However, the revision reports aggregate success counts rather than the per-trial outcome matrix, and does not include results from independently trained critic seeds. We therefore do not claim that these evaluations quantify training-seed variability.

We agree that smaller differences, including lid-sealing’s 10/24 → 15/24, should be interpreted with appropriate caution. The new ablations strengthen the evidence that the main design choices help across tasks, while additional trials and independent training runs would improve the precision and reproducibility assessment. Further, the project website provides evidence of qualitative differences in the rollout with and without steering to better demonstrate the impact of the SeeQ method.

A full-task critic with the same auxiliary supervision would be a cleaner comparison.

We agree and have added precisely this comparison in Section 4.4 and Table 4. The new task-level TD-BoN critic learns the same subtask-prediction objective and conditions action values on its decoded subtask. Its backups nevertheless cross subtask boundaries and propagate the full-task completion reward.

On shirt-hang, this critic achieves 13/24, compared with 12/24 for the original task-level baseline and 22/24 for SeeQ. Thus, the additional language supervision and conditioning do not close the gap to the subtask-level formulation.

Methodshirt-hang success
Task-level TD-BoN12/24
Task-level TD-BoN + subtask prediction and conditioning13/24
Subtask-level TD-BoN + subtask prediction and conditioning (SeeQ)22/24

Stopping backups at subtask boundaries ignores effects on later subtasks. Can the authors measure this tradeoff?

We agree that this is a real limitation of the objective, and this is why we included it in the limitation section (A.4) of the paper. Equation 5 deliberately stops bootstrapping at subtask boundaries to shorten credit assignment. Consequently, an action can receive a high value for completing the current subtask even if it makes a later subtask harder. Appendix A.4 explicitly discusses this failure mode, including the example of disturbing another object while packing the current one. Ultimately, there’s no free lunch, approximating global progress with local progress will always come at a cost, but with our ablations we show that this cost is necessary to obtain high performance gains.

Our empirical claim is that this tradeoff is beneficial for steering the evaluated base policies: the metric in Table 1 is complete-task success, and SeeQ improves that metric on all four tasks.

We hope the new experiments and these clarifications address your concerns about the contribution and the scope of the evaluation.

Reviewer_sekkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SeeQ, a framework for training generalist Q-functions for long-horizon robotic manipulation. The key idea is to learn Q-values for the currently active subtask rather than the full task, thereby shortening the temporal credit-assignment horizon. SeeQ uses a pretrained vision-language model to predict the active subtask in natural language and conditions value estimation on this prediction. The Q-function is pretrained on diverse robotic data using subtask-level temporal-difference learning with best-of-N backups, then finetuned on downstream tasks to steer a fixed base policy.

The authors evaluate SeeQ on four real-world bimanual manipulation tasks across two robot platforms. Best-of-8 steering improves average success from 35.4% to 66.7%. Ablations examine alternative value-learning objectives, robot pretraining, and explicit subtask prediction and conditioning.

Critical strengths
  1. Strong real-world experimental results. The paper demonstrates substantial improvements across four challenging long-horizon bimanual manipulation tasks on two robotic platforms, increasing average success rates from 35.4% to 66.7%. The improvements are particularly notable on shirt-hang (10/24 to 22/24) and grocery-packing (9/24 to 17/24). These results provide convincing evidence of the practical effectiveness of SeeQ for policy steering, including in settings with different data collection regimes.

  2. Novel combination of subtask-level value learning and language-based subtask prediction. The paper presents an interesting approach that combines subtask-level returns to shorten the temporal credit-assignment horizon with explicit natural-language prediction of the active subtask. This allows the critic to exploit task structure while automatically determining which subtask to evaluate at inference time.

Critical weaknesses
  1. The reported performance gains do not isolate the effect of auxiliary language supervision - In Table 2, SeeQ achieves 22/24 success on shirt-hang, compared with 12/24 for task-level TD-BoN. However, these methods differ not only in their value-learning horizon but also in the use of auxiliary subtask-prediction supervision and explicit subtask conditioning. Furthermore, Table 4 shows that subtask-level TD without language supervision achieves only 8/24, which improves to 15/24 with the auxiliary subtask-prediction loss and further to 22/24 with explicit subtask conditioning. These results suggest that language supervision plays a substantial role in the observed improvements. However, the paper does not evaluate task-level TD-BoN with the same auxiliary language supervision and subtask conditioning as SeeQ. Such a controlled comparison is necessary to determine whether the gains primarily arise from shortening the TD horizon or from improved language-supervised representations.

  2. The paper lacks a controlled analysis of how the advantage scales with task horizon - Although the paper evaluates several long-horizon manipulation tasks, it does not systematically vary task length or the number of sequential subtasks while keeping the underlying manipulation difficulty comparable. Since the central motivation of SeeQ is to address long-horizon credit assignment through subtask-level value learning, the absence of such an analysis weakens the empirical support for this motivation. In particular, it would be valuable to evaluate the different components of SeeQ across progressively longer task horizons to understand which contributions become more beneficial as the horizon increases and which remain effective even on shorter tasks. For example, comparing subtask-level TD, auxiliary subtask prediction, and explicit subtask conditioning across different horizon lengths would help distinguish improvements specifically attributable to horizon reduction from those arising from generally better representations. Without this analysis, it remains unclear which components are responsible for SeeQ's effectiveness in long-horizon settings.

Questions for authors

Disentangling horizon reduction from language supervision - Could the authors evaluate task-level TD-BoN with the same auxiliary subtask-prediction loss and explicit subtask conditioning as SeeQ? Given that task-level TD-BoN (12/24, Table 2) outperforms subtask-level TD without auxiliary prediction or conditioning (8/24, Table 4) on shirt-hang, this comparison would help clarify whether SeeQ's improvements primarily arise from shorter-horizon value learning or language-supervised representations.

Effect of task horizon on individual contributions - Could the authors provide a controlled analysis of performance as a function of task horizon or the number of sequential subtasks? In particular, how do the benefits of subtask-level returns, auxiliary subtask prediction, and explicit subtask conditioning change as the task horizon increases? Such an analysis would help determine which components specifically address long-horizon credit assignment and which provide improvements independent of task length.

1 comment
AuthorsOpenReview ↗

Response to Reviewer sekk

Thank you for your thoughtful review and positive assessment of our real-world results! We agree that ablating the effect of language supervision and the return formulation is important for understanding the performance gains of our proposed method. Thus, we have added the requested comparisons and expanded the component ablations to a second task. Changes to the paper’s draft have been marked in blue, the changes are in sub-sections 4.3, 4.4 and 4.5. We have also created new sub-sections A.2.3 and A.2.4 in the appendix.

Could the authors evaluate task-level TD-BoN with the same auxiliary subtask-prediction loss and explicit subtask conditioning as SeeQ?

We have now performed this experiment. The new critic is pretrained on RoboCOIN and finetuned on shirt-hang. As in SeeQ, it learns to predict the active subtask and conditions its action values on the decoded subtask at test time. However, it uses task-level rewards and TD-BoN backups that cross subtask boundaries.

Table 2 reports the task-level baseline, and Table 4 reports the two critics with subtask prediction and conditioning.

Methodshirt-hang
Task-level TD-BoN12/24
Task-level TD-BoN + subtask prediction and conditioning13/24
Subtask-level TD-BoN + subtask prediction and conditioning — SeeQ22/24

Thus, supplying the task-level critic with the same language supervision and conditioning does not recover SeeQ’s performance. This supports the contribution of the subtask-level return formulation beyond the auxiliary language objective.

We also observed the 8/24 result without language supervision, which in fact shows that shortening the target horizon alone is insufficient in this setting. The intended conclusion is that subtask-level targets and explicit subtask elicitation work together. The critic needs both a learnable local value target and a representation of which subtask that value refers to.

To test whether the language components help beyond shirt-hang, we extended the ablation to grocery-packing. Subtask-level TD achieves 10/24 without either component, 11/24 with the prediction loss alone, and 17/24 with both prediction and explicit conditioning. On shirt-hang, the corresponding results are 8/24, 15/24, and 22/24. These results show that the auxiliary loss alone has a task-dependent effect, while explicit conditioning improves steering on both tasks.

The clearest evidence currently available is therefore the new matched-language comparison above: even after adding subtask prediction and conditioning, task-level TD-BoN remains substantially below SeeQ on shirt-hang. We will make the distinction between this empirical result and the broader hypothesis about horizon dependence explicit in the paper.

We hope this new comparison addresses the main concern about attributing SeeQ’s gains to language supervision alone.

How do the benefits of the individual components change as task horizon increases?

We agree that a controlled sweep over task horizon would provide a stronger characterization of this dependence. We choose the packing task as a basis for such an analysis since subtasks can be arbitrarily composed here, and we can evaluate the performance of each subsequent subtask (analogous to results on the CALVIN benchmark previously). Figure 12 shows the success rate vs number of subtasks chart for all the packing evaluations we conducted (including ablations). Here we see that SeeQ is able to complete more subtasks before it’s first failure as compared to the other checkpoints. This indicates that SeeQ is more robust to longer horizons as compared to other methods.

Grocery-packing trials (out of 24) that packed all of the first k objects in the instruction.

Methodk = 1k = 2k = 3k = 4k = 5
SeeQ2222212019
Base policy2320181715
Task-level MC2120161412
Task-level TD2118131311
Subtask-level SARSA1914131312
Task-specific ResNet-502121201613
SeeQ minus pretraining2219151413
No prediction, no conditioning1917131311
Prediction, no conditioning2220191615

The motivation for SeeQ is that task-level TD must propagate a sparse completion signal across the entire remaining task, allowing estimation errors to accumulate across successive backups. SeeQ terminates backups at subtask boundaries, so the prediction horizon depends on the active subtask rather than the full task. Appendix A.5 reports a median subtask duration of 7.5 seconds in RoboCOIN, and average subtask durations of approximately 5–25 seconds in the downstream tasks.

New qualitative analysis: We also add a qualitative analysis to supplement this motivation in sub-section A.2.3. Figure 11 shows that with task-level returns the action-candidates have significantly lower spread than the error band, inhibiting it’s ability to accurately rank actions.

Reviewer_p9rROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Learning value functions over long time horizons via TD-leanring is really challenging, especially when the task space is large (i.e. you are learning a multitask / "generalist" policy). This paper proposes a method SeeQ that breaks down tasks into subtasks via language and then learns a Q-fn over these shorter horizons (subtasks) rather than the full task. This is made possible by availability of offline robot data that already provides subtask annotation. Experiments are conducted on four tasks / two bimanual setups in the real world and results indicate that hte proposed framework learns meaningful subtask progress and helps with best-of-N policy steering.

Critical strengths

The paper is well written and easy to follow. The idea of breaking down tasks into subtasks is pretty well accepted and this work extends that to Q-fn learning / task progress estimation which I think is a small but meaningful contribution. The experimental setup looks pretty reasonable, with nontrivial tasks where the subtasks are pretty clear and the authors include a set of ablations that all make intuitive sense to me.

Critical weaknesses

I do have a few concerns regarding the experiments and overall justification of the method:

  • As afar I can tell, the main claim that subtask-level return makes TD-learning effective is never actually tested directly. The comparison to task-level TD in table 2 seems to change both subtask prediction loss, conditioning, discount factor and whether it is subtask or task (what authors refer to as return scope). Comparing task-level returns in Tab 2 with subtask-level returns in Tab 4 (ablation) even seems to indicate that it's a negative gain on the one task where both baselines are evaluated (shirt-hang)? Am I misunderstanding? Would be great if the authors can please clarify. Additionally, it seems a bit odd to me that these ablations are separated into two separate tables; I would strongly suggest running the Tab 4 ablations on grocery-packing from Tab 2 as well and just have a single table that ablates all of these different factors independently, otherwise it's hard to tell where the gain actually comes from.
  • I kind of allude to this above, most ablations are done only on a single task (shirt-hang). It would be useful to have at least one other task to do ablations on so that the effect of task selection and design choice is more clear. The number of trials (24) is also quite low, I understand that evals are time consuming but when some differences are quite small (e.g. 8 vs 12 successes) increasing the number of trials is necessary in order to have something that resembles statistical significance.
  • The value / subtask boundary prediction is only evaluated qualititatively on a single trajectory ( Fig 4), it would be a lot more meaningful if the authors could (1) include more qualitative examples across tasks, and (2) also report some quantitative metrics to show that it works on average.
  • It's not clear how performance improves/degrades with N in the best-of-N evals. It would be useful to know whether larger N improves results / stagnates or if performance starts degrading past N=8 that the authors use because it becomes difficult for the Q-fn to differentiate the value of trajectories.
  • I think the proposed method is also quite similar to the language-conditioned skill value fn in SayCan? Can the authors please explain how these two methods differ?

This is a pretty small but important detail, for some reason none of the links in the paper are clickable. It would be great if the authors can please fix that in a future revision since it makes checking citations difficult.

Questions for authors

I would appreciate if the authors can please respond to my comments in the "weaknesses" section above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer p9rR (1/2)

Thank you for your thoughtful review and suggestions! To address your concerns, we have added new experiments to compare task-level and subtask-level TD learning under matched language supervision. We have also added new experiments to extend our ablations to grocery-packing and find the same trends between various comparisons as we found on shirt-hang task in our submission. Finally, we have also added quantitative evaluations of subtask and value predictions on held-out episodes. We will respond to your concerns below.

Does the comparison isolate the benefit of subtask-level returns from subtask prediction and conditioning?

We agree that the original comparison in Table 2 did not separate these factors sufficiently. To address this, we have added a task-level TD-BoN critic with the same auxiliary subtask-prediction objective and explicit subtask conditioning as SeeQ. This critic is pretrained on RoboCOIN dataset and finetuned on shirt-hang task data. It predicts the active subtask and conditions action values on its prediction at test time, but it propagates the full-task completion reward and backups crossing subtask boundaries.

This variant achieves 13/24. In comparison, we reported 12/24 for task-level TD without subtask prediction or conditioning, and 22/24 for SeeQ. Thus, adding the same language supervision and conditioning to the task-level critic does not close the gap. With these new results, we believe the latest experimental evidence isolates the benefit of subtask-level returns from subtask prediction and conditioning. These results are now included in Section 4.4 and Table 4.

For clarity, we collect the relevant comparisons from Tables 2 and 4 here:

Return scopeSubtask prediction lossExplicit subtask conditioningshirt-hanggrocery-packing
TaskNoNo12/249/24
TaskYesYes13/24—
SubtaskNoNo8/2410/24
SubtaskYesNo15/2411/24
Subtask — SeeQYesYes22/2417/24

While we also observed that subtask-level TD without either language component achieves 8/24, below the task-level baseline’s 12/24, we do not interpret this as evidence that shortening the return horizon alone always improves performance. Rather, the results support the combination of subtask-level targets and explicit identification of the subtask being evaluated. Without such identification, the critic must implicitly infer which local objective its value represents. Section 4.4 and Figure 8 examine this issue.

Regarding discount factors, task-level methods use 0.9995 versus 0.999 for subtask-level methods to accommodate their longer return horizon and avoid excessively attenuating the terminal reward (Appendix A.1.3). Without this, ~6.6% of the pre-training data would correspond to a return-to-go (Monte Carlo return) of < 1e-4. We retain this distinction in the new comparison. It therefore controls for language supervision and conditioning, while comparing the task-level and subtask-level return formulations with their respective discounts.

Most ablations use only shirt-hang, and 24 trials may be insufficient to interpret small differences.

We have extended the pretraining and subtask-conditioning ablations to grocery-packing (Sections 4.3–4.4; Tables 3–4). This also tests the design choices under a different data regime: shirt-hang uses mixed policy and human-intervention data, while grocery-packing uses expert teleoperation demonstrations.

On grocery-packing, SeeQ achieves 17/24, compared with 11/24 without robot pretraining and 13/24 for the task-specific ResNet critic. With subtask-level returns, adding the prediction loss alone changes performance from 10/24 to 11/24, whereas explicitly conditioning values on the predicted subtask raises it to 17/24. Thus, the benefit of explicit conditioning appears on both evaluated tasks, while the effect of the auxiliary loss alone is task dependent.

Methodshirt-hanggrocery-packing
Base policy10/249/24
Task-specific critic (ResNet-50 + MLP, TD-BoN)16/2413/24
SeeQ minus robot pretraining8/2411/24
Subtask-level TD, no prediction or conditioning8/2410/24
Subtask-level TD, prediction without conditioning15/2411/24
Task-level TD-BoN, prediction and conditioning13/24—
SeeQ (subtask-level TD, prediction and conditioning)22/2417/24

We agree that small differences should be interpreted cautiously, and we will run more trials for the final version of the paper. Appendix A.1.5 specifies that the trial uses the same physical setup and instruction across methods, reducing variation from evaluation conditions. The additional task broadens the evidence for the main design choices. Further, the project website provides evidence of qualitative differences in the rollout with and without steering to better demonstrate the impact of the SeeQ method.

AuthorsOpenReview ↗

Response to Reviewer p9rR (2/2)

Please provide more qualitative examples and quantitative evaluation of subtask/value predictions.

We have added a quantitative evaluation in Section 4.5 and Figure 6, using 1,879 held-out RoboCOIN episodes and 410,610 annotated frames. The pretrained critic predicts the annotated subtask correctly on 86.9% of frames, with a median per-task accuracy of 94.2%. Matching ignores capitalization and a trailing period, and Figure 6 additionally breaks results down by embodiment.

We also compare predicted values with the recorded discounted Monte Carlo return to the end of the active subtask. The mean signed Q − MC gap is +0.044 with the ground-truth subtask and +0.042 with the predicted subtask. This is a diagnostic of value predictions relative to recorded behavior, rather than an absolute-error metric: SeeQ evaluates best-of-N continuation, which can differ from the recorded continuation. In other words this value is not an error value, it is instead an upper bound on the overestimation. The best-of-N continuation is what allows the critic to evaluate an improved policy (over the demonstration policy).

For additional qualitative examples, Figure 9 shows held-out pour-tea, grocery-packing, and shirt-hang trajectories. These illustrate value changes during reorientation, failed grasps, and unintended releases. Figure 8 examines behavior around subtask transitions: explicit conditioning aligns value resets with predicted subtask changes, whereas the variant without conditioning exhibits more gradual or misaligned resets. The frame-level accuracy above complements these examples, although it does not directly measure boundary-localization accuracy.

How does performance change with the number of candidates N?

Our experiments use N = 8 for both TD-BoN backups and test-time steering, consistently across the relevant methods (Section 4). The claim supported by these experiments is that SeeQ improves a fixed base policy at this candidate budget. In general, the improvements from best-of-N depend upon whether the base policy can propose diverse actions or not, and whether the value function is able to meaningfully differentiate them. We do not intend to claim that increasing N yields monotonic improvements or constitutes an established scaling axis. Given the time spent in running other ablations, we have not been able to run a complete sweep over N values. But we agree that an N-sweep would help characterize the tradeoff between candidate diversity and critic ranking errors. The current revision does not report this experiment, so performance beyond N = 8 remains an open empirical question, and we will mark this as a limitation for now (and aim to run this experiment for the final version).

How does SeeQ differ from SayCan?

We agree that SayCan is related in its use of language-conditioned skill values and we will add a discussion detailing differences in Section 5. The distinction lies in how those values are learned and used.

In SayCan (Ahn et al., 2022), an LLM scores candidate skills for their relevance to the task, and affordance/value estimates help select which skill to execute. SeeQ instead learns a critic that predicts the currently active subtask from the observation and task instruction, then ranks low-level action chunks proposed by a generalist policy. Its subtask prediction is generated as text rather than selected by a frozen LLM from an enumerated (fixed set) skill menu.

Our contribution is therefore not the general idea of language-conditioned skill values. It is a recipe for training a generalist action-value function using offline subtask-level TD learning, explicit subtask elicitation, and broad robot pretraining, then adapting that critic to steer policies on downstream manipulation tasks. We will make this relationship more explicit in the related-work discussion.

Links in the paper are not clickable.

Thank you for pointing this out. The revised PDF now includes clickable citations and URLs.

We hope the new controlled comparison and broader evaluations address your main concerns, and would appreciate your reconsideration of the assessment in light of these results. Please let us know if you find your concerns addressed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions and new results

Dear Chairs and Reviewers,

Thank you to the reviewers for the thoughtful questions that helped us improve the paper.

SeeQ introduces a recipe for learning a generalist, language-conditioned Q-function offline. We pretrain a vision-language critic on diverse robot demonstrations to predict the active subtask and learn its action values through temporal-difference learning. The resulting critic can then be adapted to steer a policy on a new task.

The reviews prompted three revisions.

  • Language supervision and the return horizon. All reviewers asked whether subtask language supervision could explain SeeQ’s gains on its own. In the new comparison, adding the same supervision to a task-level critic does little to close the gap (Section 4.4, Table 4). This is consistent with the difficulty of assigning credit from a distant task reward.
  • Ablations on a second task. We extended the core ablations to grocery-packing and found the same overall trends as on shirt-hang for robot pretraining and conditioning on the predicted subtask (Section 4.3, Tables 3 and 4).
  • Understanding the critic. The main paper evaluates how accurately SeeQ predicts the active subtask and estimates its value (Section 4.5, Figure 6). The appendix examines why subtask-level values help distinguish candidate actions (Appendix A.2.3, Table 5 and Figure 11) and how the benefits of steering evolve as the robot completes more subtasks (Appendix A.2.4, Figure 12).

Together with our original set of ablations and qualitative analyses, these additions show how a critic trained on broad robot data can improve policy execution on challenging, long-horizon bimanual tasks across robot platforms. We hope the revisions address your questions, and we would be grateful for your consideration of the revised paper.

Thank you again for your time and feedback.

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.