- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces ReSQ, an ALFRED-based benchmark linking question asking, human responses, and post-answer recovery. ReSQ-Q contains 2,160 questions with annotated intervention points and rationales, derived from 678 failures of an autonomous LLM agent. ReSQ-A supplies three independently collected human answers per question and trains a fixed, privileged-context response oracle. ReSQ-R contributes 514 simulator-replay-verified successful recovery trajectories. Agents are fine-tuned with successful autonomous trajectories alone (S), additional question supervision (SQ), or question and recovery supervision (SQR). Across ten model configurations and three training seeds, SQR improves average success over S by approximately nine percentage points on both ALFRED validation splits. Additional analyses decompose interaction gains and harms, compare question triggers, and study sensitivity to the respondent.
Critical strengths
A useful, linked supervision resource grounded in actual agent failures. The questions are collected at concrete failed-execution prefixes, have three independent human responses, and connect to replay-verified recoveries. The question train/test partitions are disjoint by source trajectory and question annotator, and online evaluation uses the original ALFRED validation splits. This makes the benchmark a substantive resource for studying the complete help-seeking loop, rather than only evaluating question generation in isolation. The empirical analysis explains both benefits and costs of interaction. SQR outperforms SQ for all ten configurations on both validation splits, with three training seeds, and the B=0 comparisons help distinguish autonomous improvements from interaction gains. Equation (3) and Figure 3 expose how broad asking can harm tasks that would succeed without assistance. The timing and respondent comparisons further show why a fixed interaction protocol matters when evaluating candidate agents.
Critical weaknesses
The actual training objective is unresolved where the main validation depends on it. Section 4.1 says every decision in successful trajectories, and the question plus all post-answer decisions in joint recovery examples, contribute to the loss. Appendix B instead says mask_history supervises only the final assistant turn of each context-window example. These descriptions can be compatible if preprocessing creates an example ending at every intended target turn, but the manuscript does not specify that expansion or target-turn coverage. This is a central reproducibility issue: without the processed examples and masks, it is unclear what the S, SQ, and SQR comparisons actually train. The consistent SQR gains are promising, but they cannot resolve the objective ambiguity. I would not assume that the undocumented preprocessing implements the stated supervision in order to recommend acceptance. This is not a claim that the implementation has been proven incorrect.
The online respondent's validity is not demonstrated on the distribution that produces benchmark scores. The offline evaluation uses held-out human-authored questions, whereas the evaluated agents generate their own questions and execution histories. No representative online human-reference or factual/referential audit is reported. Polarity agreement is conditioned on explicitly expressed yes/no responses, and the oracle's additional-information rate differs from humans (31.0% versus 57.2%); the latter is a distribution difference, not itself an error metric. The manual analysis in Appendix E shows that deictic questions can produce polarity–factuality disagreement, making the on-policy gap consequential. Cross-judge agreement and GPT respondent swaps are useful, but neither establishes that the fixed oracle correctly resolves the agents' actual questions. Since the benchmark evaluates recovery through this respondent, a model-dependent response mismatch could affect the measured ability. This requires evidence or a narrower benchmark-validity claim, rather than an assumption of human-equivalent online assistance.
Questions for authors
Clarification of the implemented objective: How are trajectories converted into context-window examples under mask_history? Please give a concrete S and DQR example showing which assistant turns receive loss, and confirm whether every intended post-answer reasoning/action turn and the annotated ask are supervised. What fraction of target turns is lost through truncation?
Targeted validation of online measurement: Do you have a representative audit of the oracle's responses to the agents' actual generated questions, checking factuality, relevance, and referent resolution against the environment and a human/reference answer? Please clarify the exact runtime oracle inputs, including how any uncertainty-type field used offline is obtained online. If this audit is unavailable, what evidence supports the online benchmark-validity claim beyond performance with this particular synthetic respondent?
Clarification of the learning claim: Please distinguish the demonstrated benefit of adding successful recovery demonstrations from evidence that the model specifically learns to use the answer. SQR also changes successful-action supervision and optimizer-update counts. An answer-removal/replacement or supervision-matched control would strengthen that mechanism claim, but is a suggested targeted experiment; its absence alone is not the basis of my rejection recommendation.
7 comments
Response to Reviewer UdeU (2/3)
W2 & Q2. Validity of the oracle on the questions agents actually ask
We agree that the offline evaluation in Table 4 uses human-authored questions and does not directly establish the oracle's validity on questions generated during agent execution. To address this concern, we conducted an additional evaluation using agent-generated questions and newly collected human responses.
Human reference on agent-generated questions. After paper submission, we collected additional human responses to the questions asked online by the Qwen3-8B and Qwen3-14B SQ agents on Valid-Seen (the first question in each asking episode; 497 questions, 246 and 251). The responses were written by one of the three ReSQ-A annotators, given the same interface and privileged context as in ReSQ-A. We then evaluated the oracle's responses to the same questions against this human reference using the Table 4 protocol.
| Respondent | Pol. | AnsFact (Claude / GPT) | Extra (Claude / GPT) |
|---|---|---|---|
| Human | – | 85.3 / 77.9 | 48.9 / 59.0 |
| ReSQ oracle | 91.2 | 81.3 / 76.7 | 38.6 / 40.2 |
The oracle agrees with the human response on polarity in 91.2% of the 114 pairs where both responses express an explicit yes/no polarity. Across all 497 questions, its AnsFact, measuring whether the direct answer is supported by the privileged context, is 4.0 / 1.2 percentage points below the human reference under the Claude / GPT judges. As in the offline evaluation, the oracle provides additional information beyond the question less frequently than humans. Extra measures this difference in response scope rather than correctness.
Runtime inputs. As described in Section 4.2, the oracle receives the agent's question, the task instruction, the interaction history, and privileged simulator context, which consists of goal-object locations, goal progress, the agent's state, visible objects, and receptacle contents. A concrete runtime input is shown below (abbreviated for readability).
[Task]
Put a heated potato slice in the sink.
[Initial obs.]
You arrive at the kitchen ... you see Toaster (1),
..., Fridge (1), CounterTop (1, 2, 3), ...
[History]
[1] recall location of Knife
-> You have not seen Knife before.
...
[8] pick up Potato (1)
-> I am holding Potato (1) now.
[Goal objects]
Potato (1): holding
SinkBasin (1): hidden, 3.21 m, near Sink (1)
Microwave (1): hidden, 2.54 m, near StoveBurner (3)
[Goal progress]
✓ holding target
✗ sliced
✗ heated
✗ placed on SinkBasin
[Agent state]
Holding: Potato (1)
Last skill: pick up Potato (1)
[Visible]
Bowl (1), Cabinet (26), Potato (1) [HELD],
Fridge (1) [OPEN]
[Receptacles]
CounterTop (1): ..., Knife (1), ...
Cabinet (22): Bowl (2)
...
[Question]
Is this the potato you were talking about?
Uncertainty-type field. In the offline evaluation (Table 4), the oracle was additionally given the annotated uncertainty type of each ReSQ-Q question (Appendix E.1), e.g., "[object_identification] Is this the potato you were talking about?". Since agents do not generate uncertainty-type labels, this field is not included in online evaluation. To assess its effect, we regenerated the 432 offline responses without the type field. Of these, 69% were identical, and most remaining responses differed only in wording or additional guidance. Manual inspection identified 17 responses (3.9%) with a changed direct answer. Using the same evaluation protocol as Table 4 (Claude Opus 5 for AnsFact and Extra), the metrics changed modestly: Pol. 85.6 → 86.0; AnsFact 87.0 → 86.1; Extra 31.0 → 34.3. These results extend our oracle evaluation from human-authored questions to questions generated during agent execution, providing additional evidence of its validity under the evaluated settings. We will incorporate these findings and the uncertainty-type ablation into the revised paper and clarify the evaluation protocol.
Response to Reviewer UdeU (3/3)
Q3. Benefit of recovery demonstration vs learning to use the answer
We agree that SQR differs from SQ in more than answer conditioning: it adds 5,866 recovery examples, providing additional successful-action supervision and optimizer updates. Thus, the SQ-SQR comparison alone cannot isolate the effect of learning to use answers. Following the reviewer's suggestion, we conducted an answer-replacement control to examine whether the benefits of interaction depend on the information provided, without retraining the agents.
Setup. At evaluation time, we keep each trained agent fixed and replace the oracle's answer with (a) an empty response, (b) "I don't know", or (c) the oracle's answer to a question from a different task, drawn deterministically from the same agent's original oracle-run answers (excluding the same task). We compare these conditions with the original oracle answers and no asking (B=0). We evaluate on both Valid-Seen and Valid-Unseen using a single training seed (42). Values are SR (%) with questions per episode in parentheses.
Valid-Seen
| Agent | B=0 | Empty | I don't know | Other-task | Oracle |
|---|---|---|---|---|---|
| SQ-4B | 42.1 | 42.9 (0.86) | 43.9 (0.74) | 42.7 (0.91) | 49.0 (0.75) |
| SQR-4B | 43.8 | 44.3 (0.32) | 44.8 (0.31) | 43.5 (0.31) | 52.6 (0.31) |
| SQ-8B | 42.9 | 43.5 (0.73) | 43.3 (0.86) | 43.8 (0.86) | 49.3 (0.71) |
| SQR-8B | 43.2 | 44.1 (0.27) | 44.4 (0.27) | 43.3 (0.27) | 50.4 (0.28) |
Valid-Unseen
| Agent | B=0 | Empty | I don't know | Other-task | Oracle |
|---|---|---|---|---|---|
| SQ-4B | 44.3 | 45.3 (0.94) | 45.2 (0.88) | 46.7 (0.97) | 50.5 (0.80) |
| SQR-4B | 48.1 | 47.6 (0.31) | 47.7 (0.31) | 46.8 (0.31) | 53.3 (0.30) |
| SQ-8B | 46.0 | 45.8 (0.69) | 45.9 (0.93) | 46.3 (0.90) | 49.0 (0.69) |
| SQR-8B | 46.8 | 48.5 (0.22) | 48.7 (0.22) | 46.4 (0.22) | 52.9 (0.22) |
Results. Across both splits and all four agents, the original oracle answers improve success over the no-asking baseline by 3.0–8.8 percentage points. In contrast, replacing the answers with empty, non-committal, or other-task responses yields success rates close to the no-asking baseline (−1.3 to +2.4 points). These results show that the benefits of interaction depend substantially on the information provided, rather than merely issuing questions or receiving additional text. The results also show that SQR agents benefit from informative answers, consistent with learning answer-conditioned recovery behavior. However, this control does not fully separate the effects of recovery supervision from those of additional successful-action training. We will clarify this distinction in the revised paper.
We hope these responses address the reviewer's two main concerns regarding training reproducibility and oracle validity. We have clarified the preprocessing procedure with concrete examples and coverage statistics, evaluated the oracle on agent-generated questions against newly collected human responses, and conducted answer-replacement experiments to examine the role of answer content. We will incorporate these clarifications and additional results into the revised paper. We would appreciate the reviewer's feedback on whether any concerns remain and would be glad to provide further details.
Response to Reviewer UdeU (1/3)
We sincerely thank the reviewer for the careful reading and constructive feedback. We appreciate the reviewer’s recognition of ReSQ as a substantive resource for studying the complete help-seeking loop, as well as the insights from our empirical analysis of the benefits and costs of interaction. We address each concern below.
W1 & Q1. Training Objective
We thank the reviewer for identifying this ambiguity. Section 4.1 defines the supervision objective, while Appendix B describes its implementation but omits a key preprocessing step connecting the two. We clarify this step below with concrete examples and coverage statistics.
Objective. As defined in Eq. (1), every decision in successful trajectories DS is supervised; for question examples DQ, only the annotated Ask decision is supervised; and for question-recovery examples DQR, the annotated Ask and all post-answer decisions are supervised.
Implementation. Each trajectory is stored as a multi-turn ShareGPT conversation. User turns contain task instructions, observations, and human responses; each assistant turn contains zero to two Think lines followed by one Act or Ask line. We use LLaMA-Factory (v0.9.5) with
mask_history=True, applying loss only to the final assistant turn of each training example. We therefore construct a separate prefix-conditioned example ending at every assistant turn selected for supervision. All preceding turns serve as context without contributing to the loss. Supervising the entire final turn covers every Think, Act, and Ask decision in Eq. (1).Examples. The following are actual DS and DQR instances, with observations abbreviated. Each numbered target is a separate training example containing the full conversation prefix. Only its final target receives loss; earlier turns are context only.
Example 1: DS (6 training examples)
Example 2: DQR (Ask and 4 post-answer target turns)
Coverage. We verified the processed training examples against Eq. (1). The 1,530 successful trajectories yield 22,729 training examples covering 35,601 supervised decisions. The 1,728 ReSQ-Q training questions contribute 1,728 supervised Ask decisions; 514 recovery trajectories contribute 5,866 supervised post-answer turns covering 9,084 decisions. Every intended decision is supervised exactly once; preceding turns provide context only.
Truncation. Appendix B states that examples exceeding the 8,192-token limit are truncated. Verification confirms that no training example was truncated. The longest example is 6,204 tokens with the Qwen3 tokenizer (at most 6,162 with the other three tokenizer families); packing is disabled. Thus, no supervised target or history was lost. We apologize for the misleading description.
We will revise Appendix B to explicitly document the per-turn preprocessing procedure, supervision examples, coverage statistics, and verified sequence lengths.