- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces SCOPER-bench and SCOPER-harness for observation-grounded planning and recovery. The benchmark comprises 870 tasks from 35 base templates and their compositions, combining shared rule-based skills, controlled execution failures and situated QA. The training-free harness integrates evidence memory, progress verification, an adaptive agenda and anti-deadlock control.
Critical strengths
S1. Systematic benchmark construction and validation.
S2. Clear gains with informative component ablations.
Critical weaknesses
W1. Architectural gains are not isolated from extra inference and control. A direct baseline sharing the controller and a comparable inference budget is needed to establish the distinctive benefit of the memory and verification design.
W2. Task-aware binding narrows the grounding challenge. The runtime resolver can select privately designated task instances among same-category objects, so the executor supplies part of target disambiguation. This support is disclosed, but its effect on SR and model rankings is unquantified. The results therefore leave open how well System 2 performs instance grounding without task-aware selection.
W3. Situated QA can affect task-execution outcomes. QA answers during execution consume the same 24/36/48-call budgets as physical actions, while trajectories trigger different numbers of questions. This can leave unequal remaining budgets for execution. A QA-free or separate-budget control is missing. Therefore, the diagnostic protocol's effect on SR remains unclear.
W4. Real-observation repair results mix distinct phenomena. The evaluation pools naturally occurring anomalies with incomplete-action nodes created by truncation, which need not represent actual failures or retries. Table 3 does not report separate node counts or accuracies for these sources. Consequently, it seems that the reported 28.4-point repair gains do not establish the improvement on natural anomalies specifically.
Questions for authors
Q1. Does a direct policy with the same anti-deadlock controller and comparable inference budget retain the performance gap? Please also report token usage and latency.
Q2. How often does task-aware binding disambiguate same-category objects, and how does performance change when resolution uses only observation-derived candidates?
Q3. Are task-success rankings and harness gains stable when QA has a separate budget? Please distinguish answers given during execution from those after stopping.
Q4. What are the episode/node counts and repair-call accuracies for natural anomalies and truncated-action cases separately?
Q5. What decoding settings, random seeds and repeat counts were used, and what are the paired confidence intervals for the memory/recovery ablations?