- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
PIVOSM builds 434 tasks from 185 SpreadsheetBench-Verified seed workbooks by having an LLM inject one problem into each task: a missing parameter (MK), an ambiguous reference (MR), or a false premise (EI). Synthesized Python graders score an execution layer (any reasonable reading passes) and an intent layer (only the withheld reading passes). The headline is that the injected problem costs about 0.35 intent accuracy but only 0.05 execution accuracy, and that letting agents ask recovers a bit over half of that loss.
Critical strengths
- Good motivation: "verified" benchmarks delete the interpretive decision agents face in deployment (Figure 1).
- Splitting execution from intent, with each grader checked against an alternative workbook, is the most reusable idea here.
- Separating ask rate from hit-given-ask is a useful lens (ask rate ranges from 0.26 to 0.95).
Critical weaknesses
-
Human validation of the final items and grader verdicts is not established. "Intent" is the seed gold answer, so everything rests on that reading being a reasonable target. Humans helped design the filter rubrics, but no human check of the final items is reported. The paper says items are audited "without human annotation" (§1) and also "by human annotators" (§2.5), and B.2 shows the filter was applied by two LLMs, one of which is also evaluated. Graders are checked against only one alternative workbook, which says little about how they treat real agent outputs. For a benchmark paper, this is the biggest gap.
-
The headline gap lacks a human baseline, and "silent" failure is not directly measured. An agent loses intent points whenever its defensible guess differs from the seed intent, so 0.35 needs a baseline (e.g., human performance on the implicit prompts) before it can be read as a model weakness. Also, a valid-looking wrong workbook does not show the agent failed to notice the problem; no systematic transcript analysis of flagged assumptions is reported.
-
The interactive results depend on one unvalidated simulator and prompt. There is no control on unambiguous tasks to measure unnecessary asking, and the simulator model is unnamed and its "hit" labels are not checked against humans. Opus-5's 95% ask rate may reflect prompt compliance as much as judgment. Forced-ask and oracle-clarification conditions would show whether the gap comes from not asking or from not using the answer.
-
The explicit setting may not be a clean ceiling. Interactive execution accuracy exceeds explicit on average (0.740 vs. 0.721), which is surprising if explicit differs only in the prompt, though it may also reflect noise or the extra interaction scaffold. The explicit setting may in fact use the original workbook while the implicit one uses an edited one (§4, B.1), which would mix a prompt change with an environment change.
-
EI measures a different skill. A false premise tests contradiction detection, not recovery of missing information. In Figure 6 the agent fails intent for applying the formula the user explicitly gave. Table 3 breaks EI out, but it should also be kept out of the headline intent-gap and recovery numbers.
-
Related work is incomplete. Interactive underspecified variants of SWE-bench Verified (Vijayvargiya et al., 2025) use a very similar full/hidden/interactive design, and HumanEvalComm (Wu & Fard, 2024) uses an almost identical ambiguity/inconsistency/incompleteness taxonomy. Neither is discussed.
Questions for authors
- Was any human check done on the final 434 items and on grader verdicts for real agent outputs? How do the §1 and §2.5 descriptions fit together?
- Which workbook and prompt does the explicit setting use for each axis?
- Which models are the injector and the user simulator, and how often do humans agree with the simulator's hit labels?
- How often do agents ask on fully specified tasks, and how do results change with an oracle-clarification condition?
- In implicit runs, how often does the agent mention the ambiguity or state an assumption?
- How many runs per task were used, and what are the confidence intervals for Tables 2-4?
5 comments
(Cond.) Response to Reviewer p73C
[W5] EI as a distinct skill
We appreciate this insightful observation. The reviewer makes a valid conceptual distinction: while Missing Knowledge (MK) and Multiple Reference (MR) evaluate the recovery of incomplete information, Erroneous Information (EI) tests contradiction detection and premise verification against the observable environment.
- Core headline findings are invariant to excluding EI. To directly address the reviewer's request, we recomputed the headline aggregates over all configurations with EI excluded (MK + MR only). As shown in the table below, the primary conclusions of the paper remain completely robust:
(Cond.) Response to Reviewer p73C
[Q1] Clarification of Wording "Without Human Annotation" and "By Human Annotators".
We acknowledge the phrasing was ambiguous. Please let us clarify here that these descriptions refer to distinct phases of our pipeline and are not contradictory.
- §1 ("without human annotation") describes the scalable task construction pipeline, where instruction injection and paired workbook generation run end-to-end programmatically without requiring manual per-task gold annotation.
- §2.5 ("by human annotators") describes the quality-control rubric formulation, where human experts inspected pilot task samples to design and calibrate the five filtering rubrics prior to automated execution.
[Q4]Oracle-Clarification
Please refer to "Oracle Clarification" in the common official comment for details.
Response to Reviewer p73C
[W1&Q1, W1&Q3] Human Validation on Final Benchmark Item and Curation Component
Since human validation is a common concern, we put the human annotation results at the corresponding parts:
To answer the question about the models used for the simulator and the injector, we would like to clarify that the simulator model is DeepSeek-V4-Flash, and the injector model is Claude-Opus-5.
[W2&Q5] — Human Baseline and Silent fallback
We analyzed 403 Implicit transcripts (31 per configuration across all 13 configurations, balanced over EI/MK/MR and intent outcome). Using the agent's messages and the withheld DoF description, an LLM judge flagged a trial only when the agent explicitly noted the ambiguity or stated an assumption on that DoF (generic planning choices excluded).
DoF flaggedNot flaggedIntent pass89113Intent fail88113
Thus, merely noticing an ambiguity in a non-interactive setting still leaves the agent guessing; resolving it requires querying the user, motivating our Interactive evaluation.
[W4&Q2]— The Explicit Setting as a Ceiling
For all three axes (EI, MK, MR), the Explicit setting uses the unmodified seed instruction and original input workbook directly from SpreadsheetBench-Verified.
Thanks for the careful reading, which correctly notes that Implicit and Interactive use curated workbooks (modified in 239 of 434 tasks). However, this does not introduce an environmental confounder:
For "why interactive execution is slightly higher", we argue that the difference (0.715 vs 0.710) can be due to two reasons:
In summary, workbook curation strictly prevents context leakage without introducing an environmental confounder or altering explicit task difficulty. With the minor execution gap accounted for by sampling noise and interaction scaffolding, the Explicit setting remains a clean, valid upper bound for intent alignment and joint task success.