acceptodds
Under review as a conference paper at ICLR 2027

Faultless Misreading: Diagnosing Professional Agents Under Incomplete Spreadsheet Requests

Abstract

Large Language Models (LLMs) are increasingly deployed to automate professional work, yet real-world user requests in these domains are inherently underspecified. Evaluating an agent's ability to clarify ambiguities and align with human intent is therefore critical. We study this challenge in spreadsheet manipulation, the everyday task of editing workbooks from written instructions. We introduce PIVOSM (Paired Instructions with Verifiable Open-decision Spreadsheet Manipulation), a benchmark of 434 matched instruction pairs over spreadsheet workbooks. It isolates decision-critical degrees of freedom across three axes of underspecification (Missing Key Information, Multiple Reference and Erroneous Information). Powered by double-layer programmatic rubrics, it provides fully automated scoring that disentangles code execution from human intent alignment. Our evaluation of nine frontier models across explicit, implicit, and interactive settings shows that underspecifying one degree of freedom reduces intent accuracy by , while execution accuracy moves by only , and that letting agents ask closes only half of the resulting intent gap on average. This suggests that while execution capabilities remain stable, recovering unstated human intent via proactive clarification represents the true bottleneck for professional agents operating under realistic underspecification.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Spreadsheet Manipulation, Interactive Clarification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_p73COpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

PIVOSM builds 434 tasks from 185 SpreadsheetBench-Verified seed workbooks by having an LLM inject one problem into each task: a missing parameter (MK), an ambiguous reference (MR), or a false premise (EI). Synthesized Python graders score an execution layer (any reasonable reading passes) and an intent layer (only the withheld reading passes). The headline is that the injected problem costs about 0.35 intent accuracy but only 0.05 execution accuracy, and that letting agents ask recovers a bit over half of that loss.

Critical strengths
  • Good motivation: "verified" benchmarks delete the interpretive decision agents face in deployment (Figure 1).
  • Splitting execution from intent, with each grader checked against an alternative workbook, is the most reusable idea here.
  • Separating ask rate from hit-given-ask is a useful lens (ask rate ranges from 0.26 to 0.95).
Critical weaknesses
  1. Human validation of the final items and grader verdicts is not established. "Intent" is the seed gold answer, so everything rests on that reading being a reasonable target. Humans helped design the filter rubrics, but no human check of the final items is reported. The paper says items are audited "without human annotation" (§1) and also "by human annotators" (§2.5), and B.2 shows the filter was applied by two LLMs, one of which is also evaluated. Graders are checked against only one alternative workbook, which says little about how they treat real agent outputs. For a benchmark paper, this is the biggest gap.

  2. The headline gap lacks a human baseline, and "silent" failure is not directly measured. An agent loses intent points whenever its defensible guess differs from the seed intent, so 0.35 needs a baseline (e.g., human performance on the implicit prompts) before it can be read as a model weakness. Also, a valid-looking wrong workbook does not show the agent failed to notice the problem; no systematic transcript analysis of flagged assumptions is reported.

  3. The interactive results depend on one unvalidated simulator and prompt. There is no control on unambiguous tasks to measure unnecessary asking, and the simulator model is unnamed and its "hit" labels are not checked against humans. Opus-5's 95% ask rate may reflect prompt compliance as much as judgment. Forced-ask and oracle-clarification conditions would show whether the gap comes from not asking or from not using the answer.

  4. The explicit setting may not be a clean ceiling. Interactive execution accuracy exceeds explicit on average (0.740 vs. 0.721), which is surprising if explicit differs only in the prompt, though it may also reflect noise or the extra interaction scaffold. The explicit setting may in fact use the original workbook while the implicit one uses an edited one (§4, B.1), which would mix a prompt change with an environment change.

  5. EI measures a different skill. A false premise tests contradiction detection, not recovery of missing information. In Figure 6 the agent fails intent for applying the formula the user explicitly gave. Table 3 breaks EI out, but it should also be kept out of the headline intent-gap and recovery numbers.

  6. Related work is incomplete. Interactive underspecified variants of SWE-bench Verified (Vijayvargiya et al., 2025) use a very similar full/hidden/interactive design, and HumanEvalComm (Wu & Fard, 2024) uses an almost identical ambiguity/inconsistency/incompleteness taxonomy. Neither is discussed.

Questions for authors
  1. Was any human check done on the final 434 items and on grader verdicts for real agent outputs? How do the §1 and §2.5 descriptions fit together?
  2. Which workbook and prompt does the explicit setting use for each axis?
  3. Which models are the injector and the user simulator, and how often do humans agree with the simulator's hit labels?
  4. How often do agents ask on fully specified tasks, and how do results change with an oracle-clarification condition?
  5. In implicit runs, how often does the agent mention the ambiguity or state an assumption?
  6. How many runs per task were used, and what are the confidence intervals for Tables 2-4?
5 comments
AuthorsOpenReview ↗

Response to Reviewer p73C

[W1&Q1, W1&Q3] Human Validation on Final Benchmark Item and Curation Component

Since human validation is a common concern, we put the human annotation results at the corresponding parts:

  • Final Benchmark Items: Please refer to "Human validation of the Final Benchmark" in the common official comment for details.
  • Grader Verdicts: Please refer to "Grader Validity on Real Agent Outputs" in the common official comment for details.
  • Simulator Hit Labels: Please refer to "Human Check on Simulator Model" in the common official comment for details.

To answer the question about the models used for the simulator and the injector, we would like to clarify that the simulator model is DeepSeek-V4-Flash, and the injector model is Claude-Opus-5.

[W2&Q5] — Human Baseline and Silent fallback

We analyzed 403 Implicit transcripts (31 per configuration across all 13 configurations, balanced over EI/MK/MR and intent outcome). Using the agent's messages and the withheld DoF description, an LLM judge flagged a trial only when the agent explicitly noted the ambiguity or stated an assumption on that DoF (generic planning choices excluded).

  • Most intent failures are silent end to end. In 113 of 201 intent failures (56%), nothing in the transcript flags the withheld DoF: the wrong reading is invisible in the workbook and in the transcript alike.
  • Noticing the DoF does not resolve it. Agents flag the DoF in 177/403 trials (44%, 95% CI [39%, 49%]; 41% reweighted), yet intent accuracy is 50.3% when flagged vs. 50.0% when unflagged:

Intent passIntent failDoF flagged8988Not flagged113113\begin{array}{lcc} & \text{Intent pass} & \text{Intent fail} \\\\ \hline \text{DoF flagged} & 89 & 88 \\\\ \text{Not flagged} & 113 & 113 \end{array}

Thus, merely noticing an ambiguity in a non-interactive setting still leaves the agent guessing; resolving it requires querying the user, motivating our Interactive evaluation.

[W4&Q2]— The Explicit Setting as a Ceiling

For all three axes (EI, MK, MR), the Explicit setting uses the unmodified seed instruction and original input workbook directly from SpreadsheetBench-Verified.

Thanks for the careful reading, which correctly notes that Implicit and Interactive use curated workbooks (modified in 239 of 434 tasks). However, this does not introduce an environmental confounder:

  • Curation only removes context leaks. Our tool strictly deleted pre-filled demo rows, helper artifacts, or stray output cells that would inadvertently reveal the withheld reading and dissolve the injected ambiguity (Appendix B.2).
  • No information loss under Explicit. In the Explicit setting, the user instruction already explicitly states the intended reading. Removing those stray hints takes away zero decision-relevant information under an explicit prompt, while ensuring the implicit prompt remains genuinely ambiguous.

For "why interactive execution is slightly higher", we argue that the difference (0.715 vs 0.710) can be due to two reasons:

  • Sampling Noise. The difference of 0.005 [−0.014, 0.023] has a 95% bootstrap CI spanning zero and its sign varies across configurations, indicating sampling noise.
  • Multi-turn Scaffolding. Interactive agents benefit from multi-turn scaffolding (inspection steps and clarification replies) before final code execution.

In summary, workbook curation strictly prevents context leakage without introducing an environmental confounder or altering explicit task difficulty. With the minor execution gap accounted for by sampling noise and interaction scaffolding, the Explicit setting remains a clean, valid upper bound for intent alignment and joint task success.

AuthorsOpenReview ↗

(Cond.) Response to Reviewer p73C

[W5] EI as a distinct skill

We appreciate this insightful observation. The reviewer makes a valid conceptual distinction: while Missing Knowledge (MK) and Multiple Reference (MR) evaluate the recovery of incomplete information, Erroneous Information (EI) tests contradiction detection and premise verification against the observable environment.

  1. Core headline findings are invariant to excluding EI. To directly address the reviewer's request, we recomputed the headline aggregates over all configurations with EI excluded (MK + MR only). As shown in the table below, the primary conclusions of the paper remain completely robust:
\text{Metric} & \text{Full benchmark (All)} & \text{EI excluded (MK + MR)} \\\\ \hline A_i^{\text{Expl}}\ (\text{ceiling}) & 0.719 & 0.723 \\\\ A_i^{\text{Impl}} & 0.381 & 0.399 \\\\ \Delta_i = A_i^{\text{Expl}} - A_i^{\text{Impl}} & 0.338 & 0.324 \\\\ R_i\ (\text{recovery ratio}) & 0.53 & 0.57 \\\\ \end{array}$$ Even when evaluated strictly on missing-information recovery (MK and MR), removing the withheld clause drops intent accuracy by 0.324, and interactive clarification only recovers 57.5% of that gap. The core phenomena of silent fallback and incomplete interactive recovery do not depend on the inclusion of EI. 2. **Why EI remains essential for benchmark breadth.** In real-world spreadsheet collaboration, users frequently supply instructions containing false premises or buggy formulas that contradict their actual data. Figure 6 reflects this exact failure mode: the agent blindly executes a mathematically impossible formula instead of flagging the mismatch with the underlying table. Rather than evaluating standard execution, EI serves as a crucial test of anti-sycophancy and premise grounding. ## [W6] Incomplete related work Thank you for pointing us to these two papers. We will discuss them in the related work (Appendix A) and add them to the benchmark comparison in Table 1. Below we explain how PIVOSM relates to each. ### Ambig-SWE (Vijayvargiya et al.) Ambig-SWE (Vijayvargiya et al.) is the interactive, underspecified SWE-bench Verified setup the reviewer refers to, and we agree it is one of the closest prior works. It is easy to see two main differences between our work and Ambig-SWE: 1. Task domain. Ambig-SWE evaluates **software engineering**: the agent resolves a GitHub issue by patching a code repository. PIVOSM targets general professional work, instantiated as spreadsheet manipulation tasks that originate from real user requests. 2. Controllability of ambiguity. Ambig-SWE creates underspecified issues via unconstrained LLM summarization and scores them with a single pass/fail test rate, leaving which details are removed uncontrolled. PIVOSM instead withholds a single known decision-relevant degree of freedom (MK/MR/EI) per task, which jointly defines (1) the hidden intent, (2) the exact information the simulator reveals, and (3) whether a clarification question hits the target. This control also enables two-layer grading: execution rubrics pass on both the intended and an alternative workbook, whereas intent rubrics pass only on the intended one. Unlike binary unit tests, this separates execution failure from valid execution under an unintended reading (silent fallback). ### HumanEvalComm (Wu & Fard) We agree the ambiguity categories overlap. In our work, the taxonomy itself is not a claimed contribution. We adopted MK/MR/EI from NoisyToolBench (cited in §2.3 and Table 1) and will also cite HumanEvalComm alongside it. Similarly, PIVOSM differs from HumanEvalComm in the following ways: - Domain: HumanEvalComm modifies the 164 single-function HumanEval problems, whereas PIVOSM covers multi-step, file-based spreadsheet work. - Construction and evaluation: HumanEvalComm's modified problems are written by hand and scored by Communication Rate, an LLM-judged Good Question Rate, and pass rate on the original tests. PIVOSM builds 434 instruction pairs by reverse construction and grades each deliverable with programmatic double-layer rubrics that score execution and intent separately. We thank the reviewer again for helping us position PIVOSM more precisely.
AuthorsOpenReview ↗

(Cond.) Response to Reviewer p73C

[Q1] Clarification of Wording "Without Human Annotation" and "By Human Annotators".

We acknowledge the phrasing was ambiguous. Please let us clarify here that these descriptions refer to distinct phases of our pipeline and are not contradictory.

  1. §1 ("without human annotation") describes the scalable task construction pipeline, where instruction injection and paired workbook generation run end-to-end programmatically without requiring manual per-task gold annotation.
  2. §2.5 ("by human annotators") describes the quality-control rubric formulation, where human experts inspected pilot task samples to design and calibrate the five filtering rubrics prior to automated execution.

[Q4]Oracle-Clarification

Please refer to "Oracle Clarification" in the common official comment for details.

Reviewer_1F8xOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This is a benchmark paper, focused on evaluating how well LLMs handle ambiguous instructions, specifically within spreadsheet manipulation requests.

Critical strengths

I want to start by thanking the authors for a paper that was genuinely a pleasure to read. I have recently had a long string of papers I have recommended to reject, so much so that I started to wonder if I am simply the evil witch of the west or that my standards were too high. I am glad to report I think this is a great little paper, and in fact, I think you can highlight its significance even further. So, strengths:

  1. The paper is not only clear to read; it tells a compelling story, gives the necessary information and does a great job of explaining the choices taken within this work.

  2. I think the work is convincing and sound, and my questions and requests are all minor.

  3. I love the inclusion of interactivity — I think it is so relevant and informative.

Bonus -- the importance -- as models become more capable and more trusted, for example, in coding -- interpretation errors are probably going to become one of our bigger concerns, so I think this work can highlight its significance even further.

Critical weaknesses

Small issues:

  • I didn't really understand the In-Pipeline Verification; maybe it can be explained better.

  • Figure 3 will be easier to read if it is not a circle.

  • Table 2: maybe add benchmark size? Also is the domain just the task type? I found that a bit confusing.

  • While I see the benefits of 'reverse construction pipeline', it does introduce more artificial failure types compared to naturally occurring ambiguity, which I think should be acknowledged / considered / tested for. Also, the limitations state this as due to a lack of resources, while the main paper presents it as a deliberate advantage, which is then undermined somewhat. I think some ablation study to test to what extent the constructed failure type (all reasonable) account for actual faliures in the data (and at what rate) would be really informative here.

  • I would add up or down errors to Table 2.

  • I found the behaviours under the three modes are very interesting; I wonder if it is possible to bring more evidence / proof into the paper (line 375).

Questions for authors

I was not sure if all tasks were human-validated. Importantly, which LLMs did you use for this step? Could it have biased the results?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1F8x

Thank you very much for the warm and encouraging review. We are glad the paper was a pleasure to read, and we fully share your view that interpretation errors will matter more as agents become more capable and more trusted. We address each of your points below.

[W1] In-Pipeline Verification

Thank you for pointing this out. In-Pipeline Verification answers one question: does the synthesized grader really separate execution from intent? We test each grader on two workbooks whose correct verdicts we already know. The intended workbook is the seed's gold answer, and the alternative workbook is a valid answer under a different reading of the implicit instruction. A grader is accepted only if it compiles and produces exactly this pattern:

& \text{Intended workbook} & \text{Alternative workbook} \\\\ \hline \text{Execution rubrics} & \text{pass} & \text{pass} \\\\ \text{Intent rubrics} & \text{pass} & \text{fail} \\\\ \end{array}$$ The right column is the decisive one. If an execution rubric rejects the alternative workbook, it is secretly testing the withheld decision, so a valid but misread output would be wrongly counted as broken. If an intent rubric accepts the alternative workbook, it does not capture the withheld decision at all. In either case, the grader is sent back for resynthesis together with the failing verdicts, and a task whose grader still fails after a fixed number of attempts is dropped. This check is what makes the two scores in our results mean different things. ## [W2] Presentation Effect of Figure 3 Thanks. We replace the sunburst with a vertical stacked bar chart grouped by ambiguity type (MK 150, EI 146, MR 138), with one labeled bar per subtype, so that every count can be read and compared directly. Please see whether it looks better. ## [W3] Benchmark size and domain in the comparison table We believe this refers to the benchmark comparison table (Table 1 in the submission). We will add a *Size* column, where PIVOSM has 434 tasks built on 185 seed workbooks. *Domain* is the application area of a benchmark's tasks, such as open QA, Text2SQL, or spreadsheets. It is neither the task type nor the ambiguity type. Within the spreadsheet domain, PIVOSM tasks are cell-level (233) or sheet-level (201) manipulations, and each carries one ambiguity type (MK, MR, or EI). We will rename the column to *Task domain* and state this in the caption. ## [W4] Constructed vs. Naturally Occurring Ambiguity Please refer to "Constructed Ambiguity and Ecological Validity" in the common official comment for why we isolate one decision per task, and to "Human validation of the final benchmark" for whether the constructed ambiguities are real: 97/100 sampled items are judged genuinely ambiguous, and 88/100 pass all four criteria. You are right that §6 frames reverse construction as a cost concern while the main text presents it as a design choice. In fact, during early pilot studies, we genuinely attempted a purely human-driven construction pipeline. However, we quickly found it methodologically intractable: asking spreadsheet experts to manually write or annotate ambiguous instructions that strictly isolate *exactly one* degree of freedom—while simultaneously formulating programmatic dual-layer rubrics for complex workbook states—imposes immense cognitive overload. In multi-cell environments, manual authoring almost invariably introduces uncontrolled, confounding ambiguities (unlike in text QA). Reverse construction was therefore our conscious solution to guarantee single-DoF isolation and deterministic programmatic grading. We will rewrite §6 so that it states this trade-off consistently with the main text. Thank you for pointing out the inconsistency in the paper's framing.
AuthorsOpenReview ↗

(Cond.) Response to Reviewer 1F8x

[W5] Error bars in Table 2

We now report a 95% bootstrap interval for every entry of Table 2. The full intervals are posted under Reviewer p73C's review "Runs per task and confidence intervals", and the revised paper shows them as ± half-widths. The two quantities our conclusions rest on, the intent gap Δi\Delta_i and the gap between Interactive and the Explicit ceiling, exclude zero for every configuration.

[W6] More evidence for the behaviour around line 375

Thank you, we find these behaviours the most interesting part of the paper too. We expand Appendix C.3 into one illustrated case study per ambiguity type. You are welcome to see the added demos.

[Q1] Human validation and the LLMs used

Human experts first audited pipeline outputs to derive the five filtering criteria (Appendix B.2), and two LLMs then applied these criteria to every task. An item is removed if either model flags any criterion. For the final benchmark, we additionally had humans validate a balanced random sample of 100 items. Please refer to "Human validation of the final benchmark" in the common official comment: 88/100 pass all four criteria.

The LLMs are Claude-Opus-5 as the injector, DeepSeek-V4-Flash and GPT-5.6-Sol as the two filters, and DeepSeek-V4-Flash as the user simulator. We find no sign that they bias the results:

  • The LLMs never decide the intended reading. It is the seed's gold answer from SpreadsheetBench-Verified, and the filters can only remove tasks.
  • DeepSeek-V4-Flash gains no advantage. It is among the lowest-scoring configurations (Explicit AjA_j 0.532), and humans agree with the simulator on 15/16 of its questions ("Details and Human Check on Simulator Model").
  • Claude-Opus-5's lead does not come from injection. It is also the strongest model in the Explicit setting (AjA_j 0.848), which uses the original seed prompt and workbook that the injector never touched.

Thank you. I am happy with this response and will keep my score. I wish you luck!

Reviewer_wZzUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper contributes a benchmark dataset that consists of 434 tasks in spreadsheet workbooks. Analysis on the benchmark shows that understanding under specification is still a bottleneck for agents. Results show that agents often execute plausible interpretations of underspecified requests rather than appropriately clarifying them. Moreover, it shows that although agents often ask relevant questions once they choose to clarify, interaction only partially resolves the gap.

Critical strengths
  • Well designed experiments. The matched explicit/implicit/interactive design isolates what changes when decision-critical information is unavailable, rather than simply showing that some tasks are harder than others.
  • Identifies meaningful failure mode. Results show that models remain technically competent while confidently executing from insufficient or unreliable information. Specifically, they fail to identify that the user instruction does not uniquely determine the intended outcome.
  • Useful study of clarification behavior. The interactive condition shows that when agents choose to ask, their questions frequently come close to identifying the relevant ambiguity, but the resulting interaction recovers only part of the missing intent. This suggests that systems need to consider more than a simple “ask the user” mechanism. Agents need to recognize consequential uncertainty, formulate useful clarification, interpret the answer, and successfully incorporate it into subsequent action.
Critical weaknesses
  • The focus on spreadsheets is quite narrow. The interesting question the authors identify is whether an agent can judge when available evidence is sufficient to act and when clarification is required. Demonstrating this only in spreadsheet editing substantially limits the scope of the benchmark. A stronger benchmark would explore this issue across coding, document editing, research/information tasks, planning, etc.
  • Intent ambiguity is artificially constructed. The benchmark deliberately removes or corrupts specific decision-relevant information, which gives it strong internal validity but uncertain ecological validity. Real uncertainty in the wild may involve multiple interacting ambiguities, implicit context, conflicting evidence, prior interaction history, or evolving goals.
  • The interactive evaluation depends on a simulated user. The model is evaluated within a controlled user-simulation framework rather than through interaction with real users. That makes the experiment scalable and reproducible, but leaves open whether agents would behave similarly with real users who give incomplete, unexpected, or ambiguous responses.
Questions for authors

See comments above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer wZzU

We sincerely thank you for the positive assessment of our experimental design and of the silent-fallback and clarification findings. We address your three concerns below.

[W1] Scope beyond spreadsheets

We agree that the question of when evidence is sufficient to act reaches far beyond spreadsheets, and our §6 already names this scope. We chose spreadsheets as the first instance for two reasons.

  • The workbook is the evidence the agent must weigh. Every ambiguity type in PIVOSM is defined against the workbook itself: an MR reference resolves to several targets in the sheet, and an EI premise contradicts what the sheet contains. Deciding whether to act therefore requires inspecting the environment, not only reading the prompt. The seed requests are also real user questions collected from online forums (SpreadsheetBench), covering both cell-level (233) and sheet-level (201) manipulation.
  • The deliverable can be checked deterministically. This is what makes the double-layer rubric possible: an execution layer that every reasonable reading passes, and an intent layer that only the withheld reading passes.

The construction framework itself is not tied to spreadsheets. Reverse construction, double-layer rubrics, and the two-workbook check in §2.5 only require a verified seed task with a reference deliverable and a programmatic checker. Coding tasks with unit tests and Text2SQL tasks with execution results already meet both requirements. We welcome future work in similar domains.

[W2] Constructed Ambiguity and Ecological Validity

Please refer to "Constructed Ambiguity and Ecological Validity" in the common official comment for why we isolate a single degree of freedom, and to "Human validation of the final benchmark" for the human check: 97/100 sampled items are judged genuinely ambiguous and 88/100 pass all four criteria.

Several of the real-world factors you list are already part of PIVOSM:

  • Implicit context. The agent must ground the request in a workbook it has not seen, and MR ambiguity only appears once the sheet is inspected.
  • Conflicting evidence. EI is exactly a conflict between what the user asserts and what the workbook contains.

Multiple interacting ambiguities, interaction history, and evolving goals are beyond a single-DoF design. Real requests with several entangled ambiguities are likely to make the right reading even harder to recover, so we read PIVOSM's gap as a conservative estimate of silent fallback in the wild.

[W3] Simulated user

The simulated user makes interactive evaluation scalable and reproducible, and we validated its central judgment against humans. Please refer to "Details and Human Check on Simulator Model" in the common official comment: on 100 sampled clarification questions, humans agree with the simulator's hit label in 92 cases, with disagreements split evenly between the two directions.

We also designed the simulator to behave like a terse user rather than an overly helpful one. When a question targets the withheld decision, it only points toward the intended choice in one or two sentences, without explaining its reasoning or revealing any cell values. When a question is vague or only adjacent to that decision, it answers "I don't have a preference on that, use your best judgment." Agents in PIVOSM therefore already face the incomplete replies that you expect from real users.

More importantly, our main conclusion does not depend on how the simulated user words its replies.

  • The largest difference between models arises before the user says anything. Ask rate ranges from 0.26 to 0.95 across configurations, while the share of asked questions that hit the withheld decision stays between 0.74 and 0.97 (Table 5).
  • Agents use a clarification well once they have it. Please refer to "Oracle Clarification" in the common official comment: when the simulator's reply is placed directly in the prompt, every tested configuration reaches its Explicit ceiling.

In summary, the remaining Interactive gap comes from not locating and asking about the ambiguity, a decision the agent makes before any user reply, whether the user is simulated or real.

Reviewer_7ygZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PIVOSM, a benchmark of 434 paired spreadsheet-manipulation tasks built from 185 seed workbooks of SpreadsheetBench-Verified. For each explicit task, an LLM injector creates an implicit version that leaves one decision-relevant degree of freedom unspecified. There are three injection types: missing key parameter, multi-target reference, and erroneous information. The injector also produces an alternative workbook that is consistent with a different reading. An LLM then writes a per-task Python grader with two rubric layers. Execution rubrics must pass on both workbooks. Intent rubrics must pass on the intended workbook and fail on the alternative. Five quality-control criteria, derived from a human audit and applied by two LLMs, filter out cosmetic ambiguities.

Critical strengths
  • The two-layer scoring on matched pairs is a sound, reusable idea. The approach in 2.5 provides a clear way to distinguish between an artifact that is actually broken and one that simply fails to match the user's intent. By requiring the alternative workbook to pass execution checks but fail intent checks, the authors make this distinction measurable. The results show a consistent pattern across all configurations: intent scores drop by roughly 0.35, while execution scores decrease by only about 0.05. Since the process is automated, it could also be applied to other tasks that involve generating artifacts. I see this as one of the strongest contributions of the paper.

  • The benchmark makes it possible to compare how models handle different types of user requests. By testing the same tasks with explicit, implicit, and interactive instructions, the authors can measure how much clarification actually helps. The results across different models and ambiguity types show that allowing models to ask questions improves performance, although there is still considerable room for improvement. If the benchmark's validity can be confirmed, I think it could be a useful tool for studying and improving how models handle ambiguous requests.

Critical weaknesses
  • The paper doesn't fully support its claim that each pair isolates a single decision. Since the injector can modify workbooks, it is unclear whether explicit and underspecified requests use the same input. Figure 7 also introduces another ambiguity by asking for rounding without specifying the increment. The SKU example seems to favor one interpretation over the other. These issues raise concerns about whether the benchmark truly isolates ambiguity.

  • The benchmark's evaluation methods need stronger validation. The two-witness checks only test graders against two workbooks, not other valid or incorrect solutions. Appendix B.2 describes LLM-based filtering but gives no human-audit results or error rates. The user simulator's judgments of clarification questions are also not independently verified. Since these methods drive the main results, the lack of validation makes the conclusions less convincing.

Questions for authors

These questions could affect my assessment:

  1. Do all three settings use the same input workbook? In Figure 7, how is the reporting increment defined, and why does the SKU wording still qualify as a single, meaningful ambiguity?

  2. How were the final tasks, graders, and simulator validated? Please provide sample sizes, error/agreement rates, and checks beyond the two reference workbooks. Which model and settings were used for the simulator?

  3. How do the authors distinguish missed ambiguity from reluctance to ask questions? Results with forced clarification or successful clarification hits would help. Also, which configurations are included in each aggregate, and why do the reported counts vary between 10, 12, and 13?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 7ygZ

We would like to thank you for the careful reading and for recognizing the double-layer scoring on matched pairs as one of the strongest contributions of the paper. Your questions helped us find a real error in Figure 7 and make the validation of PIVOSM much more complete. We address each point below.

[W1&Q1] Input Workbook in Three Settings

Implicit and Interactive use exactly the same workbook and the same prompt, and differ only in whether the ask channel is open. Explicit uses the original seed workbook and prompt from SpreadsheetBench-Verified. The two workbooks differ in 239 of 434 tasks, where the injector removed cells that would give away the withheld reading, such as pre-filled demo results.

Please refer to "Input Workbooks and Prompts in Explicit, Interactive, and Implicit Settings" in the common official comment for a full example. In short, the removed cells only leak the intended reading, which the Explicit prompt already states. Without removing them, the implicit prompt would not be ambiguous at all.

[W1&Q1] SKU Example Explained

Thank you for catching this. You are right that the instruction shown in Figure 7 contains two ambiguities. The cause is a manual editing error we made when assembling the figure: its "User" box shows the instruction of the sibling task 44628-MK instead of 44628-MR. Each seed yields up to three tasks, one per ambiguity type, and each task withholds exactly one decision:

\textbf{Task} & \textbf{Instruction change from the seed} & \textbf{Withheld decision} \\\\ \hline \text{Explicit (seed)} & \text{none: names the SKU, e.g. 'A' from 'A2-3442-2244'} & \text{none} \\\\ \texttt{44628-MK} & \text{adds "round to the nearest reporting increment"} & \text{the increment} \\\\ \texttt{44628-MR} & \text{removes the SKU example; "SKUs" becomes "products"} & \text{which column is the name} \\\\ \end{array}$$ - **The reporting increment belongs only to `44628-MK`.** There, the SKU wording is kept, so the increment is the only open decision, and the intent rubric checks that the totals are rounded to the intended increment of 5. - **The SKU wording does not appear in `44628-MR`.** The served prompt is "Group and sum the total sales of all products by using the first letter of each product's name, and aggregate the sales for all products whose name starts with the same letter. Display these results in D3:G3." With the SKU hint removed, "product's name" can refer to the SKU codes in column A or to the descriptive names in column C, and neither reading is favored. The two readings give C=35/D=35 and C=45/D=25. Only the text box is wrong. The trajectories in Figure 7 come from runs on the correct `44628-MR` prompt. In those runs, Claude-Opus-5 asks which column to group by, and Claude-Sonnet-5 silently groups by column C. We will correct Figure 7 to show the served `44628-MR` prompt. ## [W2&Q2] Human Validation Since human validation is a common concern, we put the annotation results in the common official comment: - **Final tasks:** Please refer to "Human validation of the final benchmark". On a balanced random sample of 100 tasks, 97 are judged genuinely ambiguous and 88 pass all four criteria. - **Graders on real agent outputs, beyond the two reference workbooks:** Please refer to "Grader Validity on Real Agent Outputs". On 100 real agent outputs, execution rubrics accept every reasonable reading in 92 cases, intent rubrics accept only the intended reading in 95, and the grader verdict is valid in 87. - **Simulator:** Please refer to "Details and Human Check on Simulator Model". Humans agree with the simulator's hit label on 92 of 100 clarification questions, with disagreements split evenly between the two directions. The simulator is DeepSeek-V4-Flash. It is stateless and answers each question in a single call, using the prompt given in the appendix (User-Simulator Implementation). An agent may ask up to 3 questions within its 5-turn budget.
AuthorsOpenReview ↗

(Cond.) Response to Reviewer 7ygZ

[Q3] Distinguish between Missed Ambiguity and Reluctance to Ask

Thank you for this question. We now report both analyses you suggested.

  • A successful clarification hit recovers the intent. On tasks where the Interactive trial hit the withheld decision, intent accuracy reaches the Explicit level: 0.857 vs. 0.887 for Claude-Opus-5, 0.840 vs. 0.756 for Qwen3.7-Max, and 0.788 vs. 0.798 for HY3.0.
  • A forced clarification also recovers it. We add an Oracle setting that gives every agent the clarification whether or not it asked. Please refer to "Oracle Clarification" in the common official comment: all three tested configurations reach their Explicit ceiling.

So once the ambiguity is resolved, agents use the answer well, and the remaining gap lies in getting to the question. Within that step, the asking decision matters more than question quality. Ask rate ranges from 0.26 to 0.95 across configurations, while 0.74 to 0.97 of asked questions hit the withheld decision (Table 5). Missing the ambiguity is the more common cause. In Implicit transcripts, 113 of 201 intent failures (56%) never mention the withheld decision at all.

[Q3] Configurations in Each Aggregate

We evaluate 9 models in 13 configurations, since four open-weight models are run in both thinking and non-thinking modes (marked †). The counts differ for two reasons.

  • 13 vs. 12. DeepSeek-V4-Pro† is reported under Implicit and Interactive only. Analyses that do not need the Explicit ceiling (Figure 5, Table 3 and Table 5) cover all 13 configurations. Analyses that use it (Table 2, Table 4 and the §5.1 averages, which rely on Δi\Delta_i and RiR_i) cover the other 12.
  • 11 and 10. These were omissions in our writing. Table 3 left out Qwen3.8-27B† and DeepSeek-V4-Pro†, and Table 4 together with the §5.1 averages left out Qwen3.8-27B† and HY3.0†. We have added the missing rows in the latest version of the paper.

The conclusions are unchanged after the fix. Averaged over all 12 configurations, Δi=0.338\Delta_i = 0.338, Δe=0.041\Delta_e = 0.041 and Ri=0.53R_i = 0.53, compared with 0.348, 0.050 and 0.58 over the previous 10. EI remains the hardest type, with the lowest Interactive joint accuracy in 11 of 13 configurations.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We would like to give sincere thanks to all reviewers who give insightful feedbacks. We address common concerns that reflects incomplete writings in our paper, and we will take valid feedbacks into our final version of paper.

Input Workbooks and Prompts in Explicit, Interactive, and Implicit Settings

In explicit settings, we directly use the workbook from the seed benchmark. In interactive and implicit settings, we use the workbook modified via our construction pipeline. In PIVOSM, 55.07% (239/434) of samples use a workbook different from the one in the explicit setting.

In section 2.3 AMBIGUITY INJECTION, we mentioned:

... The injector is given basic information about the seed task, along with a tool for modifying the input file so that the injected ambiguity is not entangled with the file’s existing content.

This might cause confusion: whether using different workbooks across settings confounds the intended prompt-only comparison.

The answer is NO. With detailed curated task inspection, we found that simply modifying the prompt, without giving the injector the necessary harness, CANNOT produce a task with genuine ambiguity. A great deal of task environment (i.e.: the workbook itself) from the original (verified) workbook contains few-shot examples that make the implicit prompt, along with the workbook. Here we demonstrate an example (10452-MK) where simply making the prompt itself ambiguous is not enough to create a genuine ambiguity, so this workbook-editing tool is needed.

USER INSTRUCTION: I need a method to perform a vertical lookup in Excel that only returns values which begin with a specific material prefix … E4 needs to lookup Column B for all materials starting with the prefix while need to skip all others.

The workbook modified before tool

RowCol BE (Result Required)
4FABRIC CAREPK01/P819760979
5BONUS GOLDPK01/P819761058
6PK01/P819760979PK01/P701001013
7PK01/P819761058
8PK01/P701001013
9BRITE MAXIMUM POWER
10PK01/P701001002
11PK01/P810760028

The workbook modified with tool removes all the few-shot example at Col E, so that the prefix mentioned in the instruction can no longer be inferred from the workbook itself (it could be any sub-prefix of the string PK01/P).

We have released a new appendix that shows our exact prompts for ambiguity injection and evaluation.

Human validation of the final benchmark

We drew a balanced random sample of 100 items from the final 434-task benchmark, spanning all three ambiguity types. Two annotators independently reviewed each item with full access to the implicit instruction, input workbook, annotated degree of freedom, and both intended and alternative output workbooks. Each item was judged on four criteria plus an overall pass.

\textbf{Criterion} & \textbf{Valid} \\\\ \hline \text{Genuinely ambiguous} & 97/100 \\\\ \text{Intended reading reasonable} & 95/100 \\\\ \text{Alternative reading reasonable} & 90/100 \\\\ \text{Decision-relevant} & 88/100 \\\\ \text{Overall pass} & \textbf{88/100} \\\\ \end{array}$$ **88 of 100 sampled items pass all four criteria at once**, and 97 of 100 are judged genuinely ambiguous. The four criteria degrade in a consistent order: annotators rarely doubt that an ambiguity exists, and are progressively stricter about whether both readings are reasonable and whether they lead to different workbook outputs. This tells us the pipeline is good at creating ambiguity but occasionally produces a pair of readings that converge on the same result. The remaining 12 items are borderline only on the two stricter criteria: the alternative reading was judged less natural given the workbook, or the two readings touched too few cells to be decision-relevant. Most of these still do not fail the more basic test of whether an ambiguity is present. We see this as the trade-off that comes with machine-curated ambiguity at scale, and will mention this in limitation and welcome more elegant methods, or automated brute force solutions, to improve this.
AuthorsOpenReview ↗

(Cond.) General Response to All Reviewers

Grader Validity on Real Agent Outputs

We audited the graders on 100 real agent outputs, evenly stratified across evaluation settings, model configurations, and ambiguity types. Human annotators inspected the workbook, full trajectory, and both rubric layers, and judged whether the rubrics evaluated that output correctly. Specifically, three criterions are commented by real intelligent workers via first understanding the modifed (ambiguous) instruction under the workbook, then check:

  • Execution rubrics accept every reasonable reading. Human annotators are given the annotated DoF (Degree of Freedom), and decide if such DoF seems reasonable. If reasonable, annoators decide if the execution rubrics covers expected workbook output set.
  • Intent rubrics accept only the intended reading. Human annotators are given the intended output workbook and alternative output workbook, then decide if the intent rubrics only satisfied under the intend output.
  • Grader results on agent's output is valid. Human annotators are given the workbook tested agent generate via spreadsheet manipulation, then decide if the generated workbook is logically a desired result (might be different from the workbook) based on the workbook and the instruction.

This annotation process takes huge time. The final results are as follows:

\text{Criterion} & \text{Valid} \\\\ \hline \text{Execution rubrics accept every reasonable reading} & 92/100 \\\\ \text{Intent rubrics accept only the intended reading} & 95/100 \\\\ \text{Grader result on agent's output is valid} & 87/100 \\\\ \end{array}$$ 1. Both the execution rubrics (92/100) and intent rubrics (95/100) achieve high validity rates, confirming that the dual-layer rubric design reliably covers all reasonable interpretations while remaining anchored to the intended reading. 2. The end-to-end grader verdict validity (87/100), though lower, still demonstrates strong overall reliability. 3. We do observe failure cases where machine-curated rubrics fail to capture valid real-world agent outputs. In 13 out of 100 cases, the execution rubrics impose overly narrow constraints that reject valid workbook results under alternative interpretations. One practical remedy is to invest additional filtering compute in the construction pipeline—for instance, prompting LLMs to generate diverse candidate solution paths after ambiguity injection and having the verifier check for non-cosmetic ambiguities. We will leave that as future work. ## Details and Human Check on Simulator Model The user simulator is **DeepSeek-V4-Flash**. To validate the simulator's hit labels, we sampled 100 clarification questions from Interactive trajectories, stratified by the simulator's label (50 hit / 50 miss) and balanced across all 13 configurations and the three ambiguity types. Annotators saw the implicit instruction, the input workbook, the withheld DoF, and both readings, but not the simulator's label, and judged whether each question targets the withheld DoF. $$\begin{array}{lcc} \textbf{Slice} & \textbf{Agreement} & \textbf{Remark} \\\\ \hline \text{All questions} & 92/100 & 95\\%\ \text{Wilson CI }[0.85,\\,0.96] \\\\ \text{Simulator labeled hit} & 46/50 & \\\\ \text{Simulator labeled miss} & 46/50 & \\\\ \quad\text{EI} & 30/33 & \\\\ \quad\text{MK} & 29/33 & \\\\ \quad\text{MR} & 33/34 & \\\\ \text{DeepSeek-V4-Flash} & 15/16 & \text{no self-preference} \\\\ \end{array}$$ Crucially, the disagreements are evenly balanced between false positives and false negatives, demonstrating that the simulator exhibits no systematic bias toward leniency or strictness. Annotation results also shows that agreement is flat across ambiguity types (29–33 of 33–34) and across configurations. Showing simulator exibits no-bias behavior for giving hints during the evlauation process. Note that the the denominators (typically 7–8 per configuration) reflect the balanced stratification of the 100 audited questions across all 13 models settings. $$\begin{array}{lcclc} \textbf{Configuration} & \textbf{Agr.} & & \textbf{Configuration} & \textbf{Agr.} \\\\ \hline \text{Claude-Opus-5} & 8/8 & & \text{HY3.0} & 8/8 \\\\ \text{Claude-Sonnet-5} & 8/8 & & \text{HY3.0}^{\dagger} & 8/8 \\\\ \text{GPT-5.5} & 6/7 & & \text{Kimi-K3} & 7/8 \\\\ \text{Qwen3.7-Max} & 6/8 & & \text{Qwen3.8-27B} & 8/8 \\\\ \text{DeepSeek-V4-Flash} & 8/8 & & \text{Qwen3.8-27B}^{\dagger} & 7/8 \\\\ \text{DeepSeek-V4-Flash}^{\dagger} & 7/8 & & \text{DeepSeek-V4-Pro}^{\dagger} & 7/8 \\\\ & & & \text{DeepSeek-V4-Pro} & 4/5 \\\\ \end{array}$$
AuthorsOpenReview ↗

(Cond.) General Response to All Reviewers

Oracle Clarification

To separate not asking from not using the answer, we add an Oracle setting. The agent receives the Implicit instruction and the same workbook as in Interactive, and the user's clarification of the withheld DoF is placed directly in the prompt, whether or not the agent would have asked. The clarification is the reply of the same DeepSeek-V4-Flash simulator, under the same prompt, to a question that targets exactly the withheld DoF. It is generated once per task and shared by all models. We ran three configurations on all 434 tasks.

\textbf{Configuration} & \textbf{Setting} & A_e & A_i & A_j \\\\ \hline \text{Claude-Opus-5} & \text{Interactive} & 0.866 & 0.802 & 0.786 \\\\ & \textbf{Oracle} & \textbf{0.876} & \textbf{0.869} & \textbf{0.846} \\\\ & \text{Explicit} & 0.878 & 0.876 & 0.848 \\\\ \hline \text{Qwen3.7-Max} & \text{Interactive} & 0.795 & 0.588 & 0.562 \\\\ & \textbf{Oracle} & \textbf{0.818} & \textbf{0.836} & \textbf{0.790} \\\\ & \text{Explicit} & 0.756 & 0.774 & 0.719 \\\\ \hline \text{HY3.0} & \text{Interactive} & 0.767 & 0.631 & 0.594 \\\\ & \textbf{Oracle} & \textbf{0.770} & \textbf{0.781} & \textbf{0.728} \\\\ & \text{Explicit} & 0.769 & 0.764 & 0.723 \\\\ \end{array}$$ **Once the clarification is given, every configuration reaches the Explicit ceiling.** In paired task-level bootstrap (95% CI), Oracle $-$ Explicit on $A_i$ is $-0.007$ $[-0.037, 0.021]$ for Claude-Opus-5, $+0.016$ $[-0.030, 0.060]$ for HY3.0, and $+0.062$ $[0.023, 0.101]$ for Qwen3.7-Max. Oracle improves over Interactive by $+0.067$, $+0.249$, and $+0.148$ in $A_i$, all with CIs excluding 0. The gain comes from tasks where the agent did not ask the right question. On tasks whose Interactive trial never hit the withheld DoF, $A_i$ rises from $0.476 / 0.446 / 0.275$ under Interactive to $0.810 / 0.835 / 0.733$ under Oracle (Claude-Opus-5 / Qwen3.7-Max / HY3.0), on par with Explicit on the same tasks. Agents use a clarification well once they have it. **The Interactive\$-\$Explicit gap is therefore a gap in locating and asking about the ambiguity**, which is exactly the capability PIVOSM is built to measure. ## Constructed Ambiguity and Ecological Validity Reverse-constructing single-DoF ambiguity is a deliberate design choice prioritizing internal validity. Crucially, single-DoF isolation is what allows us to fundamentally decouple execution competence from intent alignment: entangling multiple ambiguities would trigger a combinatorial explosion of plausible workbook states, making deterministic verification impossible. By bounding the solution space, PIVOSM uniquely achieves deterministic verification of non-intended yet valid interpretation, which is a capability largely absent in prior benchmarks. Meanwhile, controlled construction does not sacrifice task realism: our human validation confirms that 88/100 audited items pass all four criteria for genuine, decision-relevant ambiguity. The remaining 12 borderline cases reflect natural low-stakes output convergence rather than artificial artifacts. We are welcome for any multi-DoF future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.