acceptodds
Under review as a conference paper at ICLR 2027

PPTXEdit-Bench: A Diagnostic Benchmark for Language-Guided PowerPoint Editing

11 views

Abstract

Language-driven presentation editing requires a system to map user-facing expressions such as “the card on the left” to native PPTX objects, apply the requested change, and preserve unrelated content. Existing evaluations do not jointly represent editing intent, object roles, and outcome states, so an end-to-end failure cannot reveal whether a system misunderstood the request, selected the wrong objects, or executed the edit incorrectly. We introduce PPTXEdit-Bench, containing 22,100 verifiable editing instances over 4,420 source slides. Each instance pairs a natural-language instruction with a structured edit specification, role-aware object bindings, and state checks for target satisfaction and non-target preservation, covering referents from individual elements to composite entities and predicate-defined sets. We further develop SlideDiag, an inspectable intent–ground–execute pipeline. Gold interventions on intent and grounding measure the performance recovered by correcting upstream predictions and isolate execution errors that remain under fully specified edit plans. We evaluate general-purpose coding agents, one-shot structured editors, and SlideDiag under a unified protocol. PPTXEdit-Bench therefore supports both end-to-end comparison and diagnosis of where precise presentation editing fails.

Primary area:
datasets and benchmarks
Keywords:
PowerPoint editing, language-guided editing, diagnostic evaluation, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TH44OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PPTXEdit-Bench for language-guided PowerPoint editing and SlideDiag, a baseline that separates intent parsing, object grounding, and execution. The benchmark contains 22,100 tasks derived from AI-generated slides. Its main contribution is explicit annotation of object membership and editing roles, which supports diagnosis beyond final editing success.

Critical strengths
  1. The focus on complete object membership is useful. A card may look like one unit while consisting of several native objects, and modifying only some of them is a meaningful failure. The executable property checks and stage-wise interventions also make the evaluation more informative than a single success score.

  2. Paper writing is good and easy to follow.

Critical weaknesses
  1. The benchmark lacks validation with real user requests.

Users increasingly generate a draft with AI and then refine it. But in that workflow, the editing requests still come from real people.Here, both the slides and the requests are constructed. Instructions are instantiated from predefined templates and expanded through LLM paraphrases. This ensures consistency with the action library, but does not establish that the benchmark captures how users actually ask for changes.

  1. There is no multi-turn evaluation.

Each task starts from the original PPT. The five tasks per slide are independent. Presentation editing is often iterative: “move this card,” “now make it match the other one,” “keep the title where it was.” These requests require tracking the evolving document, resolving references to earlier edits, and preserving changes across turns. Internal model calls and code retries do not test these capabilities.

  1. Annotation reliability needs clearer reporting.

Appendix B.3 describes PPT-editing experience, shared training, a pilot round, and two review passes. These are useful safeguards. However, the annotators’ recruitment criteria and professional or educational backgrounds remain unclear. Were they predominantly CS researchers, presentation designers, or ordinary PowerPoint users?

  1. The strongest available model baselines are missing.

Why was Claude Fable 5 excluded? It was globally available again from July 1, before the September submission deadline. Including Claude Code as a harness does not substitute for evaluating an Anthropic frontier backbone. The authors should clarify their model-selection cutoff and any access or budget constraints.

For a new benchmark, testing leading available models is important to establish its remaining difficulty. Strong performance would not make the benchmark worthless. However, near-saturation would change its role from exposing unresolved capability gaps to serving primarily as a diagnostic or regression suite. The current experiments leave this question open, particularly for the subsets presented as challenging.

Questions for authors

Can the authors add multi-turn editing trajectories with cross-turn references and cumulative preservation checks?

What were the annotators’ backgrounds, and what was agreement before adjudication?

Now the frontier model is keeping evolving, can the authors test the benchmark on the most frontier models like GPT 6.1 or Opus4.8?

Reviewer_KFH3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PPTXEdit-Bench, a benchmark for language-guided exact editing of native PowerPoint objects. Its core contribution is an intermediate annotation layer (94,096 human-verified referent annotations) linking instructions to complete native-object memberships and editing roles across individual elements, ungrouped composite entities (e.g., information cards), and condition-defined selection sets, paired with Gold PPTX state checks. The authors also propose SlideDiag, a three-stage baseline separating intent parsing, role-aware object grounding, and code execution. Across nine backbones, coding agents, and one-shot editors, Gold interventions in SlideDiag show larger gains from replacing object bindings than from replacing intent, leading to the conclusion that object grounding is the primary bottleneck.

Critical strengths

The problem formulation captures a real difficulty in document editing. What looks like a single visual unit on a slide is often stored as separate background shapes, icons, and text boxes without native grouping. Likewise, reference objects do not change in the final file, so their role cannot be recovered from output diffs alone. Providing explicit membership and role annotations alongside executable state checks is therefore a clear step forward over outcome-only benchmarks such as PPTC, PPTArena, and PPT-Eval.

The dataset construction is carefully executed. Source decks come from five generation systems across eight scenarios, splits are partitioned at the presentation level to prevent same-deck leakage, and candidate memberships undergo two human review passes with DAG dependency checks between composite entities and selection sets. The evaluation is broad—covering nine backbones, four coding agents, and three one-shot editors—and tracks API calls and processing time alongside accuracy.

Critical weaknesses

1. Intent and grounding errors are not cleanly separated by the metrics

Stage-wise error attribution is the central claim of the paper, but the evaluation metrics blur the boundary between Stage 1 (intent parsing) and Stage 2 (grounding). Section 4.1 defines intent as I=(a,v,R)I=(a, v, R), where RR maps roles to referring expressions, yet Section 5.1 defines intent accuracy using only the action aa and arguments vv, ignoring RR. For reference or relational edits where v={}v=\{\} (e.g., copy_width), a parser that swaps the target and reference descriptions or drops an exclusion clause is still counted as intent-correct. When the grounding module follows those faulty descriptions, the error is attributed to grounding. Similarly, substituting Gold bindings in Figure 4 (+8.6 pp vs. +4.0 pp for Gold intent) overwrites and repairs upstream errors in RR, not just grounding failures. Moreover, because instructions are template-instantiated from the same 27-action catalog given to the parser (Figure 8), predicting (a,v)(a,v) is close to synthetic slot-filling. To support the diagnostic claim, the paper needs to evaluate RR directly and report binding accuracy conditioned on Gold intent.

2. Information parity between SlideDiag and the baselines is unclear

In Table 1, SlideDiag outperforms all baselines, sometimes by narrow margins (e.g., 84.23% vs. 83.62% on Qwen3.8-27B). However, Appendix C shows that SlideDiag's prompts include the exact benchmark RGB color dictionary (Figure 8) and the deterministic tie-breaking rules used to build the Gold PPTX files for ambiguous relational edits (Table 8 / Figure 10, such as using the minimum edge for left alignment and maximum width for equal-width operations). Its runtime also pre-computes page-space bounding boxes (bounds_pt) for composite units and coordinate transforms for groups, and uses three model calls versus one for the one-shot baselines. The paper does not clarify whether baselines receive these same conventions. Without them, a baseline can make a reasonable interpretation of an instruction and still fail the exact checker.

3. Confounded subgroup analysis and text-only grounding

In Figure 5(c), the lower success rate on composite entities (8.3–12.9 pp below individual elements) is attributed to the difficulty of recovering complete membership. However, Appendix B.5 restricts which actions apply to each referent type: composite entities are mostly used in harder geometric edits, whereas individual elements include easy text updates (93–98% SR). Without action-matched comparisons and binding accuracy broken down by referent type, this gap is confounded by action difficulty. Furthermore, SlideDiag performs grounding purely over extracted object metadata O(S) without seeing the rendered slide image. Since human annotators needed the rendered slide to identify ungrouped cards and text-free AutoShapes (the weakest category in Appendix E.3), it is unclear how much of the grounding bottleneck simply comes from not giving VLM backbones visual input.

Questions for authors
  1. What is SlideDiag's object-binding accuracy when evaluated under Gold intent, and how often does the parser produce errors in RR (role assignment, scope, or exclusions)? Does binding exact match check logical-unit partitions or only role-level ID unions?
  2. On an action-matched subset (e.g., translation and alignment only), what are the binding accuracies and final SRs across individual elements, composite entities, and selection sets?
  3. Did the one-shot editors and coding agents receive the benchmark color dictionary (Table 7) and relational tie-breaking policies (Table 8)? How does Direct Code perform when given the same conventions and object inventory O(S) as SlideDiag?
  4. Please provide the O(S) schema, the evaluator's numerical tolerances and group-ID normalization rules, and inter-annotator agreement / candidate modification rates from the two-pass human review.
Reviewer_v3L2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PPTXEdit-Bench, a diagnostic benchmark for language-guided editing of existing PowerPoint presentations. PPTXEdit-Bench provides fine-grained annotations of native-object membership and editing roles. The authors also introduce SlideDiag, a diagnostic baseline that decomposes PPT editing into intent parsing, role-aware object grounding, and code execution.

Critical strengths
  1. I think the fine-grained benchmark annotations is valuable. The benchmark goes beyond final editing by annotating complete native-object membership and editing roles.

  2. The decomposition of editing into intent parsing, object grounding, and execution enables more interpretable failure analysis and reasoning.

Critical weaknesses
  1. The main concern is the use of fully synthetic data. All source presentations are AI-generated, and editing instructions are constructed using predefined templates and LLM paraphrases, making the data not mimic the complex and diverse real-world presentations and user requests.

  2. The benchmark focuses on deterministic edits rather than more open-ended editing requirements, limiting the scope.

  3. Another major concern i have is the insufficient quantitative evidence of annotation quality. The paper does not report inter-annotator agreement and annotation correction rates.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.