- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PPTXEdit-Bench for language-guided PowerPoint editing and SlideDiag, a baseline that separates intent parsing, object grounding, and execution. The benchmark contains 22,100 tasks derived from AI-generated slides. Its main contribution is explicit annotation of object membership and editing roles, which supports diagnosis beyond final editing success.
Critical strengths
-
The focus on complete object membership is useful. A card may look like one unit while consisting of several native objects, and modifying only some of them is a meaningful failure. The executable property checks and stage-wise interventions also make the evaluation more informative than a single success score.
-
Paper writing is good and easy to follow.
Critical weaknesses
- The benchmark lacks validation with real user requests.
Users increasingly generate a draft with AI and then refine it. But in that workflow, the editing requests still come from real people.Here, both the slides and the requests are constructed. Instructions are instantiated from predefined templates and expanded through LLM paraphrases. This ensures consistency with the action library, but does not establish that the benchmark captures how users actually ask for changes.
- There is no multi-turn evaluation.
Each task starts from the original PPT. The five tasks per slide are independent. Presentation editing is often iterative: “move this card,” “now make it match the other one,” “keep the title where it was.” These requests require tracking the evolving document, resolving references to earlier edits, and preserving changes across turns. Internal model calls and code retries do not test these capabilities.
- Annotation reliability needs clearer reporting.
Appendix B.3 describes PPT-editing experience, shared training, a pilot round, and two review passes. These are useful safeguards. However, the annotators’ recruitment criteria and professional or educational backgrounds remain unclear. Were they predominantly CS researchers, presentation designers, or ordinary PowerPoint users?
- The strongest available model baselines are missing.
Why was Claude Fable 5 excluded? It was globally available again from July 1, before the September submission deadline. Including Claude Code as a harness does not substitute for evaluating an Anthropic frontier backbone. The authors should clarify their model-selection cutoff and any access or budget constraints.
For a new benchmark, testing leading available models is important to establish its remaining difficulty. Strong performance would not make the benchmark worthless. However, near-saturation would change its role from exposing unresolved capability gaps to serving primarily as a diagnostic or regression suite. The current experiments leave this question open, particularly for the subsets presented as challenging.
Questions for authors
Can the authors add multi-turn editing trajectories with cross-turn references and cumulative preservation checks?
What were the annotators’ backgrounds, and what was agreement before adjudication?
Now the frontier model is keeping evolving, can the authors test the benchmark on the most frontier models like GPT 6.1 or Opus4.8?