acceptodds
Under review as a conference paper at ICLR 2027

StateEdit-Bench: From Local Edits to Stateful Presentation Workflows

Abstract

Presentation editing requires interpreting later requests in light of previously established objects, decisions, constraints, and edit boundaries. Yet existing evaluations primarily measure fulfillment of the current request or properties of the final output, leaving the preservation and selective revision of accepted state weakly observed. We introduce STATEEDIT-BENCH, a 126-unit benchmark suite with 218 goal instances on 58 distinct input decks: 70 single-turn local edits, 10 continuation or template extensions, and 46 three-goal stateful sessions (138 sequential goals). Each stateful goal advances the committed artifact and is evaluated with an edit contract covering current postconditions, retained constraints, artifact validity, and authorization-aware preservation. We also design SE-PPT-EDIT, a presentation-editing skill that combines explicit state tracking, scope-aware planning, and artifact verification. In paired no-skill/full-skill runs across three models, the equal-model descriptive macro changes from 69.8% to 75.8% (+6.0 pp), with the largest gain on continuation and template filling, followed by local editing. On stateful multi-turn sessions, however, the macro changes by only +2.2 pp, and the model-level changes are not statistically significant. These findings position StateEdit-Bench as a diagnostic benchmark: it exposes contract-state failures that current-request evaluation can miss, while showing that the evaluated procedural deployments do not yet reliably improve multi-turn state maintenance.

Primary area:
datasets and benchmarks
Keywords:
Agent Evaluation, Presentation Editing, Stateful Interaction, Benchmark, Artifact Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EZjbOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper introduces StateEdit-Bench, a benchmark for evaluating stateful editing by presentation-editing agents. It uses cumulative native-artifact chains and edit contracts to assess current-request fulfillment, retention of prior constraints, artifact validity, and compliance with authorized edit scope.

The benchmark contains 126 evaluation units and 218 goal instances: 70 single-turn local-edit tasks, 10 continuation or template-extension tasks, and 46 three-goal stateful sessions. The authors also compare paired no-skill and full-skill deployments across three models, reporting strict-success rates and multi-turn results.

Critical strengths

The paper identifies an important evaluation gap: success on the latest editing request does not ensure that an agent preserves previously accepted decisions or respects the authorized edit scope. Figure 1 illustrates this distinction with a concrete example in which an agent changes texture and whitespace but also removes part of a previously established botanical motif.

The evaluation rule in Equation (2) directly reflects the paper’s preservation objective. A transition succeeds only when the current request is fulfilled, active constraints remain true, the artifact is valid, and mutations stay within the authorized scope; session success requires every transition to succeed.

The authors explicitly state that StateEdit-Bench is designed for controlled diagnosis and that its authored tasks and source decks are not representative of presentation workflows. They also acknowledge that the failure audit is small and that visual calibration and validator coverage remain limited.

Critical weaknesses

The evidence that SE-PPT-EDIT improves stateful editing is limited. The multi-turn macro changes by only 2.2 percentage points, and the model-level changes are not statistically significant. The authors state that the tests establish neither improvement nor equivalence. In addition, the bundled comparison does not identify component-level causal effects, and the authors note that component ablations are needed.

The reliability of judge-dependent success decisions remains unclear. The paper states that VLM-based validity judgments enter the success criterion for extension tasks, and that validity, persona-adherence, and selective-update judgments enter the criterion for five aesthetic-memory sessions. The authors also acknowledge that visual calibration remains limited, leaving uncertainty about the reliability of these judge-dependent outcomes.

Questions for authors

How were the VLM-based validity, persona-adherence, and selective-update judgments calibrated, and what human–VLM agreement results support their use in strict-success scoring?

Can the authors provide component ablations to isolate the effects of state externalization, scope-constrained planning, and verify-and-commit execution?

3 comments

Thank you for recognizing the value of cumulative artifact chains and explicit edit contracts. The benchmark is our primary contribution. SE-PPT-Edit illustrates how its diagnostics can guide procedural improvement; the new ablation and human audit strengthen this case study.

  1. Component ablations We have obtained the requested ablation with Kimi-K3 and SE-PPT-Edit. The four conditions use the same tasks, execution budget, and evaluator across the full benchmark strata: | Condition | Local (n=70) | Extension (n=10) | Multi-turn (n=46) | | Full | 90.0% | 70.0% | 73.9% | | Without state externalization | 90.0% | 60.0% | 65.2% | | Without scope planning | 84.3% | 50.0% | 71.7% | | Without verify-and-commit | 88.6% | 50.0% | 67.4% | Full performs best or ties across all three strata. State removal leaves local editing unchanged but reduces multi-turn success from 73.9% to 65.2%. Verification removal reduces it to 67.4%, while scope removal reduces local success from 90.0% to 84.3% and extension success from 70.0% to 50.0%. The results therefore provide descriptive evidence that the components contribute differently across editing demands, rather than treating the skill as an unexplained bundle. The original comparison in Table 6 has limited sensitivity to small effects: with at most 46 sessions, its 80%-power MDE is 18.3–29.8 percentage points (Supplementary H.7). The new component experiment provides complementary descriptive evidence for diagnosing procedural improvements. We will describe the configuration in Section 4.1 and report these descriptive component comparisons in Section 5.2 and a new supplementary table. Visual scores provide a separate descriptive endpoint: | Condition | Local T4 (14) | Extension (10) | Multi-turn T4 (15) | |---|---:|---:|---:| | Full | 3.50 | 3.30 | 3.53 | | Without state | 3.57 | 3.20 | 3.20 | | Without scope | 3.21 | 3.00 | 3.47 | | Without verification | 3.36 | 3.00 | 3.33 | Full has the highest means on extensions and multi-turn T4, while state removal is slightly higher on local T4. These results suggest that the visual benefits vary with the editing setting. We will report the auxiliary scores in a separate supplementary table.

  2. Calibration of correctness gates and human support for benefits The original validity audit directly assesses one correctness gate. Supplementary H.5 and Table 7 report 34/41 human-VLM agreement (82.9%), including detection of 12/19 human-invalid outputs and seven false passes. Separate agreement estimates for persona-adherence and selective-update gates are not available. Section 3.3 and Appendix F distinguish these gates from auxiliary aesthetic preferences and deterministic checks. Structural checks and task-specific visual gates jointly determine success, while preference scores provide auxiliary evidence. The new blinded audit answers a complementary question: whether humans recognize the visual benefits identified by the VLM. Two independent annotators assess 60 Full-versus-ablation comparisons. VLM agreement with them is 42/60 (70.0%) and 47/60 (78.3%). Among the 40 comparisons where the VLM prefers Full, humans endorse 35/40 (87.5%) and 39/40 (97.5%). They themselves prefer Full in 48/60 (80.0%) and 51/60 (85.0%) comparisons. Thus, the visual benefits receive human support in this sample, beyond the automated scores. We will add this new audit to Section 5.4 and the supplementary calibration discussion, reporting agreement with each annotator separately. It provides complementary human support for the visual benefits of the full procedure. Together, the component ablation and human audit strengthen the diagnostic case study, while the original calibration quantifies validity-judge agreement. The benchmark contribution under Equation (2) is independently meaningful: it exposes violations of accepted state and authorized scope throughout the editing trajectory, providing a concrete target for evaluating and improving editing systems. We sincerely thank you for your time and thoughtful consideration of our work.

Reviewer_EZjbOpenReview ↗

Post-rebuttal assessment

The rebuttal clarifies the distinction between StateEdit-Bench and PPTC and adds component ablations for SE-PPT-EDIT, as well as human–VLM agreement results. These additions help clarify the evaluation protocol and the diagnostic role of the skill.

However, I remain concerned about the paper’s overall contribution. The benchmark’s main advance is to combine history-dependent editing tasks, cumulative artifact chains, and edit contracts in a presentation-editing evaluation. The rebuttal does not fully change my assessment that the conceptual and methodological novelty relative to prior work is limited. The added component ablation is conducted only with Kimi-K3 and provides descriptive evidence, while the original multi-model results on multi-turn editing remain limited by sample size and statistical power. The human–VLM agreement result addresses validity judgments; the authors also note that separate agreement estimates for persona-adherence and selective-update judgments are unavailable.

I appreciate the clarifications and additional analyses, but they do not sufficiently resolve my main concerns about novelty and the strength of the evidence. I therefore revise my overall rating to 2 (Weak Reject).

Thank you for the follow-up. Your original review recognized the evaluation gap and contract-based evaluation rule. We address your further concern about novelty, alongside ablation scope, statistical power, and visual judgment reliability.

  1. State requirements grounded in real editing workflows StateEdit-Bench draws on de-identified customer presentation data and screening more than 3,000 editing queries (Section 3.1). These workflows require agents to preserve accepted decisions, interpret later requests using those decisions, and selectively revise artifacts while maintaining other active constraints. We operationalize these requirements through history-dependent tasks, explicit edit contracts, and cumulative native-artifact chains. Sessions progress from establishment or clarification to context-conditioned reuse and selective revision. Each goal receives the agent's actual preceding output. The key distinction from PPTC is between completing successive requests and explicitly evaluating state preservation. Changing one text element's color and replacing another's content illustrates compound editing, represented by our TC category. Our stateful tasks additionally require later edits to depend on earlier decisions that are not restated, selectively revise them, and preserve other active constraints. PPTC's multi-turn setup does not provide the explicit evolving contracts and conjunctive transition-level checks used to evaluate these obligations. Supplementary C.3 illustrates these dependencies. Supplementary A.1 distinguishes new, retained, revised, and revoked constraints; Equation (2) jointly checks current fulfillment, retained constraints, artifact validity, and authorized scope at every transition. Section 3.2 and Table 3 identify this joint coverage gap. Our contribution makes history-dependent state preservation an executable evaluation target grounded in real needs.

  2. The evaluation reveals failures hidden by endpoint success For Opus with the skill, final cumulative success is 44/46 (95.7%), versus strict session success of 38/46 (82.6%) (Supplementary Table 4, Panel B; main Table 6). Six sessions pass the endpoint assessment but fail the requirement that every transition satisfy its contract: a 13.0-percentage-point difference. Our scoring framework separates correctness from visual quality: strict success assesses editing obligations, while auxiliary visual scores assess aesthetics (Section 3.3 and Appendix F). Visual appeal cannot compensate for violating an active state constraint. For statistical power, Supplementary H.7 reports an 80%-power minimum detectable effect of 18.3–29.8 percentage points at 46 sessions under its stated assumptions. This quantifies sensitivity to skill effects. The six-session discrepancy establishes a different result: the benchmark detects failures hidden by endpoint assessment.

  3. Kimi-K3 provides controlled component diagnosis The ablation fixes the model, tasks, budget, and evaluator. State removal leaves local success at 90.0% but reduces multi-turn success from 73.9% to 65.2%. Scope removal reduces local success to 84.3% and extension success from 70.0% to 50.0%. These descriptive results distinguish the procedural requirements of different editing strata. Controlled single-model component comparisons complement the original multi-model evaluation. External TSBench results also support practical utility: Kimi's artifact delivery, complete postconditions, and preservation each improve from 54/56 to 56/56 (Section 5.4; Supplementary Table 6).

  4. Human assessment supports the visual benefits Supplementary H.5 and Table 7 directly assess validity judgments, with human–VLM agreement of 34/41 (82.9%). The new blind audit assesses whether humans recognize VLM-identified visual benefits. Among 40 comparisons where the VLM prefers Full, the two annotators endorse 35/40 (87.5%) and 39/40 (97.5%). The component results show a consistent descriptive pattern: state removal reduces multi-turn success from 73.9% to 65.2% and mean visual quality on 15 multi-turn T4 sessions from 3.53 to 3.20. Together with the blind audit, these findings support visual benefits through automated evaluation and human assessment. The original audit measures validity-gate agreement; the new audit measures visual-preference agreement. Persona-adherence and selective-update require their own gate-specific agreement estimates. We will clarify these evaluation targets in Section 3.3 and Appendix F. Together, these results support a benchmark that makes historical obligations explicit, exposes failures hidden by endpoint assessment, and identifies targets for improvement. We will foreground these contributions and their evidence in the Introduction and evaluation discussion.

Reviewer_WAWGOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces StateEdit-Bench to evaluate current-request fulfillment, retained constraints, artifact validity, and authorized scope in presentation editing. The benchmark contains 126 evaluation units, including 46 stateful sessions, with 218 goals across 58 input decks. The paper also evaluates SE-PPT-Edit on three models. Aggregate success improves by 6.0 percentage points, but the multi-turn improvement is smaller and not statistically significant.

Critical strengths

● Each goal uses the agent’s actual preceding output. Checking every transition can expose intermediate errors that evaluation of the final file alone may miss. ● The authors report paired outcomes, confidence intervals, and statistical power limitations. They do not interpret the lack of significant multi-turn improvement as evidence of equivalence.

Critical weaknesses

● Section 3.1 discloses that tasks were available during development. Which evaluation tasks were used in skill development, and when were the configurations frozen? These boundaries determine how broadly the results can be interpreted; the observed gains do not by themselves establish generalization to unseen tasks. ● Section 4.1 acknowledges smaller evaluation samples for GPT and Kimi, but the main text does not explain the inclusion rules. What are the exact denominators for each model? Do the reported model rankings still hold on the shared task set?

● The 10 extension tasks account for approximately three-quarters of the aggregate gain in Table 4, and their success criteria include VLM judgments. Which outcomes that change between conditions depend on visual judgments? Were these cases independently checked by humans?

Questions for authors

see the weakness above

2 comments

Thank you for highlighting the evidence needed to interpret the results. Our central contribution is a benchmark that makes state-dependent editing measurable. Below we clarify its diagnostic use, evaluation populations, and the evidence supporting procedural improvements.

  1. Benchmark-guided development and external evidence SE-PPT-Edit is a case study of using StateEdit-Bench to diagnose failures and guide procedural improvement. Section 4.1 introduces this diagnostic role, and Section 3.1 discloses task availability during development. We used benchmark feedback to iterate on the skill, fixing each final configuration before its corresponding formal evaluation. We will make this purpose and sequence explicit in Section 4.1. The resulting procedure also shows structural benefits on another benchmark. Section 5.4 and Supplementary Table 6 report that, on 56 paired TSBench tasks, Kimi's artifact delivery, complete postconditions, and preservation each increase from 54/56 to 56/56. Thus, the case study illustrates how the benchmark can support procedural improvement, with external evidence complementing the development-set results.

  2. Evaluation denominators and diagnostic comparisons The paired sample sizes for local editing, extension, and multi-turn evaluation are respectively 70/10/46 for Opus, 69/10/41 for GPT, and 68/10/42 for Kimi. All analyzed tasks come from the same fixed benchmark suite. The discrepancy may stem from collecting results across separately stored run batches. We will report these denominators explicitly in Section 4.1 and Tables 4/6. For a conservative full-suite sensitivity check, we retain the reported successes and assign no success credit to results outside the reported paired sets, using denominators of 70/10/46 for every model. This checks the ranking under a common denominator convention rather than reconstructing task identities. With skill, GPT remains highest on local editing at 91.4%, followed by Kimi at 90.0% and Opus at 85.7%. On multi-turn sessions, Opus ranks highest at 82.6%, followed by GPT at 78.3% and Kimi at 73.9%. Kimi remains highest on extensions at 70.0%. These differentiated profiles illustrate the benchmark's diagnostic value: different editing demands reveal different strengths, which an overall score would obscure. We will add this sensitivity check to Section 5.1 and the supplement. The new Kimi-K3 ablation further uses the same complete 70/10/46 task set, budget, and evaluator for all four conditions, providing a direct common-task comparison of procedural components.

  3. Extension weighting and visual evidence To test whether the aggregate benefit persists when extensions receive less weight, we additionally pool the reported success counts across model–task pairs. This reduces the extension weight from one-third under Appendix F's equal-stratum macro to 30/366 (8.2%). Success still increases from 299/366 (81.7%) to 311/366 (85.0%). Here 366 is the number of included model–task pairs. Thus, the positive aggregate direction is not specific to giving extensions equal weight with the other strata. We will report this supplementary sensitivity analysis alongside the original metric. For validity calibration, Supplementary H.5 and Table 7 report 34/41 human–VLM agreement (82.9%), including detection of 12/19 human-invalid outputs and seven false passes. This is aggregate calibration, rather than case-level confirmation of the original Table 4 success flips. Main Table 5 separately reports postcondition attainment increasing from 89.3% to 92.4% and preservation from 95.6% to 98.2%. The new blinded audit adds human evidence for the ablation's visual benefits. Across 60 comparisons, VLM agreement with two independent annotators is 42/60 and 47/60. Among 40 VLM preferences for Full, humans endorse 35/40 (87.5%) and 39/40 (97.5%); humans prefer Full in 48/60 and 51/60 comparisons overall. This provides complementary human support for the benefits observed in the component study. We will add the audit to Section 5.4 and the supplementary calibration discussion, identifying its preference endpoint. Together, the task-dependent performance profiles, component study, and external evidence demonstrate the benchmark's practical diagnostic use: exposing editing weaknesses and providing concrete targets for procedural improvement. We sincerely appreciate your careful review and constructive feedback.

Reviewer_WAWGOpenReview ↗

maintain my score

Thanks a lot for the authors response and address some of my concerns, but I maintain my score

Reviewer_5gaTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces StateEdit-Bench to test presentation editing across multiple requests. It checks whether the model finishes the current request while keeping earlier decisions and leaving unrelated content unchanged. The authors also introduce SE-PPT-Edit, a skill tested with three models.

Critical strengths

The paper identifies an interesting gap in existing benchmarks: finishing the current request is not enough if the model changes something the user had already accepted. Testing this is useful.

Critical weaknesses

The comparison with other benchmarks feels superficial. The coverage table3, fig 2 shows StateEdit winning the chosen requirements, but does not clearly demonstrate what existing evals miss.

  1. PPTC was a release issue, rather than proof that PPTC lacks those capabilities. PPTC already evaluates cumulative sessions and tests the effect of removing history. A closer comparison with that setup would help explain what StateEdit adds.

  2. The contribution of the skill is unclear. It combines better instructions, state tracking, helpers, and verification, but we do not know which part helps. The multi-turn improvements are small and not statistically significant. This does not make the benchmark less useful, but it raises questions about presenting the skill as a separate contribution.

  3. It is unclear how the evaluation checks are implemented. The formula explains what needs to pass, but not how they check it. The supplement explains how scores are combined and gives some thresholds, but details such as matching objects between turns, comparing protected properties, and handling small differences are still unclear.

  4. The writing is repetitive. The same benchmark counts and evaluation principles appear many times.

Questions for authors

See weakness above.

1 comment

Thank you for recognizing the importance of preserving previously accepted decisions. Your questions help clarify the evaluation advance and diagnostic evidence.

  1. Why this benchmark, and what does it add beyond PPTC? The contribution is a benchmark for state-dependent editing needs grounded in actual workflows. Section 3.1 describes source decks drawn from de-identified customer usage data and screening of more than 3,000 customer editing queries. This motivates evaluating whether later edits preserve accepted decisions, rather than checking only the latest instruction. The benchmark turns that requirement into authored tasks, cumulative native-artifact chains, and explicit edit contracts. Supplementary C.3 illustrates the distinction: first identify a particular blue bar, then apply the same treatment to the next chart, and finally use a less saturated red there while keeping the rest. Correctness requires recovering the bound object, accepted treatment, and protected properties. Separately specified commands such as recoloring text A and rewriting text B need not exercise those dependencies. Appendix A.1 formalizes this distinction and represents retained, revised, and revoked constraints. PPTC supports cumulative sessions and history ablations. Our advance is the explicit combination of history-dependent task design and trajectory-wide contract checking. Its session evaluator checks accumulated requirements at the endpoint; its turn-based setting is separate. Equation (2) instead requires every transition to satisfy the current request, active retained constraints, validity, and authorized scope. The difference is measurable within one deployment: Full-skill Opus passes the final cumulative check on 44/46 sessions (95.7%), but every transition on only 38/46 (82.6%), a six-session gap (13.0 percentage points) (Supplementary Table 4, Panel B). Thus, endpoint acceptance misses intermediate failures that our criterion exposes. We will clarify this protocol distinction in Section 3.2 and Table 3, separately from Figure 2's release-verifiability audit. Together, the real-workflow motivation, state-dependent tasks, and executable contracts establish the benchmark's value beyond cumulative task completion.

  2. Which skill components help? SE-PPT-Edit is a case study of using the benchmark to diagnose failures and guide procedural improvement. To address component attribution, we added a Kimi-K3 ablation. All four conditions use the same tasks, execution budget, and evaluator: | Condition | Local (n=70) | Extension (n=10) | Multi-turn (n=46) | |---|---:|---:|---:| | Full | 90.0% | 70.0% | 73.9% | | Without state externalization | 90.0% | 60.0% | 65.2% | | Without scope planning | 84.3% | 50.0% | 71.7% | | Without verify-and-commit | 88.6% | 50.0% | 67.4% | Full performs best or ties in every stratum. Removing state externalization leaves local success unchanged but reduces multi-turn success by four sessions, or 8.7 percentage points. Removing verification reduces multi-turn success by three sessions. The original comparison has at most 46 sessions and an 80%-power MDE of 18.3–29.8 percentage points (Section 4.1; Supplementary H.7), limiting its sensitivity to small effects. The new ablation supplies complementary descriptive evidence about component roles, supporting the benchmark’s use for diagnosing sequential-editing procedures. We will describe the configuration in Section 4.1 and add a supplementary ablation table, discussed in Section 5.2.

  3. How are checks implemented? Equation (2) defines the acceptance rule; the released evaluator supplies concrete checks. Object lookup uses shape IDs with slide context and rejects ambiguous matches. Certain postconditions additionally use ID/type matching and a same-slide geometric fallback. Generic protected-object fingerprints compare text, bounding boxes, and image hashes exactly; task-specific postconditions provide additional checks. Image-equivalence checks have separate tolerances, including a 0.02 aspect-ratio tolerance and a maximum mean RGB-channel difference of 8. These are check-specific rules, not a universal visual-difference threshold. We will expand Section 3.3 and Appendix F with check-level descriptions of correspondence, protected properties, baselines, tolerances, and unsupported cases. This will make the implemented coverage explicit and the evaluation easier to reproduce.

  4. Presentation We will consolidate repeated composition counts around Table 1 and shorten the figure caption and conclusion. Together, these changes will clarify a benchmark that exposes overlooked failures and supports research on improving editing procedures. Thank you again for your thoughtful review and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.