- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 2
Read review
Paper summary
This paper introduces StateEdit-Bench, a benchmark for evaluating stateful editing by presentation-editing agents. It uses cumulative native-artifact chains and edit contracts to assess current-request fulfillment, retention of prior constraints, artifact validity, and compliance with authorized edit scope.
The benchmark contains 126 evaluation units and 218 goal instances: 70 single-turn local-edit tasks, 10 continuation or template-extension tasks, and 46 three-goal stateful sessions. The authors also compare paired no-skill and full-skill deployments across three models, reporting strict-success rates and multi-turn results.
Critical strengths
The paper identifies an important evaluation gap: success on the latest editing request does not ensure that an agent preserves previously accepted decisions or respects the authorized edit scope. Figure 1 illustrates this distinction with a concrete example in which an agent changes texture and whitespace but also removes part of a previously established botanical motif.
The evaluation rule in Equation (2) directly reflects the paper’s preservation objective. A transition succeeds only when the current request is fulfilled, active constraints remain true, the artifact is valid, and mutations stay within the authorized scope; session success requires every transition to succeed.
The authors explicitly state that StateEdit-Bench is designed for controlled diagnosis and that its authored tasks and source decks are not representative of presentation workflows. They also acknowledge that the failure audit is small and that visual calibration and validator coverage remain limited.
Critical weaknesses
The evidence that SE-PPT-EDIT improves stateful editing is limited. The multi-turn macro changes by only 2.2 percentage points, and the model-level changes are not statistically significant. The authors state that the tests establish neither improvement nor equivalence. In addition, the bundled comparison does not identify component-level causal effects, and the authors note that component ablations are needed.
The reliability of judge-dependent success decisions remains unclear. The paper states that VLM-based validity judgments enter the success criterion for extension tasks, and that validity, persona-adherence, and selective-update judgments enter the criterion for five aesthetic-memory sessions. The authors also acknowledge that visual calibration remains limited, leaving uncertainty about the reliability of these judge-dependent outcomes.
Questions for authors
How were the VLM-based validity, persona-adherence, and selective-update judgments calibrated, and what human–VLM agreement results support their use in strict-success scoring?
Can the authors provide component ablations to isolate the effects of state externalization, scope-constrained planning, and verify-and-commit execution?
3 comments
Post-rebuttal assessment
The rebuttal clarifies the distinction between StateEdit-Bench and PPTC and adds component ablations for SE-PPT-EDIT, as well as human–VLM agreement results. These additions help clarify the evaluation protocol and the diagnostic role of the skill.
However, I remain concerned about the paper’s overall contribution. The benchmark’s main advance is to combine history-dependent editing tasks, cumulative artifact chains, and edit contracts in a presentation-editing evaluation. The rebuttal does not fully change my assessment that the conceptual and methodological novelty relative to prior work is limited. The added component ablation is conducted only with Kimi-K3 and provides descriptive evidence, while the original multi-model results on multi-turn editing remain limited by sample size and statistical power. The human–VLM agreement result addresses validity judgments; the authors also note that separate agreement estimates for persona-adherence and selective-update judgments are unavailable.
I appreciate the clarifications and additional analyses, but they do not sufficiently resolve my main concerns about novelty and the strength of the evidence. I therefore revise my overall rating to 2 (Weak Reject).
Thank you for the follow-up. Your original review recognized the evaluation gap and contract-based evaluation rule. We address your further concern about novelty, alongside ablation scope, statistical power, and visual judgment reliability.
-
State requirements grounded in real editing workflows StateEdit-Bench draws on de-identified customer presentation data and screening more than 3,000 editing queries (Section 3.1). These workflows require agents to preserve accepted decisions, interpret later requests using those decisions, and selectively revise artifacts while maintaining other active constraints. We operationalize these requirements through history-dependent tasks, explicit edit contracts, and cumulative native-artifact chains. Sessions progress from establishment or clarification to context-conditioned reuse and selective revision. Each goal receives the agent's actual preceding output. The key distinction from PPTC is between completing successive requests and explicitly evaluating state preservation. Changing one text element's color and replacing another's content illustrates compound editing, represented by our TC category. Our stateful tasks additionally require later edits to depend on earlier decisions that are not restated, selectively revise them, and preserve other active constraints. PPTC's multi-turn setup does not provide the explicit evolving contracts and conjunctive transition-level checks used to evaluate these obligations. Supplementary C.3 illustrates these dependencies. Supplementary A.1 distinguishes new, retained, revised, and revoked constraints; Equation (2) jointly checks current fulfillment, retained constraints, artifact validity, and authorized scope at every transition. Section 3.2 and Table 3 identify this joint coverage gap. Our contribution makes history-dependent state preservation an executable evaluation target grounded in real needs.
-
The evaluation reveals failures hidden by endpoint success For Opus with the skill, final cumulative success is 44/46 (95.7%), versus strict session success of 38/46 (82.6%) (Supplementary Table 4, Panel B; main Table 6). Six sessions pass the endpoint assessment but fail the requirement that every transition satisfy its contract: a 13.0-percentage-point difference. Our scoring framework separates correctness from visual quality: strict success assesses editing obligations, while auxiliary visual scores assess aesthetics (Section 3.3 and Appendix F). Visual appeal cannot compensate for violating an active state constraint. For statistical power, Supplementary H.7 reports an 80%-power minimum detectable effect of 18.3–29.8 percentage points at 46 sessions under its stated assumptions. This quantifies sensitivity to skill effects. The six-session discrepancy establishes a different result: the benchmark detects failures hidden by endpoint assessment.
-
Kimi-K3 provides controlled component diagnosis The ablation fixes the model, tasks, budget, and evaluator. State removal leaves local success at 90.0% but reduces multi-turn success from 73.9% to 65.2%. Scope removal reduces local success to 84.3% and extension success from 70.0% to 50.0%. These descriptive results distinguish the procedural requirements of different editing strata. Controlled single-model component comparisons complement the original multi-model evaluation. External TSBench results also support practical utility: Kimi's artifact delivery, complete postconditions, and preservation each improve from 54/56 to 56/56 (Section 5.4; Supplementary Table 6).
-
Human assessment supports the visual benefits Supplementary H.5 and Table 7 directly assess validity judgments, with human–VLM agreement of 34/41 (82.9%). The new blind audit assesses whether humans recognize VLM-identified visual benefits. Among 40 comparisons where the VLM prefers Full, the two annotators endorse 35/40 (87.5%) and 39/40 (97.5%). The component results show a consistent descriptive pattern: state removal reduces multi-turn success from 73.9% to 65.2% and mean visual quality on 15 multi-turn T4 sessions from 3.53 to 3.20. Together with the blind audit, these findings support visual benefits through automated evaluation and human assessment. The original audit measures validity-gate agreement; the new audit measures visual-preference agreement. Persona-adherence and selective-update require their own gate-specific agreement estimates. We will clarify these evaluation targets in Section 3.3 and Appendix F. Together, these results support a benchmark that makes historical obligations explicit, exposes failures hidden by endpoint assessment, and identifies targets for improvement. We will foreground these contributions and their evidence in the Introduction and evaluation discussion.
Thank you for recognizing the value of cumulative artifact chains and explicit edit contracts. The benchmark is our primary contribution. SE-PPT-Edit illustrates how its diagnostics can guide procedural improvement; the new ablation and human audit strengthen this case study.
Component ablations We have obtained the requested ablation with Kimi-K3 and SE-PPT-Edit. The four conditions use the same tasks, execution budget, and evaluator across the full benchmark strata: | Condition | Local (n=70) | Extension (n=10) | Multi-turn (n=46) | | Full | 90.0% | 70.0% | 73.9% | | Without state externalization | 90.0% | 60.0% | 65.2% | | Without scope planning | 84.3% | 50.0% | 71.7% | | Without verify-and-commit | 88.6% | 50.0% | 67.4% | Full performs best or ties across all three strata. State removal leaves local editing unchanged but reduces multi-turn success from 73.9% to 65.2%. Verification removal reduces it to 67.4%, while scope removal reduces local success from 90.0% to 84.3% and extension success from 70.0% to 50.0%. The results therefore provide descriptive evidence that the components contribute differently across editing demands, rather than treating the skill as an unexplained bundle. The original comparison in Table 6 has limited sensitivity to small effects: with at most 46 sessions, its 80%-power MDE is 18.3–29.8 percentage points (Supplementary H.7). The new component experiment provides complementary descriptive evidence for diagnosing procedural improvements. We will describe the configuration in Section 4.1 and report these descriptive component comparisons in Section 5.2 and a new supplementary table. Visual scores provide a separate descriptive endpoint: | Condition | Local T4 (14) | Extension (10) | Multi-turn T4 (15) | |---|---:|---:|---:| | Full | 3.50 | 3.30 | 3.53 | | Without state | 3.57 | 3.20 | 3.20 | | Without scope | 3.21 | 3.00 | 3.47 | | Without verification | 3.36 | 3.00 | 3.33 | Full has the highest means on extensions and multi-turn T4, while state removal is slightly higher on local T4. These results suggest that the visual benefits vary with the editing setting. We will report the auxiliary scores in a separate supplementary table.
Calibration of correctness gates and human support for benefits The original validity audit directly assesses one correctness gate. Supplementary H.5 and Table 7 report 34/41 human-VLM agreement (82.9%), including detection of 12/19 human-invalid outputs and seven false passes. Separate agreement estimates for persona-adherence and selective-update gates are not available. Section 3.3 and Appendix F distinguish these gates from auxiliary aesthetic preferences and deterministic checks. Structural checks and task-specific visual gates jointly determine success, while preference scores provide auxiliary evidence. The new blinded audit answers a complementary question: whether humans recognize the visual benefits identified by the VLM. Two independent annotators assess 60 Full-versus-ablation comparisons. VLM agreement with them is 42/60 (70.0%) and 47/60 (78.3%). Among the 40 comparisons where the VLM prefers Full, humans endorse 35/40 (87.5%) and 39/40 (97.5%). They themselves prefer Full in 48/60 (80.0%) and 51/60 (85.0%) comparisons. Thus, the visual benefits receive human support in this sample, beyond the automated scores. We will add this new audit to Section 5.4 and the supplementary calibration discussion, reporting agreement with each annotator separately. It provides complementary human support for the visual benefits of the full procedure. Together, the component ablation and human audit strengthen the diagnostic case study, while the original calibration quantifies validity-judge agreement. The benchmark contribution under Equation (2) is independently meaningful: it exposes violations of accepted state and authorized scope throughout the editing trajectory, providing a concrete target for evaluating and improving editing systems. We sincerely thank you for your time and thoughtful consideration of our work.