acceptodds
Under review as a conference paper at ICLR 2027

SlideGate: Benchmarking Presentation Generation with Dimension-Specific Verification

Abstract

Automated presentation generation has entered practical use, enabling users to obtain complete slide decks from natural-language requests and optional reference materials. Such requests often involve delivery requirements, including slide count, formatting, source grounding, and mandatory content, while also requiring slides to be free of severe visual defects such as occlusion and cropping. We introduce SlideGate, a benchmark comprising 40 presentation-generation tasks across seven application scenarios and five input forms, with 2,191 task-specific verification items and a protocol that separately reports instruction following, content quality, and visual presentation. Applying SlideGate to 240 outputs from six publicly available commercial slide-generation systems, we obtain binary content-item judgments that agree with two independent blinded annotators at rates of 80.7% and 85.6%, respectively, and system rankings based on mean per-slide visual scores that correlate with human preference rankings at Spearman rho = 0.829. Across these outputs, SlideGate detects 124 decks with blocking failures, including hard instruction violations, missing required content, or page-level cropping or occlusion. Using scoring schemes based on the published criteria of six representative evaluation frameworks, the initial grouping analysis finds that three schemes assign significantly lower scores to decks with detected blocking failures, but the score gaps are small. When thresholds are set to detect approximately 90% of failing decks in leave-one-task-out validation, these three schemes incur false-positive rates of 76.8% to 98.8%. These findings highlight the value of combining task-specific verification with explicit failure diagnosis to reveal delivery issues that holistic scores may obscure.

Primary area:
datasets and benchmarks
Keywords:
presentation generation, benchmark, instruction following, multimodal evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uYyYOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduce SlideGate, a benchmark for presentation generation that asks whether a generated slide satisfy delivery requirements. It contains 40 tasks across 7 scenarios and five input forms. For each task, an LLM pipeline constructs instructions and requiremsnts from requests and source materials, verifies their grounding, and freezes them across systems. Claude then judges hard constraint satisfaction, content coverage, depth, and faithfulness, together with page level visual defects and design. A slide is flagged when it violates a hard instruction, omits required content, or contains cropping or occlusion. On 240 decks from 6 commercial systems, the paper then reports 124 flagged decks and finds that 6 existing quality scores screen these outputs poorly. They then do the validation with programmatic page counts, two content annotators, and three designers.

Critical strengths
  1. The benchmark differ the presentation quality from fulfilling the actual request, A deck can look organized while still using the wrong lesson, omitting a required section, or make the explanation unreadable. SlideGate then ties these failures to task requirements and output evidence instead of relying on a simple overall quality score. Tasks with attachments, templates, and edits, together with real commercial outputs, bring the benchmark closer to actual use than reference based slide benchmarks.

  2. The shared specifications and separate failure categories make the evaluation diagnostic and usefull, Requirements retain request quotations or source locations and remain fixed across the systems on the same task. Failures are reported per dimensions with page location, and the three failure types mostly affect different decks. The same backbone experiment shows that the protocol responds to real workflow changes, and the categories reveal different improvement needs. This design remains a valuable contribution even if the claims about existing scores were narrowed.

Critical weaknesses

1 The validation not directly establish the accuracy of the final blocking failure labels, I noticed the Section 4.4 flags an entire deck when any required check fails. So these labels determin the reported 124 failure decks and the reference groups used to evaluate other scores. A single erroneous requirement or verdict can therefore affect both headline findings. Appendix H validates content depth and visual preferences, which do not directly establish whether required content is missing or a page is cropped or occluded. The specification review is useful, but Table 12 also contains 12 items both annotators considered unnecessary and 15 disputed items. Their effect on the final flags is not reported. Recomputing the findings under these existing annotations, together with a small human audit of flagged and unflagged cases, would connect the validation to the actual claims.

2 The comparison does not isolate the benefit of SlideGate’s verification design from differences in available information and judges Section 5.2 states that most comparison scores dose not receive the request or source materials. A score without those inputs cannot reliably determine whether a requested section or source fact is missing. Appendix E also uses GPT-5.6-sol for prompt based comparison schemes, whereas SlideGate uses Claude Sonnet 4.6. Though the author acknowledges these differences and the experiment tests whether existing scores can be repurposed for failure screening. It does not establish how much improvement comes from SlideGate’s specification construction rather than simply supplying task information. Moreover, several comparison schemes already assess visual defects, so their performance on the combined failure label does not establish that they miss cropping or occlusion.

Questions for authors

Here are some of my questions in details for this paper, the author can answer some of them or not.

  1. How do the content failure count, the 124 deck union, and screening results change after excluding the 12 consensus unnecessary items and varying the treatment of the 15 disputed items in Table 12? Does the results still stable?
  2. How does the author calculate the agreement between the two annotators, for example, Cohen’s kappa. In Table 12 the kappa is .65, but if calculate the standard binary Cohen’s kappa pooled over those 270 items, it can only be approximately .585 to .590, not .65?
  3. With the judge, requests, and source materials held fixed, how does a single candidate specification compare with the full procedure of candidate consolidation, critique, and rule validation? I am curious if we do distinguish missed requirements, unsupported additions, and changes in final failure labels. Though Section 5.4 reports consistency with generated candidates, this does not independently measure completeness.
  4. Table 4 describes ordered rendered pages, while Table 5 identifies a web slides interface with PPTX export for Manus. How was each system’s output saved and converted into page images, including the renderer and fonts? Were flagged cases checked against the delivered artifact and the system’s own display? I think this matters because export or rendering can introduce new cropping and occlusion?
  5. Who wrote and selected the 40 requests? Appendix A distinguishes the formal request from an earlier short message. Were requests finalized before inspecting system outputs, and were generations retained after one attempt or selected after retries or manual edits?
4 comments
AuthorsOpenReview ↗

Response to Reviewer uYyY (1/2): Blocking-Label Audit, Specification Annotations, and Controlled Comparisons

We are grateful for the reviewer's detailed reading and concrete suggestions, and for the note that the design remains valuable even if the claims about existing scores were narrowed. We followed the suggested analyses.

W1 / Q1 / Q2. Accuracy of the final blocking labels and the effect of specification annotations.

Following your suggestion, two annotators reviewed 30 decks (15 flagged, 15 not) blind to system identity and to SlideGate's verdicts, checking each against the request, the original attachments and every page. Both confirmed a blocking failure in all 15 flagged decks; inter-annotator κ per failure type is .66–.86. Among the 15 unflagged decks, annotators A and B identified a blocking failure in six and five, three of them in common (Section 5.4, Appendix J). If 3/15 of the 82 screening negatives contained blocking failures and all of them were among SlideAudit's false alarms, its FPR at the same operating point would still be about 71%.

We also recomputed the findings under the specification annotations (Table 13):

Content itemsContent failuresBlocked decksSlideAudit AUROCSlideAudit FPR
All 270 items71124.67276.8%
Without the 12 items both annotators judged unnecessary64–69121–123.676–.68677.1–77.6%
Also without the 14 disputed items51–67117–122.674–.68377.1–77.9%

Ranges treat decks whose status cannot be determined after removal as failing or as excluded. Thank you also for checking the agreement statistic: the revised Table 12 is computed directly from the released item-level annotation file (244 items required by both annotators, 12 by neither, 14 disputed; Cohen's κ = .61 with 94.8% raw agreement and Gwet's AC1 = .94, as about 90% of items fall in one class), and the analysis above uses exactly these labels. A blocking flag states a failed necessary condition for delivery; partially covered content, which SlideGate scores as partial by design, is reported but not gated.

W2. Information, judge, and visual defects.

We agree these factors should be separated. Given the request, the full attachment text and every page, but no frozen specification, the same Sonnet 4.6 judge reaches AUROC .765, yet its own flags recover only 69 of 124 blocked decks and 17 of 71 content failures, while both annotators confirmed a blocking failure in each of the 15 audited flagged decks. Re-running SlideAudit with Sonnet 4.6 raises AUROC from .672 to .725, with FPR still 79.3% at 89.5% recall (Section 5.2, Appendix K).

We also agree that several schemes assess overlap and bounds, and they do respond to defective pages: within the same deck, their page scores rank defect pages below the others with AUROC .70–.91, and SlideAudit's categories fire on 92.0% of the 88 flagged pages. The signal is diluted at deck level: one hit lowers a 12-page deck's SlideAudit score by about .004; AutoPresent's worst page instead of its mean raises visual-failure AUROC from .66 to .81; and alarming on any SlideAudit hit reaches 95% recall but flags 54% of decks without a detected failure. We therefore see detecting a defect on a page and making a delivery decision as related but different capabilities.

AuthorsOpenReview ↗

Response to Reviewer uYyY (2/2): Specification Construction, Rendering, and Request Provenance

Q3. A single candidate specification versus the full procedure.

With judge, prompt, requests and sources fixed, we scored 72 decks (12 tasks × 6 systems) with each of three single-candidate instruction specifications and with the final one (Appendix C). On the deck-level 1–5 instruction rating that the judge returns with its item verdicts, a single candidate agrees exactly with the final specification on 71–74% of decks (Spearman .59–.68), and the system ranking changes (Kendall .60–.83): all three single candidates rank Doubao first, while the final specification ranks AnyGen first. Deck-level instruction-failure flags differ on 6–7 of the 72 decks; the final specification flags 15 decks and each single candidate 11–13, so most changes are failures a single candidate misses. For example, on the task "organize the contents and images of the two files", all single candidates give a score of 5 to two decks that use at most one of the attachments' 27 embedded images; only the consolidated specification retains the image requirement as a hard item, and both decks receive 2. In an audit of earlier single-pass labels, an internal page-count estimate was treated as a hard constraint on 19 of 40 tasks, inferred requirements were scored on 17, and optional wording was upgraded to mandatory on 9; consolidation against the request text, critique and rule validation are designed to remove these, and freezing ensures all systems face the same items. We have also clarified in Section 4.1 that the three-candidate procedure applies to instruction specifications, while content specifications use source-constrained extraction, code validation and a critic that may only remove items.

Q4. Saving and rendering.

Thank you for raising this. We evaluate the file each system delivers: the exported PPTX for all six systems, including Manus's PPTX export from its web-slide interface. All files go through one pipeline without per-system tuning: LibreOffice 6.4.7 headless to PDF, then pdftoppm at 150 dpi, yielding 3,026 pages (Appendix J). We view defects introduced by a product's own export as part of its delivery quality. Fonts are chosen by the generating system; fonts not installed on the rendering machine are replaced by LibreOffice's fallback fonts, as they would be for a user opening the file in the same environment. Flags are determined on the delivered file rather than on in-app previews, because the file is what users receive. Since text overflow is the defect most sensitive to font substitution, we also keep only defects classified as occlusion or elements beyond the page: 120 of 124 decks remain blocked. In the Codex physics deck, the same pipeline renders all of roughly 100 formula objects in the original lecture, whereas the delivered file contains none of them, so the information is lost before rendering.

Q5. Requests, selection, and generation protocol.

We have added these details to Appendix A. We selected the 40 tasks by stratifying over scenario and input form. The 28 tasks with attachments come from real use: we invited practitioners from a range of industries to use an internal presentation-generation system for their own work, and de-identified the content while keeping input forms and scenarios. In that system a user types a message and also selects the scenario, audience, length and role; the formal request merges the message, the selected options and the attachment names into one instruction, and is identical to the message when the message already states them. The 12 text-only tasks cover brief and detailed requests without materials; their topics are drawn from a library of roles and presentation topics with sampled style parameters, and during test-set construction each short topic message was expanded into a detailed request stating audience, structure, style and length, so that the set spans a controlled range of request complexity; their input-form labels follow the original test-set design (six brief, six detailed). The released task files contain both the opening message and the formal request. All formal requests were fixed before any system output was collected. Runs that produced no deliverable were resubmitted; the first delivered deck was kept, without manual edits and without selection among multiple results.


Sorry due to CVPR ddl I only have time to review the author huge effort on rebuttal and reply these. The author's comments solve part of my concerns. Though I think there is still some way awary from strong accept (some how like 2.8/4), due to ICLR's 4 score rating scale this year, I still decide to give 3 score and with good luck to author. And please add all these materials in camera ready if accepted!

Thank you very much for taking the time to go through our response despite the CVPR deadline, and for raising your score. Your suggestions, especially the blinded audit of the final blocking decisions and the recomputation under the specification annotations, have made the paper clearer and more convincing. All of these materials are already included in the revised PDF, and we will keep them in the camera-ready version if the paper is accepted. Best of luck with your CVPR submission!

Reviewer_uV2jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a presentation generation benchmark, named SlideGate, which contains 40 presentation-generation tasks and can effectively evaluate LLMs on instruction following, content quality and visual presentation. Experimental results show that the benchmark can diagnose failures of different types.

Critical strengths

Presentation generation is a widely-used task skill, therefore its evaluation is helpful. Compared with previous quality score-based evaluation, the benchmark proposed in this paper also provides task-specific diagnosis ability in different aspects and different generation settings(e.g., direct generation vs agentic workflow), which is helpful for real-world usage.

Critical weaknesses
  1. The paper focuses on a relatively narrow task, resulting a focused contribution.
  2. The evaluation metric system needs to be compared against many existing presentation generation methods. For example, the completeness of the proposed metric system should be justified: is the current evaluation system sufficient to cover the different aspects of presentation generation?
  3. The authors need to further verify the generalizability of the benchmarks and the conclusions. Currently the authors verify their benchmarks on 6 comemercial systems only using LM-based evaluator, the ground truth labels may be noisy. Furthermore, it is hard to attribute different failure to LM's abilities.
  4. The conclusions and findings can be more insightful. I recommend the authors to further conduct intervention analysis on different failure types.
Questions for authors

See weaknesses in above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer uV2j: Task Scope, Evaluation Coverage, Label Reliability, and Interventions

We thank the reviewer for the helpful comments and for noting that task-specific diagnosis across generation settings "is helpful for real-world usage."

W1. Scope of the task.

We appreciate this point. While a presentation is a single artifact, producing one involves several capabilities at once: understanding the user's constraints, reading heterogeneous attachments, organizing content logic, laying out text and visuals with aesthetic quality, and delivering a file in which every page is readable. In our set, one request asks for a lesson-two deck that follows a separate production guideline; another asks to restyle a deck "without changing the main content or background"; a third asks to keep every formula of a 17-page physics lecture. Covering seven scenarios and five input forms, we hope the benchmark can serve as a demanding testbed for these combined abilities.

W2. Comparison with more methods; completeness of the metric system.

Beyond 6 systems, a same-backbone workflow comparison and 6 evaluation frameworks, we now add 2 systems and controls for information and judge (Section 5.2, Appendices K–L). On completeness, the evaluation covers instruction following, content coverage, depth and source faithfulness, four visual-defect checks, six design criteria and three aesthetic criteria with cross-page consistency (Table 2, Appendix B). These are organized by what a deck can be checked against: the request, the supplied materials and its rendered pages. SlideGate defines one delivery condition for each (a failed hard instruction, a missing required-content item, and page-level cropping or occlusion), applies them as non-compensatory gates, and keeps the remaining judgments as separate quality scores. Since 88 of the 124 blocked decks fail exactly one condition, dropping any condition would pass decks that the others reject.

W3. LM-based evaluation and label noise.

SlideGate is automated by design, so that it can be re-run on new systems without new annotation, as we did for Codex and NotebookLM. Its judgments are constrained to make them checkable: every item is fixed in advance with its request span or source location, so the judge answers a narrow question such as "does section 4 exist"; page counts are decided by code; and every verdict returns evidence pages. Human studies then validate the design: two blinded annotators confirm a blocking failure in all 15 audited flagged decks (Appendix J), content-item judgments agree with annotators at 80.7% and 85.6%, and controlled defect injection turns all 10 targeted instruction verdicts from pass to fail, worsens 9 of 10 targeted content verdicts and lowers 8 of 10 targeted page scores (Appendix H). Regarding attribution, the commercial results characterize end-to-end outputs rather than isolated backbone abilities, which the same-backbone comparison addresses (W4).

W4. Intervention analysis by failure type.

Thank you for this suggestion. The direct-versus-workflow comparison is an intervention on the generator: with the backbone fixed, instruction-failure decks fall from 15 to 1, while cropping/occlusion decks go from 11 to 12 (Section 5.3). We add interventions on the evaluator: using AutoPresent's worst page instead of its mean raises visual-failure AUROC from .66 to .81, and giving a judge full task information without a frozen specification flags 17 of the 71 SlideGate content failures (Appendix K). Different failure types respond to different changes, which we hope helps system builders decide where to improve.

AuthorsOpenReview ↗

Friendly Follow-up before the Discussion Closes

Dear Reviewer uV2j,

Thank you again for your review. As the discussion period closes soon, we would like to kindly check whether our responses above address your concerns, and to highlight the results most relevant to them.

On label reliability, in a blinded audit both annotators confirm a blocking failure in all 15 decks that SlideGate flags (Appendix J).

On interventions, with the backbone fixed, a multi-stage workflow reduces instruction-failure decks from 15 to 1 while cropping/occlusion decks go from 11 to 12 (Section 5.3): different failure types require different fixes, which a single quality score cannot reveal.

On scope, 124 of the 240 commercial outputs carry at least one blocking failure, while none of the six existing schemes screens individual decks at high recall without flagging most of the others.

We are happy to clarify anything further.

Reviewer_6R1GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SlideGate, a benchmark for evaluating AI-generated presentations across instruction following, content quality, and visual presentation. The benchmark contains 40 tasks covering seven application scenarios and five input types, with 2191 task-specific verification items. The proposed evaluation framework combines automatically constructed verification criteria, rule-based checks, source-grounded content assessment, and visual inspection to identify problems such as missing required content, instruction violations, and layout defects. The authors evaluate 240 presentations generated by six commercial systems and find that existing overall quality scores often fail to reliably identify serious presentation errors. They further validate the evaluation framework through human annotations, designer preferences, and comparisons with existing evaluation methods.

Critical strengths

Evaluating whether generated slides actually meet users' requirements is challenging, as a presentation can look visually appealing while still missing key information or violating explicit instructions. The benchmark covers a range of real-world scenarios and input types, making it potentially useful for evaluating presentation-generation systems in practical settings.

Critical weaknesses
  • The main observation that overall quality scores may hide specific errors is reasonable but relatively straightforward. The proposed framework also combines several established evaluation techniques, including instruction verification, content coverage checks, and visual defect detection. Similar aspects have already been studied in existing benchmarks such as PresentBench, PPTBench, and SlideAudit. I think the paper should more clearly demonstrate what new evaluation capabilities SlideGate provides beyond these existing approaches.

  • Although the paper evaluates six commercial systems, it does not include several widely used alternatives, such as ChatGPT, Claude, NotebookLM, or Microsoft Copilot for PowerPoint. These systems support different presentation-generation workflows, particularly document-based generation and slide editing, which are relevant to the benchmark's evaluation dimensions. Including a broader and more diverse range of systems would make the benchmark more representative and provide stronger evidence that the findings generalize beyond the six selected systems.

Questions for authors

Please see weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6R1G: New Evaluation Capabilities and System Coverage

We thank the reviewer for the thoughtful comments and for recognizing that a presentation "can look visually appealing while still missing key information or violating explicit instructions."

W1. New evaluation capabilities beyond PresentBench, PPTBench and SlideAudit.

We would like to clarify that our contribution is not the observation that averages can hide errors. The failures that decide delivery are relations between a deck and its request, its source material, or its rendered page; an evaluator that reads only the slides cannot observe the first two, however it aggregates (see the walk-through in the General Response and Appendix I). SlideGate adds three capabilities that, to our knowledge, the cited works do not provide together:

  • Specifications built automatically from the user's own request and attachments, frozen before any output is seen. PresentBench writes a controlled instruction for each instance and designs its checklist manually. SlideGate derives items from the requests actually given to the systems, which range from one sentence to multi-section briefs, templates and edits of existing decks and mostly originate from practitioners' own work; each item keeps its request quotation or source location.
  • Non-compensatory decisions. PresentBench aggregates item verdicts into an overall score. In SlideGate, an unmet hard requirement, a missing required item, or a cropped or occluded page yields a blocking flag with item IDs and page numbers that quality scores cannot offset.
  • A test of whether existing scores can screen delivery. With thresholds set for at least 90% recall, the six schemes flag 73–100% of held-out decks without a detected failure.

Our controls separate these capabilities from aggregation and from extra input. Deck-level aggregation dilutes visual signals: SlideAudit's cut-off/occlusion categories fire on 92.0% of the 88 defect pages, yet one hit lowers a 12-page deck's score by about .004. Extra input helps but does not replace item-wise checking: the same judge given the request, the full attachment text and all pages, but no frozen specification, flags 17 of the 71 SlideGate content failures (Section 5.2). Section 6 now also shows an investor deck that ranks first among six under four existing scores while omitting the core figures of its attached data.

W2. Coverage of systems.

Thank you for this suggestion. We would first like to explain how the systems were chosen. The paper's claims concern the evaluation instrument rather than a leaderboard: blocking failures fall into three largely separate types, and existing quality scores cannot screen them. We therefore selected the initial systems to represent several deck-generation workflows. They were the publicly available slide generators we could access during collection and span dedicated slide tools (AnyGen, Kimi), a chat assistant's slide entry (Doubao) and multi-step agents (Skywork, MiniMax, Manus) (Table 5). The tasks already cover the workflows you mention, with 17 document-based generation, 6 editing and 5 template tasks. Collection through these commercial interfaces involves manual submission, export and subscription access, which limited the size of the initial collection.

To test whether the findings extend beyond these six, we added Codex (code-generated PPTX) and NotebookLM (source-grounded generation, partly delivered as full-page images) on 12 tasks spanning one-sentence, detailed, document-based, template and editing requests (Appendix L). The frozen specifications apply unchanged: 3/12 Codex and 1/12 NotebookLM decks are blocked. One Codex content failure involves an item whose source name had been replaced by a placeholder in the de-identified attachment; excluding that item leaves two blocked Codex decks among the same 12 outputs. The failures again fall into the paper's categories while taking new forms: a Codex physics deck matches the original lecture page for page, yet all of its roughly 100 formula objects are gone, which only a source-grounded check reveals.

In summary, the frozen specifications apply without modification to workflows outside the original six and identify failures within the same defined categories. As the added outputs were collected later, we report them as a transfer test rather than a ranking.

AuthorsOpenReview ↗

Friendly Follow-up before the Discussion Closes

Dear Reviewer 6R1G,

Thank you again for your review. As the discussion period closes soon, we would like to kindly check whether our responses above address your concerns, and to highlight the two results most relevant to novelty.

First, one investor deck ranks first among six outputs under SlideAudit, PPTBench, AutoPresent and PPTEval, yet omits the core figures of its attached spreadsheet (Section 6). Such failures exist only relative to the request or the source material, and 124 of the 240 commercial outputs carry at least one.

Second, the gain does not come from the judge or from more information. The same judge, given the request, the full attachment text and all pages but no frozen specification, recovers only 69 of the 124 blocked decks and 17 of the 71 content failures (Appendix K).

Appendix L reports Codex and NotebookLM evaluated with the frozen specifications unchanged. We are happy to clarify anything further.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2): What SlideGate Measures, Illustrated with One Task

We sincerely thank all three reviewers for their careful reading and constructive suggestions. We are glad that the reviewers found the evaluation problem practically important, highlighted the difference between a deck that looks organized and one that fulfills the request, and saw value in task-specific diagnosis. Before the individual replies, we would like to briefly restate what SlideGate measures, using one task from the benchmark. All new results are included in the revised PDF.

The failures that decide whether a deck can be delivered are often not what a quality score measures. Some are invisible on the slides themselves: whether a requested section exists, whether the deck uses the lesson the user named, or whether an investor deck carries the figures in the attached spreadsheet. They exist only relative to the request or the source material. Others are visible on a page, such as a clipped formula or an occluded chart, but are averaged away across the deck. For example, one investor deck in our set ranks first among the six outputs for its task under SlideAudit, PPTBench, AutoPresent and PPTEval, yet it omits the full-year revenue figures and channel shares given in the attached spreadsheet (Section 6).

SlideGate on one request (Appendix I, Table 14). A hospital department head asked for an annual report in four named sections. Section 3 should cover, among others, a lifecycle-management system, on-site repair stations, and quality-control and incentive rules; section 4 lists near-term arrangements such as an inventory check, advance stock of oxygen and consumables, and two-tier on-call shifts.

  1. Specification. SlideGate turns the request into 21 hard instruction items, each tied to the request span it comes from (e.g., "Section 4: add near-term arrangements"), and derives 20 required content items from the request and the attached work plan, each tied to its request quotation or source location. Rule checks reject items without a supporting span, and the specification is frozen before any output is seen, so every system answers the same questions.
  2. Verification. Each item is judged pass, partial or fail with supporting pages; page counts are computed by code; every rendered page is checked for cropping and occlusion.
  3. Decision. One system returned 13 well-formatted slides covering three sections. SlideGate records 11 failed instruction items and 5 missing content items, and reports a blocking failure with those item IDs; quality scores are reported separately and cannot offset it. In our blinded audit, both annotators independently mark every item of section 4 as failed.

Nothing on these slides looks wrong; the failure becomes visible only against the request, and it is located precisely enough to fix. This is the capability we aim to add: request- and source-grounded acceptance items, frozen across systems, with non-compensatory decisions located at the item and page level. Across 240 commercial outputs, 124 decks carry at least one such failure.

AuthorsOpenReview ↗

General Response (2/2): Summary of Evidence and Revisions

Summary of evidence and revisions (all in the revised PDF):

QuestionEvidence
Are the final blocking flags correct?Blinded audit of 30 decks (15 flagged, 15 not; system identity and SlideGate verdicts hidden): both annotators confirm a blocking failure in all 15 flagged decks; inter-annotator κ .66–.86 per failure type (Section 5.4, Appendix J).
Do borderline specification items drive the result?Removing the 12 items both annotators judged unnecessary: 121–123 blocked decks (from 124); also removing the 14 disputed items: 117–122. SlideAudit's FPR stays at 77.1–77.9% (Table 13).
How much does specification construction matter?Single-candidate instruction specifications change deck-level instruction-failure flags on 6–7 of 72 decks, mostly by missing failures (Appendix C).
Is it the information or the judge?The same judge (Sonnet 4.6) given the request, the full attachment text and all pages, but no frozen specification, reaches AUROC .765, yet its own flags recover only 69/124 blocked decks and 17/71 content failures. SlideAudit re-run with Sonnet 4.6: AUROC .725, FPR 79.3% at 89.5% recall (Section 5.2, Appendix K).
Do comparison schemes miss visual defects?They respond to defective pages, and the signal is diluted at deck level (Appendix K).
Is it the renderer?Keeping only occlusion and out-of-page defects, which are least sensitive to font substitution, 120/124 decks remain flagged (Appendix J).
Does it transfer to other systems?Codex and NotebookLM on 12 tasks with unchanged specifications: 3/12 and 1/12 decks blocked (Appendix L).
InterventionsSame backbone, workflow changed: instruction-failure decks 15→1, cropping/occlusion decks 11→12 (Section 5.3). Controlled defect injection turns all 10 targeted instruction verdicts from pass to fail, worsens 9 of 10 targeted content verdicts, and lowers 8 of 10 targeted page scores (Appendix H.2).

Revisions to the paper: (1) a walk-through of one task from request to decision (Appendix I); (2) the blinded audit and the specification-exclusion analysis (Section 5.4, Appendix J, Table 13); (3) the single-candidate ablation and a clarified description of how instruction and content specifications are built (Section 4.1, Appendix C); (4) information- and judge-controlled comparisons and the page-level dilution analysis (Section 5.2, Appendix K); (5) the rendering pipeline, the text-overflow analysis and two additional systems (Appendices J and L); (6) Table 12 computed directly from the released item-level annotation file; (7) PPTEval recomputed using content and design only for 39 decks whose coherence response could not be parsed and had been recorded as zero (AUROC .666, p = .031, FPR 73.2%), so four of six schemes now separate the groups, while none screens individual decks at high recall without flagging most of the others; (8) request provenance and the collection protocol (Appendix A).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.