- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduce SlideGate, a benchmark for presentation generation that asks whether a generated slide satisfy delivery requirements. It contains 40 tasks across 7 scenarios and five input forms. For each task, an LLM pipeline constructs instructions and requiremsnts from requests and source materials, verifies their grounding, and freezes them across systems. Claude then judges hard constraint satisfaction, content coverage, depth, and faithfulness, together with page level visual defects and design. A slide is flagged when it violates a hard instruction, omits required content, or contains cropping or occlusion. On 240 decks from 6 commercial systems, the paper then reports 124 flagged decks and finds that 6 existing quality scores screen these outputs poorly. They then do the validation with programmatic page counts, two content annotators, and three designers.
Critical strengths
-
The benchmark differ the presentation quality from fulfilling the actual request, A deck can look organized while still using the wrong lesson, omitting a required section, or make the explanation unreadable. SlideGate then ties these failures to task requirements and output evidence instead of relying on a simple overall quality score. Tasks with attachments, templates, and edits, together with real commercial outputs, bring the benchmark closer to actual use than reference based slide benchmarks.
-
The shared specifications and separate failure categories make the evaluation diagnostic and usefull, Requirements retain request quotations or source locations and remain fixed across the systems on the same task. Failures are reported per dimensions with page location, and the three failure types mostly affect different decks. The same backbone experiment shows that the protocol responds to real workflow changes, and the categories reveal different improvement needs. This design remains a valuable contribution even if the claims about existing scores were narrowed.
Critical weaknesses
1 The validation not directly establish the accuracy of the final blocking failure labels, I noticed the Section 4.4 flags an entire deck when any required check fails. So these labels determin the reported 124 failure decks and the reference groups used to evaluate other scores. A single erroneous requirement or verdict can therefore affect both headline findings. Appendix H validates content depth and visual preferences, which do not directly establish whether required content is missing or a page is cropped or occluded. The specification review is useful, but Table 12 also contains 12 items both annotators considered unnecessary and 15 disputed items. Their effect on the final flags is not reported. Recomputing the findings under these existing annotations, together with a small human audit of flagged and unflagged cases, would connect the validation to the actual claims.
2 The comparison does not isolate the benefit of SlideGate’s verification design from differences in available information and judges Section 5.2 states that most comparison scores dose not receive the request or source materials. A score without those inputs cannot reliably determine whether a requested section or source fact is missing. Appendix E also uses GPT-5.6-sol for prompt based comparison schemes, whereas SlideGate uses Claude Sonnet 4.6. Though the author acknowledges these differences and the experiment tests whether existing scores can be repurposed for failure screening. It does not establish how much improvement comes from SlideGate’s specification construction rather than simply supplying task information. Moreover, several comparison schemes already assess visual defects, so their performance on the combined failure label does not establish that they miss cropping or occlusion.
Questions for authors
Here are some of my questions in details for this paper, the author can answer some of them or not.
- How do the content failure count, the 124 deck union, and screening results change after excluding the 12 consensus unnecessary items and varying the treatment of the 15 disputed items in Table 12? Does the results still stable?
- How does the author calculate the agreement between the two annotators, for example, Cohen’s kappa. In Table 12 the kappa is .65, but if calculate the standard binary Cohen’s kappa pooled over those 270 items, it can only be approximately .585 to .590, not .65?
- With the judge, requests, and source materials held fixed, how does a single candidate specification compare with the full procedure of candidate consolidation, critique, and rule validation? I am curious if we do distinguish missed requirements, unsupported additions, and changes in final failure labels. Though Section 5.4 reports consistency with generated candidates, this does not independently measure completeness.
- Table 4 describes ordered rendered pages, while Table 5 identifies a web slides interface with PPTX export for Manus. How was each system’s output saved and converted into page images, including the renderer and fonts? Were flagged cases checked against the delivered artifact and the system’s own display? I think this matters because export or rendering can introduce new cropping and occlusion?
- Who wrote and selected the 40 requests? Appendix A distinguishes the formal request from an earlier short message. Were requests finalized before inspecting system outputs, and were generations retained after one attempt or selected after retries or manual edits?
4 comments
Response to Reviewer uYyY (2/2): Specification Construction, Rendering, and Request Provenance
Q3. A single candidate specification versus the full procedure.
With judge, prompt, requests and sources fixed, we scored 72 decks (12 tasks × 6 systems) with each of three single-candidate instruction specifications and with the final one (Appendix C). On the deck-level 1–5 instruction rating that the judge returns with its item verdicts, a single candidate agrees exactly with the final specification on 71–74% of decks (Spearman .59–.68), and the system ranking changes (Kendall .60–.83): all three single candidates rank Doubao first, while the final specification ranks AnyGen first. Deck-level instruction-failure flags differ on 6–7 of the 72 decks; the final specification flags 15 decks and each single candidate 11–13, so most changes are failures a single candidate misses. For example, on the task "organize the contents and images of the two files", all single candidates give a score of 5 to two decks that use at most one of the attachments' 27 embedded images; only the consolidated specification retains the image requirement as a hard item, and both decks receive 2. In an audit of earlier single-pass labels, an internal page-count estimate was treated as a hard constraint on 19 of 40 tasks, inferred requirements were scored on 17, and optional wording was upgraded to mandatory on 9; consolidation against the request text, critique and rule validation are designed to remove these, and freezing ensures all systems face the same items. We have also clarified in Section 4.1 that the three-candidate procedure applies to instruction specifications, while content specifications use source-constrained extraction, code validation and a critic that may only remove items.
Q4. Saving and rendering.
Thank you for raising this. We evaluate the file each system delivers: the exported PPTX for all six systems, including Manus's PPTX export from its web-slide interface. All files go through one pipeline without per-system tuning: LibreOffice 6.4.7 headless to PDF, then pdftoppm at 150 dpi, yielding 3,026 pages (Appendix J). We view defects introduced by a product's own export as part of its delivery quality. Fonts are chosen by the generating system; fonts not installed on the rendering machine are replaced by LibreOffice's fallback fonts, as they would be for a user opening the file in the same environment. Flags are determined on the delivered file rather than on in-app previews, because the file is what users receive. Since text overflow is the defect most sensitive to font substitution, we also keep only defects classified as occlusion or elements beyond the page: 120 of 124 decks remain blocked. In the Codex physics deck, the same pipeline renders all of roughly 100 formula objects in the original lecture, whereas the delivered file contains none of them, so the information is lost before rendering.
Q5. Requests, selection, and generation protocol.
We have added these details to Appendix A. We selected the 40 tasks by stratifying over scenario and input form. The 28 tasks with attachments come from real use: we invited practitioners from a range of industries to use an internal presentation-generation system for their own work, and de-identified the content while keeping input forms and scenarios. In that system a user types a message and also selects the scenario, audience, length and role; the formal request merges the message, the selected options and the attachment names into one instruction, and is identical to the message when the message already states them. The 12 text-only tasks cover brief and detailed requests without materials; their topics are drawn from a library of roles and presentation topics with sampled style parameters, and during test-set construction each short topic message was expanded into a detailed request stating audience, structure, style and length, so that the set spans a controlled range of request complexity; their input-form labels follow the original test-set design (six brief, six detailed). The released task files contain both the opening message and the formal request. All formal requests were fixed before any system output was collected. Runs that produced no deliverable were resubmitted; the first delivered deck was kept, without manual edits and without selection among multiple results.
Sorry due to CVPR ddl I only have time to review the author huge effort on rebuttal and reply these. The author's comments solve part of my concerns. Though I think there is still some way awary from strong accept (some how like 2.8/4), due to ICLR's 4 score rating scale this year, I still decide to give 3 score and with good luck to author. And please add all these materials in camera ready if accepted!
Thank you very much for taking the time to go through our response despite the CVPR deadline, and for raising your score. Your suggestions, especially the blinded audit of the final blocking decisions and the recomputation under the specification annotations, have made the paper clearer and more convincing. All of these materials are already included in the revised PDF, and we will keep them in the camera-ready version if the paper is accepted. Best of luck with your CVPR submission!
Response to Reviewer uYyY (1/2): Blocking-Label Audit, Specification Annotations, and Controlled Comparisons
We are grateful for the reviewer's detailed reading and concrete suggestions, and for the note that the design remains valuable even if the claims about existing scores were narrowed. We followed the suggested analyses.
W1 / Q1 / Q2. Accuracy of the final blocking labels and the effect of specification annotations.
Following your suggestion, two annotators reviewed 30 decks (15 flagged, 15 not) blind to system identity and to SlideGate's verdicts, checking each against the request, the original attachments and every page. Both confirmed a blocking failure in all 15 flagged decks; inter-annotator κ per failure type is .66–.86. Among the 15 unflagged decks, annotators A and B identified a blocking failure in six and five, three of them in common (Section 5.4, Appendix J). If 3/15 of the 82 screening negatives contained blocking failures and all of them were among SlideAudit's false alarms, its FPR at the same operating point would still be about 71%.
We also recomputed the findings under the specification annotations (Table 13):
Ranges treat decks whose status cannot be determined after removal as failing or as excluded. Thank you also for checking the agreement statistic: the revised Table 12 is computed directly from the released item-level annotation file (244 items required by both annotators, 12 by neither, 14 disputed; Cohen's κ = .61 with 94.8% raw agreement and Gwet's AC1 = .94, as about 90% of items fall in one class), and the analysis above uses exactly these labels. A blocking flag states a failed necessary condition for delivery; partially covered content, which SlideGate scores as partial by design, is reported but not gated.
W2. Information, judge, and visual defects.
We agree these factors should be separated. Given the request, the full attachment text and every page, but no frozen specification, the same Sonnet 4.6 judge reaches AUROC .765, yet its own flags recover only 69 of 124 blocked decks and 17 of 71 content failures, while both annotators confirmed a blocking failure in each of the 15 audited flagged decks. Re-running SlideAudit with Sonnet 4.6 raises AUROC from .672 to .725, with FPR still 79.3% at 89.5% recall (Section 5.2, Appendix K).
We also agree that several schemes assess overlap and bounds, and they do respond to defective pages: within the same deck, their page scores rank defect pages below the others with AUROC .70–.91, and SlideAudit's categories fire on 92.0% of the 88 flagged pages. The signal is diluted at deck level: one hit lowers a 12-page deck's SlideAudit score by about .004; AutoPresent's worst page instead of its mean raises visual-failure AUROC from .66 to .81; and alarming on any SlideAudit hit reaches 95% recall but flags 54% of decks without a detected failure. We therefore see detecting a defect on a page and making a delivery decision as related but different capabilities.