acceptodds
Under review as a conference paper at ICLR 2027

PPTBench: Can Coding Agents Reconstruct the Visual World through Structured, Editable Slides

14 views

Abstract

Coding agents are increasingly moving beyond text-based software tasks to reconstruct visual targets through code. This capability, visual coding, requires agents to translate their understanding of visual targets into executable code. Slides provide a natural testbed for this capability, combining rich visual structure with objects that can be programmatically created and edited. To measure this capability, we introduce PPTBench, a benchmark for reconstructing scientific flow diagrams as editable PowerPoint slides. PPTBench covers 500 scientific flow-diagram tasks across 10 presentation domains, drawn from real research papers, and is evaluated with a four-stage agentic judge covering artifact validity, process and connector fidelity, rendering quality, and visual fidelity. Evaluation of ten models across 36 model–harness–effort configurations shows a substantial gap in reliable visual coding: the best configuration achieves 77.34, while the median across configurations is 24.38. Fine-grained analysis shows that agents can generally produce valid slide files, but struggle to produce high-quality reconstructions that faithfully recover the semantics and visual structure of the target. Further analysis shows that increasing reasoning effort primarily improves hard-gate passage rather than mean detail quality on each configuration's passing tasks, while configurations with more inspection tend to achieve higher overall scores. PPTBench establishes a measurable testbed for studying visual coding and advancing agents toward more reliable visual creation.

Primary area:
datasets and benchmarks
Keywords:
Coding Agent, Benchmark, Visual Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZLN9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

PPTBench contains 500 scientific diagram reconstruction tasks requiring editable PPTX outputs. The paper introduces a four-stage evaluation and compares 36 model–harness–effort configurations.

Critical strengths

The task is useful, and the evaluation is substantial. The paper also reports human agreement and judge repeatability.

Critical weaknesses
  1. Evaluator reliability does not adequately support the scoring scheme.

Human agreement is reported on the same cases used for judge selection, without an independent validation set. In Table 2, the judge disagrees with the human majority on 27 of 200 cases, including 15 human-approved outputs that are incorrectly assigned zero. Hard gating turns a local judgment error into complete failure, while the existing stability analyses do not adequately establish how these errors affect different models.

  1. The large evaluation leaves important questions about execution, cost comparability, and model coverage.

The paper identifies Codex, Claude Code, and OpenCode but does not sufficiently document service access and execution settings for every configuration. Cost accounting combines recorded charges and estimates and needs a consistent breakdown. Moreover, 33 of the 36 configurations come from OpenAI and Anthropic, while Kimi, Qwen, and GLM each receive only one. This limits the scope of conclusions about model capabilities and reasoning effort.

Questions for authors
  1. Figure 1 uses AI-generated artwork without clearly distinguishing illustrative content from actual benchmark samples. Decoration takes up substantial space, while the examples are difficult to inspect. Please replace it with readable, traceable task examples.
  2. Do the harnesses connect to official APIs, third-party compatible endpoints, or application sessions? Please report model identifiers, service providers, and software versions for every configuration.
  3. How are input, output, cached, and reasoning tokens counted? Are retries, failed calls, and tool calls included? Please provide consistent cost accounting and report judging costs separately.
  4. Why are reasoning-effort sweeps concentrated on OpenAI and Anthropic, with only one configuration for each remaining model? Do the conclusions hold under comparable budgets?
  5. Can the judge be validated on unseen cases, and can rankings be reported after human correction of gating errors?

Personally, I find it disappointing to see what looks like a direct output from an AI image generator used so prominently in the paper. I am comfortable with AI-assisted figure creation, including using AI to write structured plotting code. The current figure, however, does not reflect the care I expect in presenting a research contribution. Much of the writing also reads like lightly edited AI-generated prose, and several passages are difficult to follow.

The experiments are extensive, but the comparisons span different harnesses and are heavily concentrated on a small number of model providers. I understand that coding subscriptions may make some models more accessible to evaluate. For a benchmark paper, I would nevertheless expect broader model coverage, including models from other providers and competitive open-source models, with comparisons under consistent evaluation settings. I hope the authors do not find these comments discouraging. I would be willing to raise my score if they can adequately address these concerns.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZLN9

We thank the reviewer for the detailed and candid review. We are glad that the reviewer finds the task useful and the evaluation substantial, and values the reported human agreement and judge repeatability. We address each concern below.

W1 & Q5: Judge reliability and the 27 disagreements

The 200 validation samples were indeed also used for judge selection. The resulting bias is limited, and judge errors do not drive the ranking:

  • Selection could not fit the rules. The protocol, prompts and weights were developed on a separate 50 tasks and frozen beforehand (Appendix G.1); the 200 samples only chose among three frozen judges. This may make the reported agreement slightly optimistic, which we will state as a limitation.
  • Agreement matches human experts. The judge agrees with the human majority on 86.5% of samples (κ = 0.696), within the 83.5%–90.5% pairwise agreement among the three annotators.
  • Errors are not one-sided. The judge is stricter in 15 cases and more lenient in 12.
  • The ranking is robust. Four alternative scoring rules give Kendall τ ≥ 0.971 with the top five unchanged, and unanimous gating preserves the order of the Astra settings (Table 12).

W2 & Q2: Service access and execution environment

All configurations use APIs; no app sessions were used. GPT and Claude models go through a pass-through gateway to the official OpenAI and Anthropic endpoints, which preserves model identifiers, native reasoning parameters and token logs; Qwen, Kimi and GLM use their vendors' official APIs. Model identifiers are gpt-6-astra, gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, gpt-5.5, claude-opus-5, claude-sonnet-5, qwen3.8-max, kimi/kimi-k3 and glmcustom/glm-5.3-flash; the judge is gpt-5.6-luna (high) via Codex. Appendix J now lists these details for every model and effort setting.

W2 & Q3: Token accounting and judging cost

All costs are now computed with one formula: logged token usage times the official list price on the experiment date, so configurations are directly comparable.

  • Scope: the full session behind each scored artifact, including zero-scored ones; abandoned retries and sessions without an artifact are excluded, so cost and score cover the same artifacts.
  • Tokens: input, cache-read, cache-write and output are counted separately; reasoning tokens are billed as output without double counting.
  • Tool calls: their model tokens are included; local execution is not an API cost.
  • Judging is reported separately.

In total, generation costs $11,146.13 and judging $1,917.71. Per-configuration generation costs are reported in Table 1, and the accounting rules and prices in Appendix J.

W2 & Q4: Model coverage and similar budgets

  • The sweeps answer a different question from the cross-family comparison. We swept effort on OpenAI and Anthropic models because they include the top systems and their harnesses expose multiple levels; other families run at a high (mostly maximum) level to represent their strongest setting. Full sweeps are costly (Astra's five levels cost $3,228.69), so the budget went to the sweeps.
  • Conclusions hold at similar budgets. At roughly $300, Astra High scores 77.34, Qwen 3.8 Max 47.69, and Terra Max and Opus 5 High about 33. More spending does not guarantee more: Astra Max costs five times as much as Astra High yet scores 68.76. We impose no caps, since they would truncate the agent's own decision of when to stop.
  • Results refer to configurations. We agree that broader model coverage would strengthen generality and will note it as a limitation. Since the unit of evaluation is the model–harness–effort configuration, we will state that cross-harness comparisons reflect whole configurations, and we will release the data and code for further evaluation.

Q1 and writing

We accept this criticism. The AI-generated illustration has been removed: Figure 1 now shows ten real source figures, one per domain, and Figure 4 has been redrawn with readable labels. We have also revised the prose to state procedures and results directly.

We thank the reviewer again for the valuable comments, which will make the paper more rigorous and readable, and we would be grateful if the reviewer would reconsider the score.

Thank you for clarifying the costs. My concern is the allocation of the budget: the reported $3,228.69 spent on the Astra sweep could support several smaller evaluations of models from other providers.

The similar-cost comparisons are useful, but they do not show whether the findings about reasoning effort hold across model families. Testing other families only at high or maximum effort leaves this question open.

I therefore consider this concern unresolved. A small matched-subset effort comparison across additional families would address it more directly, or the conclusions should be limited to the families tested in depth.

Thank you for the constructive suggestion. Following your recommendation, we conducted an effort-level comparison of GLM-5.3-Flash across Low, High, and Max reasoning settings on a randomly sampled 100-task matched subset:

MetricLowHighMax
Overall Score (/100)10.5439.8134.86
Valid Artifact Rate95/10095/10094/100
Semantic Gate Pass Rate18/95 (18.95%)48/95 (50.53%)40/94 (42.55%)
Render Gate Pass Rate14/18 (77.78%)47/48 (97.92%)40/40 (100.0%)
All-Gate Pass Rate14/10047/10040/100
Mean Detail Score (Passed)75.2684.7087.15

The results confirm that GLM-5.3-Flash exhibits the same non-monotonic degradation at the highest reasoning effort, peaking at High (39.81) and dropping noticeably at Max (34.86). Closer case inspection reveals a consistent underlying failure mode: while Max achieves a higher detail score on passed samples (84.70 --> 87.15), its semantic gate pass rate drops from 50.53% to 42.55% because High zooms into local regions (such as computational graphs) for direct visual inspection after initial script probing, whereas Max over-relies on scripts for pixel grayscale thresholding, ASCII pixel dumps, and repeated local font/bounding-box measurements while skipping direct visual verification of global topology and edges, resulting in missed or spurious connections. Alongside adding these results to the revision, we will explicitly scope our conclusions on reasoning-effort scaling in the main text and conclusion to the model families systematically evaluated across effort levels.

Thank you again for your thorough review. We hope these additional experiments and revisions address your concern, and we would be grateful if you would reconsider your score.

I think my request was misunderstood. I was asking for a broader comparison across providers and price points, for example by adding more Qwen models and including DeepSeek and Grok. The new GLM reasoning-effort sweep tells us how one model responds to different effort settings. It adds little evidence on how models from different providers compare at similar costs.

I still think the task is useful, but I find the benchmark construction fairly straightforward and see limited novelty in the design. My original assessment and rating stand.

Reviewer_CeoyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PPTBench, a benchmark of 500 scientific flow diagrams for evaluating image-to-editable-PowerPoint reconstruction. It combines deterministic artifact checks with a model-based judge for process semantics, rendering quality, and visual detail. The authors evaluate ten models across 36 model–harness–reasoning-effort configurations and analyze gate failures, visual defects, and agent behavior. They also report human agreement, judge repeatability, and ranking sensitivity analyses. The results suggest that generating a valid slide is easier than faithfully reconstructing its process and visual structure.

Critical strengths
  • Reconstructing scientific diagrams as editable slides is a useful task. Checking process connections in addition to visual appearance is a reasonable evaluation objective, since a visually similar reconstruction can still depict an incorrect workflow.

  • The evaluation covers multiple models and reasoning-effort settings. The human-agreement and sensitivity analyses provide useful initial checks on the proposed evaluation protocol.

Critical weaknesses
  • Evaluation Protocol and Judge Validity. The evaluation protocol lacks sufficient details, and the selection and validity of the judge require further verification. Section 2.3 outlines the scoring dimensions but focuses on what is checked rather than how to execute them, leaving complete judge workflows unclear. Furthermore, using GPT-5.6 Luna High as the judge while evaluating its own outputs necessitates independent validation across different models.

  • Task Difficulty and Failure Analysis. While performance shifts are documented, the analysis of task difficulty and failure modes remains limited. The data filtering process lacks explicit difficulty criteria or inter-annotator agreement, and domain counts do not reflect structural complexities. Additionally, Astra's performance drop under higher reasoning effort lacks sufficient explanation, as trajectory analysis omits Astra and fails to distinguish generation changes from judge variance.

  • Figure Readability and Presentation. Figure 1 combines multiple domain examples and model scores in a single figure. Some text and visual details are difficult to read, making it hard to understand the tasks and their complexity from these examples. Figure 4’s layout also makes it harder to read. The authors should improve the figures by enlarging key elements and using more readable labels.

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Ceoy

We thank the reviewer for the constructive review. We are glad that the reviewer finds the task useful, considers checking process connections beyond visual appearance a sound evaluation objective, and views the human-agreement and sensitivity analyses as useful checks on the protocol. We address each concern below.

W1: Evaluation procedure and judge independence

Appendix M now details the full judge workflow**, and the judge agrees with humans on gate decisions about as closely as human annotators agree with each other.**

Workflow**.** Appendix M gives the inputs, main instructions and outputs of each stage with a worked example, and the full prompts are included in the supplementary material. After a programmatic validity check (Stage 0), the judge runs three independent rounds, each consisting of:

  • Semantic gate (Stage 1): it lists nodes and directed edges in both figures and fails if either set difference is non-empty.
  • Rendering gate (Stage 2): it checks for blank regions, distortion, ghosting and illegible text.
  • Detail assessment (Stage 3): if both gates pass, it reports structured defects in layout, text and local graphics.

Gates are then decided by majority vote, defects are merged across rounds, and a deterministic formula converts them into the score (Appendix I). The revision also states this order in the main text.

Reliability. On 200 blind-labelled cases, the judge agrees with the human majority on 86.5% of gate decisions, within the 83.5%–90.5% pairwise agreement among the three annotators (Table 2), and two independent three-round majorities agree on 96.0% (Table 3).

Independence. As detailed in our response to Reviewer w2yM, the judge sees only rendered images and must name the faulty element, and the ranking does not favour its family (the judge itself ranks 21st, behind Kimi K3 and Qwen 3.8 Max). A cross-family judge could not be run within this rebuttal and will be noted as a limitation.

W2: Task difficulty and Astra's decline at higher effort

Difficulty. Scientific workflow diagrams rarely admit a unique topology: nested stages and shared junctions allow several valid decompositions with different node and edge counts, so we did not define difficulty by graph size. Judging whether a reconstruction omits a node or misroutes an edge relative to the original is, by contrast, well posed, and humans and the judge agree on it reliably (Table 2). We will clarify that the domain distribution shows source diversity, not difficulty.

Astra's decline. Two analyses rule out judge variance:

  • It persists under unanimous gating. Counting only 3/3 passes, which removes every split verdict, the order remains High (70.20) > XHigh (68.85) > Max (62.59) (Table 12).
  • Max breaks previously correct outputs. From High to Max, 66 tasks newly pass but 115 are lost, while detail quality on the 289 shared passes is slightly higher.

This pattern points to Astra's over-editing, i.e., repeated revisions that break correct connections, rather than weaker drawing.

W3: Figure readability

We have revised both figures. Figure 1 now shows ten domain cards, each with one complete source figure sized for legibility, alongside the best score per model. Figure 4 is redrawn at print size with readable labels.

We hope these responses address the reviewer's concerns, and we would be grateful if the reviewer would take them into account in the final assessment.

Thank you for the response and additional analyses. While these modifications have improved the clarity of the manuscript to some extent, several core concerns raised in my initial review remain unresolved:

  1. Insufficient validation of judge independence. To assess whether the evaluation results are biased toward a particular model family, cross-family judge validation remains necessary.

  2. The response does not address the question regarding task difficulty. In my initial review, I asked about the actual difficulty criteria and filtering standards used during dataset construction. I expected the authors to explain what constitutes a “hard” versus an “easy” task, and whether specific difficulty criteria were applied when filtering the data. Simply replying “we did not define difficulty by graph size” does not explain how task difficulty was assessed.

  3. Insufficient explanation of Astra’s performance decline at higher reasoning effort. Table 1 shows that Astra’s performance declines as reasoning effort increases from High to XHigh and Max, which is a counterintuitive phenomenon worth further investigation. Although the authors’ additional analyses of pass rate changes, shared passing tasks, and unanimous gating help decompose the score changes, they still do not explain why higher reasoning effort leads to lower performance. A task passing at High but failing at Max only indicates a performance regression; it does not prove that repeated revisions broke previously correct connections during generation. If the authors attribute this to “over-editing,” they need to provide specific generation trajectories, intermediate outputs, or targeted case studies to support this hypothesis.

  4. Reproducibility limitations remain despite clearer evaluation procedures. While the main instructions and output formats added in Appendix M are helpful, I could not locate the supplementary material containing the “full prompts” mentioned in Appendix M on the submission page. Please provide a direct link or specify where it can be accessed. Furthermore, semantic gating, rendering checks, and defect identification currently rely on the proprietary GPT-5.6 Luna High. Even with a fixed scoring formula, reliance on proprietary APIs limits independent and long-term reproduction of the evaluation. Therefore, beyond releasing the full prompts, execution configurations, and evaluation code, I also hope the authors can provide comparative results using open-source VLM judges.

  5. Limited depth of failure-mode analysis. The authors did not address my concern regarding the depth of failure-mode analysis, which remains a critical issue.

Thank you for your follow-up feedback. We have added open-source judge validation, artifact-level case studies, and direct repository links to address your remaining concerns:

1. Judge independence & open-source VLM validation

An independent open-source judge (GLM-5.3-Flash High) achieves 92.5% agreement with human majority on the same 200 blind-labelled cases (exceeding GPT-5.6 Luna High's 86.5%), confirming cross-family reliability (Appendix N). Treating gate failure as the positive class under three-round majority voting:

MetricGLM-5.3-Flash HighGPT-5.6 Luna High
Agreement with human majority92.5% (185/200)86.5% (173/200)
Cohen's κ\kappa0.8270.696
Gate precision91.5%88.9%
Gate recall97.7%90.9%
Gate F194.5%89.9%
Three-round unanimity94.0%87.5%

2. Task difficulty and filtering criteria

Human annotators screened 1,000 candidates down to 500 tasks using qualitative structural criteria: retaining diagrams with branches, feedback loops, and cross-stage links, while excluding simple linear pipelines, out-of-scope figures, and blurry/ambiguous references. Appendix O (Figs. 14–15) shows three exclusions and one retained example. We clarify in the revision that these criteria were qualitative (without numerical difficulty thresholds) and that domain counts reflect source diversity rather than difficulty.

3. Root cause of Astra's decline at higher reasoning effort

Astra Max's drop stems from sacrificing rendering robustness for higher path fidelity, causing custom connector paths to detach in LibreOffice (Appendix P, Fig. 16). Rather than using native PPTX connectors, Astra Max generates custom XML paths for finer geometric fidelity. While intact in its XML-based preview, these custom paths shift or detach when rendered in LibreOffice—for example, paths entering the processor and linking the model output to the analysis panel disconnect, altering the process semantics:

AspectSuccessful reconstructionAstra Max
Score92.500
Gate decisionsAll three rounds passAll three rounds fail the semantic gate
Straight linesStandard preset connectorsCustom paths
Complex pathsCustom geometry in ordinary shapesCustom geometry in connector objects
ArrowheadsSeparate shapesIncluded in custom paths or drawn separately

4. Prompt access and reproducibility

All judge prompts, configurations, and evaluation scripts are directly accessible at https://anonymous.4open.science/r/pptbench-anonymous-5E5A/ , alongside the workflow and worked example in Appendix M and our open-source judge baseline in Appendix N.

5. Depth of failure-mode analysis

Section 3.3 and Appendices C, D, F, and H analyze failures across three levels:

  • Structural semantics (Appendix C.3, Fig. 6; Appendix D, Figs. 7–8): Preserving topology is the primary bottleneck; models often preserve overall appearance while introducing disconnected branches, reversed arrows, or altered feedback paths.
  • Rendering-induced visual defects (Section 3.3, Fig. 4; Appendix D.1, Table 7; Appendix H, Table 14 & Figs. 12–13): Across 5,714 gate-passing slides, text defects account for 53.0% of detail deductions (including 1,864 unexpected line-wrapping cases), showing why checking native XML geometry alone misses renderer-induced clipping and overflow.
  • Agent behavior analysis (Section 3.3; Appendix F, Fig. 9 & Table 10): Across 27 configurations, inspection frequency correlates much more strongly with final score (Pearson r=0.881r = 0.881) than revision frequency does (r=0.255r = 0.255). Section 3.3 explicitly treats this as an observational association rather than a causal claim.

We hope these additions resolve your remaining concerns.

Reviewer_w2yMOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

PPTBench asks whether a coding agent can rebuild a scientific flow diagram as a natively editable single slide PPTX given only a raster image, using 500 figures from arXiv papers screened down from 1,000 by human annotators. The score multiplies three gates, namely artifact validity, semantic correctness of nodes and directed connections, and rendering legibility, against a detail score over layout, text and local graphics. Ten models in 36 model, harness and effort configurations give 18,000 reconstructions, with a best of 77.34 and a configuration median of 24.38.

Critical strengths
  1. The evaluation validity work is the best part and is better than most benchmark papers manage. Blinded human gate agreement at 86.5 percent with a Wilson interval and Cohen kappa 0.696, human Fleiss kappa reported as a reference point, judge self consistency across repeats, two independently collected three round majorities agreeing at 96.0 percent, four alternative scoring policies that leave the ordering intact, and bootstrap ranking stability by benchmark size. They also report that human issue set agreement is poor, at 17.3 percent exact match and Jaccard 0.392, which is the number most authors would have left out.

  2. The effort sweep is analyzed properly rather than reported as a trend. Decomposing the effect shows that more reasoning raises gate passage and barely moves conditional detail, and the matched common pass comparison in Table 8 catches something a conditional mean would hide, namely that Astra at max gains 66 passing tasks while losing 115.

Critical weaknesses
  1. The judge is from the same family as six of the ten evaluated models, and GPT-6 Astra then takes four of the top five slots. The repeatability check uses Luna, Sol and Terra, which are all the same family, so it measures stability and not independence. Panickssery et al. 2024 is in the reference but self preference is never tested, and with a near binary gate this is the one thing that could move the ordering.

  2. Two further choices make the headline harder to read than it looks. Mean score and gate pass rate correlate at 0.997 and conditional detail sits between 85 and 97 for nearly every competitive configuration, so the elaborate Stage 3 rubric contributes almost no discriminative signal and the 0 to 100 scale is a pass rate wearing a disguise. And Section 3.1 says evaluation covers the whole benchmark including the 100 tasks used for protocol development and judge selection, so a fifth of the evaluation set was used to tune the judge.

Questions for authors
  1. Please rejudge a stratified sample with a judge from another family and report whether the ordering aligns.

  2. What are the mean scores on the 400 tasks untouched by protocol development and judge selection?

  3. The figures come from public arXiv papers and the models are trained on arXiv. Can you split performance by source paper date against training cutoffs?

  4. Given r equals 0.997, what does Stage 3 add over gate pass rate alone? The Kendall tau between the two orderings would settle it.

2 comments
AuthorsOpenReview ↗

Response to Reviewer w2yM

We sincerely thank the reviewer for the thoughtful and constructive review. We are glad that the reviewer finds our validity analysis thorough and transparent, and our effort analysis informative beyond the aggregate trend. We address each point below.

W1 & Q1: Judge independence and self-preference

Owing to API cost and the rebuttal timeline, we could not re-judge a sample with a cross-family judge in this round. The available evidence, however, makes same-family preference unlikely to drive the ranking:

  • The ranking does not favour the judge's family. The judge, GPT-5.6 Luna High, ranks only 21st of 36 (19.47), and GPT-5.5 ranks 26th, whereas Kimi K3 (5th), Qwen 3.8 Max (8th) and Claude Opus 5 (10th) all rank above it. Astra's top-five placements thus reflect that model rather than the OpenAI family.
  • The mechanism of self-preference is largely absent. Panickssery et al. (2024) observe it in open-ended text, where a judge recognises its own style and inflates holistic scores. Our judge sees only LibreOffice-rendered PNGs, never code or text from the agent, and assigns no scores: a gate failure must name the wrong node or edge, and the final score is computed deterministically (Eqs. 2–4).
  • The judge was chosen by human agreement, among three candidates, and fixed across all configurations.

W2 & Q2: Results on the 400 held-out tasks

Re-scoring the frozen outputs on the 400 tasks used neither for protocol development nor for judge selection leaves the ranking essentially unchanged (Kendall τ_b = 0.975). No configuration moves by more than two positions, each stays within its 95% bootstrap rank interval, the top five are the same set, and Astra High remains first (78.99). Since no conclusion changes, we keep the 500-task results as primary; the 400-task results are now included in Appendix K.

Configuration500 tasks400 tasksRank
Astra High77.3478.991 → 1
Astra XHigh72.6073.852 → 2
Kimi K3 Max67.8070.285 → 3
Astra Medium68.1669.784 → 4
Astra Max68.7668.603 → 5

Q3: Contamination by publication date

We split the tasks by whether the source paper appeared on arXiv before or after each model's published cutoff (7 models, 33 configurations; papers from the cutoff month are excluded when only the month is disclosed).

ConfigurationPapers (pre / post)PrePostDifference [95% CI]
GPT-6 Astra High476 / 2477.7968.52−9.27 [−28.00, 7.99]
GPT-5.6 Sol Max438 / 6249.7745.85−3.91 [−16.15, 8.46]
Claude Opus 5 XHigh476 / 1436.6719.80−16.87 [−35.07, 4.69]
GPT-5.6 Terra Max438 / 6233.1735.56+2.39 [−9.15, 14.20]
GPT-5.6 Luna Max438 / 6232.2322.51−9.72 [−19.46, 0.63]
GPT-5.5 Medium413 / 8711.092.80−8.29 [−12.24, −3.91]
Claude Sonnet 5 XHigh422 / 679.595.21−4.38 [−8.87, 0.50]
  • Most differences are not significant. Post-cutoff scores are lower for 27 of 33 configurations, but 21 of the intervals include zero.
  • The direction is inconsistent within a model. Astra High drops after the cutoff, whereas Astra Medium and Max rise by 14.91 and 11.58 points, which contamination alone would not predict.

We therefore report a weak temporal association rather than evidence of contamination; given the small post-cutoff samples, we do not rule out training-data overlap either. The full split is now included in Appendix L.

W2 & Q4: Information added by Stage 3

We agree that the gates currently dominate: the rankings by pass rate and by final score have Kendall τ_b = 0.948. But Stage 3 still adds information:

  • Detail scores vary widely, from 63.0 to 97.1 across configurations (Table 1), not 85 to 97; Opus 5 and Astra differ by nearly 15 points on outputs that are all semantically correct.
  • It reorders close configurations. Kimi K3 passes the gates more often than Astra Medium and Max (73.8% vs. 71.6% and 71.0%) yet ranks below both on detail.
  • It is diagnostic. 88% of gate-passing outputs still lose detail points, mostly for typography and overflow.

The dominance of the gates is itself a finding: current agents are limited by recovering connections rather than by visual detail. As that improves, differences will shift to detail quality, which Stage 3 is built to measure.

We hope these responses address the reviewer's concerns, and we would be happy to provide any further clarification.

Thank you for the response and the additional analyses. Having read the other reviewers' comments, I hope the authors will prioritize the issues of judge validity and failure analysis in the camera-ready version. I will keep my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.