acceptodds
Under review as a conference paper at ICLR 2027

CurriculumWorldBench: Benchmarking Autonomous Reconstruction of Executable University Curricula from the Web

Abstract

University curricula are distributed rule systems: requirements, prerequisites, policies, and exceptions are scattered across institutional websites and must be composed correctly to support downstream decisions. We introduce CURRICULUMWORLDBENCH, a benchmark for autonomous reconstruction of source- grounded, executable curricula from the web. It contains 735 curriculum worlds from 111 institutions and evaluates agents both on what they extract and on whether the recovered rules behave correctly and whether claims of completeness are justified. Across models, scaffolds, controlled website layouts, and frozen institutional sites, we find a persistent gap between finding courses and recon- structing the rules that govern them. Even when course recovery is high, 63,432 of 84,036 decidable obligations are omitted entirely. Stronger models achieve higher whole program behavioral agreement, yet substantial coverage failures remain. Providing agents with the correct evidence pages improves reconstruction when they cannot search for sources on their own, but offers little additional benefit once search is already available. Agents also often report that reconstruction is complete even when the resulting curriculum remains incomplete or behaviorally incorrect. Together, these findings characterize curriculum reconstruction as a distinct agent capability. Success requires more than retrieving relevant pages or extracting individual rules. An agent must identify the governing sources, integrate their requirements into a coherent executable model, and recognize when important parts of that model are still missing or unresolved.

Primary area:
datasets and benchmarks
Keywords:
datasets and benchmarks, web agents, autonomous knowledge acquisition, curriculum reconstruction, executable specifications, semantic evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 6 reviews, above 51% of the others.
Reviewer_JWWFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CurriculumWorldBench, a benchmark designed to evaluate autonomous web agents on the task of reconstructing source-grounded and executable university curriculum rule systems directly from distributed institutional websites. Unlike conventional web navigation or information extraction benchmarks that focus on retrieving facts and completing transactional tasks, this benchmark frames curriculum synthesis as a distributed program synthesis and rule-recovery problem. Given only an institutional entry page, catalog year, and curriculum scope, an agent must autonomously discover governing source pages, extract and formalize degree requirements, prerequisites, and policies into an executable rule representation with source provenance, declare unresolved obligations as explicit UNKNOWNs, and produce a meta-cognitive completion claim.

Critical strengths

The authors frame curriculum reconstruction as an autonomous web-agent task requiring the production of persistent and executable domain models. The authors have made efforts to bridge three previously disconnected research areas that encompass web-agent benchmarks (e.g., WebArena, Mind2Web, τ-bench) which focus on transactional interactions; rule extraction and “rules-as-code” benchmarks (e.g., Merigoux et al., Lorenzo et al.), which supply the governing text in-context; and execution-driven program synthesis benchmarks (e.g., SWE-bench, AppWorld), which evaluate consequences of generated artifacts. The positioning appears to be strong because university curricula constitute an ideal compound-capability testbed. They are distributed across heterogeneous web pages, hierarchically nested, version-sensitive (catalog years), logic-heavy (nested AND/OR/choose-k/credit-pool constraints), and strictly executable against student records. The paper’s central insight, that recovering vocabulary (courses) is a fundamentally different capability from recovering governing rules, is empirically validated and reframes how the community should think about agentic knowledge-base construction tasks. The explicit modeling of unresolved content via UNKNOWN declarations and completion claims as selective predictions is a methodologically mature design decision that distinguishes this benchmark from leaderboard-style evaluations.

Principled and Novel Task Formulation: The framing of autonomous executable curriculum reconstruction seems to be a new agentic capability axis that is well-motivated, cleanly defined via the tuple e = (r, y, s) and world w = (e, D⋆, C⋆, V ), and consequential for real-world applications. • Rigorous Behavioral Evaluation Protocol: Rather than relying on brittle syntactic overlap metrics (BLEU, ROUGE, AST-matching), the authors have implemented execution-based behavioral equivalence via automated counterexample search ∃x ∈ X : q(x) ̸= ˆ q(x) with independent counterexample replay, providing semantically sound rule correctness verification. • Controlled Experimental Design (Content vs. Topography Decoupling): Holding governing content D⋆ and reference rules C⋆ fixed while systematically varying website view v ∈ V across three controlled layouts (Departmental, Paginated, Deep Navigation) alongside frozen original institutional sites is a methodologically commendable decision that is able to isolates web-navigation capability from semantic parsing capability. • Multi-Model, Multi-Scaffold Breadth: The matrix of 6 LLMs × 5 scaffolds × 3 layouts × 243 worlds yields substantial empirical breadth and demonstrates that the observed “Course vs. Rule Gap” evidences as architecturally fundamental rather than scaffold- or model-specific. • Statistical Rigor: The use of institution-clustered non-parametric bootstrapping (B = 10,000) for all confidence intervals, paired-by-world comparisons, and empirical run-to-run variance thresholds for Claude model repetitions may show that the manuscript paid attention to clustered-data hierarchies and LLM API stochasticity. • Proactive Evaluator and Metric Validation: The authors audit their own evaluation tooling via (i) 22,914 synthetic rule mutants (99.8% meaning-change detection, 0 false positives on 14,265 meaning-preserving rewrites), (ii) human dual-annotator audits on 300 absent-flagged obligations (95.0% agreement), and (iii) closed-book memorization controls bounding pretraining contamination. • Deep Diagnostic Failure Decomposition: The reviewer feels that the paper is able to dissect failures into source-access deficits vs. synthesis omission vs. present-but-wrong rules vs. miscalibrated completion claims and therefore provides a clear forward research agenda. • Candid Limitations Disclosure: The limitations section (Section 8) and ethics statement admit sample size constraints, alignment brittleness, redistribution concerns, and even a specific crawler concurrency incident on 2026-09-07.

Critical weaknesses

Limited Confirmatory Tier Sample Size: All headline claims are anchored on 63 closure-complete worlds across only 18 institutions. This looks to yield wide bootstrap confidence intervals (e.g.,Sonnet 5 BM25 Executable Recall: 25.0 [5.8, 40.8]) that make fine-grained model/scaffold comparisons statistically underpowered.

Suggestion: The authors should explicitly report power analyses or clarify which pairwise comparisons achieve statistical separation after multiplicity correction. The reviewer believes that expanding the confirmatory tier via further annotation effort would also substantially increase confidence in headline numbers.

• Alignment Step as an Uncontrolled Source of Noise: The 7.0% false-absent rate (with 13/21 of those being actually equivalent) implies that Executable Recall may be understated by up to ∼3 percentage points, a non-trivial margin when comparing close configurations (e.g., Sonnet One-shot 28.8% vs. BM25 25.0%).

Suggestion: The reviewer believes that the authors may explore semantic embedding-based alignment and LLM-assisted soft-alignment and report sensitivity bounds. • Rule-Language Expressivity and UNKNOWN Fraction: Approximately 30% of enumerated obligations are declared non-decidable and excluded from executable scoring. This risks survivor bias: are the simplest, cleanest rules being evaluated while the hardest real-world policies are structurally excluded?

Suggestion: The reviewer’s humble recommendation would be to provide a categorical taxonomy of what types of policies populate the UNKNOWN pool (e.g., residency rules, GPA floors, transfer caps) and report its distribution.

• Output Schema Brittleness Confounds Model Comparisons: The incident where GPT-6 scored 0.0% under the original strict two-space indentation contract (recoverable through a whitespacetolerant parser) may raise a concern that similar schema compliance issues may partially explain the extremely low scores of open-weight models (Gemma 2.3%–8.4% Exec. Recall). Suggestion: The reviewer believes that the authors may report a per-model schema compliance rate and conduct ablations with per-model schema-tuned parsers to isolate interface compliance from semantic reasoning deficits. • Limited Repetition for Non-Claude Models: It appears that only Sonnet 5 and Opus 5 received repeated executions for variance estimation. GPT-5.6, GPT-6, Qwen3.8-27B, and Gemma-3-27B-it ran only once per cell, meaning their reported numbers lack empirical noise floors. Suggestion: The revewier believes that if compute permits, at least one repeated run across all models may strengthen the breadth comparisons. • Lack of Chance-Corrected Agreement Statistics: Human annotation reliability is reported as raw percentage agreement (87.2%–92.1%) rather than chance-corrected measures such as Cohen’s κ or Krippendorff’s α. Suggestion: For a benchmark whose ground truth depends on expert adjudication, chance corrected statistics would be more informative. • Redistribution and Longevity Risk: The ethics statement acknowledges that per-institution redistribution authorization was not obtained for the frozen website snapshots that constitute the core benchmark artifact. Suggestion: The reviewer construes that the authors may proactively develop either a synthetic replica environment, a self-hosted mock institutional site, or an opt-in redistribution protocol to future-proof the benchmark against takedown requests. • Strict WORLDPASS Metric is Instrument-Bounded: The authors acknowledge that strict WORLDPASS is attainable in only 18 of 243 worlds due to reference-level credit-contract constraints, demoting it to a diagnostic. This limits headline interpretability. Suggestion: The reviewer opines that the authors should consider defining a graded closure metric (e.g., “closure at 90% obligation threshold”) as a middle-ground primary metric.

Questions for authors

See under weaknesses.

Reviewer_wRzzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes CurriculumWorldBench, a benchmark where web agents rebuild a university curriculum from the institution’s website, starting from an entry page, a catalog year, and a program scope. Agents return executable rules with source evidence, unresolved items, and a claim of whether the reconstruction is complete. Evaluation covers course recovery, obligation-level coverage, rule equivalence checked by counterexample search, whole-program agreement on student records, and the risk of completion claims. Six models and five scaffolds are tested on re-rendered layouts and frozen original sites. The main finding is that agents find courses much better than they recover the rules, mostly because obligations are omitted.

Critical strengths
  • Evaluation goes beyond surface matching. Rules are checked for equivalence by counterexample search, and the evaluator itself is tested with a mutation audit.
  • Obligation-level metrics separate finding courses from recovering rules, which makes the main gap visible.
Critical weaknesses
  • Figure 1 is too messy. I don’t know where to start reading it. Numbering the flow would help.
  • Many key terms are used in the abstract and intro without definition: curriculum world, executable (in what language?), diagnostic reference, etc. This makes the abstract and intro hard to read.
  • There is not enough reason to study curricula. Universities already have degree audit systems. It is also not explained why curricula are a good example of general agent ability compared to other rule systems like law, tax, or policy.
  • The intro does not discuss prior work at all (web agent benchmarks like WebArena, Mind2Web, BrowseComp; deep research; information extraction; rules-to-code). So it is not clear what problem existing work cannot solve.
  • The logic is circular. In the introduction, the evaluation results show a gap and this motivates the benchmark. Results from the benchmark are used to justify the benchmark.
  • Noise is large. There is no sampling control, one run per plan, and only one repeat. One-shot test-retest correlation is −0.35 to 0.24. Models outside the Claude pair use different transports and a vendor harness prompt (~11k tokens), so comparison across model families is not controlled. Component ablations are inconclusive.
  • Output format matters. As the GPT-6 whitespace case shows, contract compliance changes results a lot, so the benchmark largely measures format compliance.
  • Strict WorldPass is attainable in only 18 of 243 worlds. It is limited by the instrument, not by the agents.
  • BA has an empty-program floor of 0.351, and 47% of Sonnet 5 cells are not better than this floor.
  • There are reproducibility issues. The scorer pins only 9 of its 88 files, and 3 unpinned files changed after the runs. The main text and appendix use different scorer and contract versions. The Gemma row-map record is missing. 9 worlds were dropped because of an HTML conversion bug.
Questions for authors
  • See the weaknesses.
  • Why curricula? What does this benchmark show that WebArena, BrowseComp, or rules-as-code benchmarks cannot?
  • What is the rule language? Can you define obligation, closure-complete, diagnostic reference, etc. early in the paper?
  • For the main claims, how large are the differences compared with run-to-run noise? Can you add more repetitions?
  • How much of the omission comes from contract failures or empty outputs, and how much from real misses?
  • Can the scorer be changed so that strict WorldPass is attainable in more worlds?
  • Given the 0.351 floor, what does BA add beyond the rule-level metrics?
Reviewer_UVdDOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

CurriculumWorldBench tests whether LLM-based web agents can find a university's degree requirements on its website and turn them into executable rules. The agent must also provide source evidence, list unresolved items, and decide whether its reconstruction is complete. The benchmark checks three things: whether agents find the right sources, whether the recovered rules behave correctly, and whether their completeness claims are justified.

Critical strengths
  • The metrics are clearly separated: vocabulary recovery, obligation coverage, executable correctness, and justified completion. They reveal a clear gap between finding courses and recovering the rules that govern them.
  • The paper validates its own evaluator with a mutation audit, an alignment audit, and a closed-book control. It also states its limitations openly.
Critical weaknesses
  • The paper does not clearly say which skills the benchmark measures. It also does not explain why testing them together is needed beyond existing web agent benchmarks (e.g., WebArena, WideSearch) and rule extraction benchmarks (e.g., RuleArena, Catala).
  • The paper claims that rebuilding rules from a website is a different skill from searching the web or extracting rules from given text. But in the main experiments, the agent can open every page, so nothing needs to be searched for. The real website tests do require searching, but they use only two models, one run per setting, and give very low scores. This is too weak to support the claim.
Questions for authors

You claim that rebuilding rules from a website is a distinct skill that needs finding sources, building rules, and knowing when something is missing. In your main results the agent can open every page, so finding sources is not tested. What evidence shows that finding sources contributes to your 'distinct skill' claim? On real websites you tested only two models with one run each and got low scores. Can you add more models or repeated runs as main results, or soften the claim?

Reviewer_PqtHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CurriculumWorldBench, a benchmark for evaluating whether web agents can reconstruct executable curriculum rule systems (requirements, prerequisites, policies) from institutional websites. It is built from 735 rendered “curriculum worlds” at 111 institutions; 450 carry adjudicated references, and headline results are stated on 63 closure-complete worlds (243 scored worlds total). Evaluation separates entity recovery, obligation coverage, behavioral equivalence (via counterexample replay against student records), and completion claims scored as selective prediction. Six models across five scaffolds are evaluated. The central finding: course recovery is largely solved (entity F1 up to 90.5%) while rule reconstruction is not (190,296 of 252,108 decidable obligations omitted), and agents frequently certify incomplete reconstructions as complete.

Critical strengths
  1. The four-layer evaluation design (vocabulary / coverage / fidelity / completion) is the real contribution. Scoring “I am done” as a selective prediction with coverage-risk is a checkable, punishing operationalization of agent self-calibration, and the finding that 98% of Sonnet’s one-shot certifications fail the closure criterion is the kind of number the field needs.
  2. Validation depth is well above benchmark-paper average: 22,914 evaluator mutants, an audited alignment step, a closed-book control bounding memorization, and honest reporting of where the instrument itself binds (strict WORLDPASS attainable in only 18/243 worlds).
Critical weaknesses
  1. The confirmatory tier is thin relative to the claims’ framing: 63 closure-complete worlds from 18 institutions, with the authors themselves noting intervals are wide and that diagnostic references flatter agents by 3–12 points of Exec. Recall. The abstract’s “persistent gap” direction is well supported, but model-level orderings outside the Claude pair rest on single executions without sampling control.
  2. Self-contained terminology load (closure-complete, reference-decidable, obligation, adjudication tiers, worksheet contract) makes the benchmark hard to approach for potential reusers; the paper reads like a technical report for a system that must be adopted as-is. Adoption cost is a real risk for a resource paper.
Questions for authors
  1. In Table 1 the structured ledger’s Entity F1 is 71.4 for every model (deterministic adapters), yet its Exec. Recall ranges 9.2–36.4. Can you decompose where this spread comes from (rule emission quality vs. contract compliance)?
  2. The 98% certification risk is against strict obligation closure. Under a usable-reconstruction criterion (e.g., ≥80% decidable obligations recovered), what would the certification risk be? This affects how practically severe the “agents claim completion” conclusion is.
  3. Will the frozen snapshots, references, and scorer be released, and under what license? How large is the frozen-site corpus in storage terms?
Reviewer_TjPSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CurriculumWorldBench gives an agent an institution's entry page, catalog year and scope and asks for an executable curriculum with provenance, explicit unresolved items and a CERTIFY COMPLETE decision. Worlds fix the governing pages and an adjudicated reference; rendering varies over three controlled layouts and, for 234 worlds, frozen original sites. Scoring is obligation-level: entity F1, omission, executable recall via counterexample search with replay, behavioural agreement against an empty-program floor, and completion as selective prediction. Six models and five scaffolds run on 63 closure-complete worlds from 18 institutions (Table 1). Course recovery far exceeds rule recovery, absent obligations dominate the errors (Table 28), and certifications are almost always wrong under obligation closure (Sec. 6.3).

Overall: The protocol and evaluator audit are strong, but the headline table violates its own definitions, the confirmatory reference has no inspectable closure evidence, and the omission count and model comparisons are confounded by contract failures and a scorer change larger than most cross-model gaps. A consistent single-scorer regeneration of all tables, a documented adjudication of the 63 worlds, and a contract-aware omission split would move this to accept.

Critical strengths

Evaluator validity is measured rather than asserted: 99.8% of decided mutants are distinguished and none of 14,265 meaning-preserving rewrites are flagged (Table 6), and a 300-obligation sample bounds alignment false-absence at roughly 3 points of Exec. Recall (Sec. 5.2). That is what makes obligation-level numbers interpretable at all, and other benchmark authors should copy it.

The metrics separate things web-agent benchmarks usually conflate: vocabulary recall, obligation coverage, behavioural fidelity with an empty-program floor (Table 21), evaluable coverage (Table 32), and completion as a coverage/risk pair (Eq. 2). The strict-WorldPass attainability audit (App. R.1) and the closed-book control (Sec. 5.3) are honest instrument checks.

Critical weaknesses
  1. Table 1 is internally inconsistent. Sec. 4.4 defines omission as 1 - min(course recall, share of reference-decidable obligations recovered), and Exec. Recall as the share of reference-decidable obligations recovered and proven equivalent. Exec. Recall is therefore at most the recovered share, so Exec. Recall + Omission <= 100 in every cell and in any average over the same cells with the same weights. Table 1 gives Opus browsing 66.2 + 58.1 = 124.3 and Sonnet browsing 41.2 + 78.6 = 119.8. Rounding cannot do that. Either the two columns use different denominators, weights or scorer versions (App. C says the Table 1 scorer differs from the archived one behind every appendix table, and GPT-5.6 one-shot is 49.8 in Table 25 vs 65.4 in Table 11 on the same cells), or a definition is wrong. Until the per-cell numerators and denominators are shown and all current tables regenerated from one scorer, the headline table cannot be read.

  2. The closed obligation inventory, the benchmark's distinguishing claim, cannot be inspected for the confirmatory tier. Sec. 4.3 defines closure-complete as adjudication finished with a recorded ruling for every UNKNOWN. App. G.1 says no adjudicated worksheet directory, per-disagreement record or adjudicator field exists, and App. D records the overall reference as incomplete (1,251 fields awaiting a ruling). A collection can be incomplete while a subset is closed, but then the evidence for the 63-world subset has to be shown, and it is not. Add that 891 of 1,071 worksheets were machine pre-filled with an association between pre-filling and agreement with the proposal (Table 7: 0.921 vs 0.872; the paper itself flags assignment confounds), natural-language fields were translated by one model and accepted by another (App. D), and agreement is not chance-corrected (Sec. 5.1). Every omission, Closed and completion-risk number in Secs. 6 and 7 is relative to this inventory.

Other issues: the abstract's "190,296 of 252,108 omitted entirely" comes from Table 28, which App. T says attributes no cause, and contract failures (GPT-6 1,440, Qwen 1,395, Gemma 1,233 of 3,645 cells, Table 17) each count as omission 1. Run-to-run noise is of the order of the reported differences (Table 19), yet Sec. 6.2 claims a 25 to 37 point Opus-over-Sonnet gain without Table 10's beyond/within annotation. Discovery is trivial by construction on controlled layouts (App. B); on original sites the evidence is reachable within the page budget but the median traversal reads 783k characters (Table 15), so the one-shot condition tests the character cap more than discovery. The +18.6 ledger gain (Sec. 6.3) rests on a restricted intersection after 27 worlds timed out (App. V). KNOWPLAN already produces a closure certificate from frozen official sites (Table 29); what this benchmark adds as an evaluation is stated in App. U but never shown.

Presentation: The main text reads well, but there are two scorers and two contracts the authors call "not interchangeable" (App. C), Table 1 reports Exec. Recall while the appendix reports SemAcc, Sec. 5.3 cites 60.0/41.1 as Table 1 values when they come from Table 11, and Fig. 2 does not appear to plot the Table 1 values its caption says it plots (Opus browsing Exec. Recall sits in the mid-50s, not at 66.2).

Questions for authors
  1. [clarification] How can Exec. Recall + Omission exceed 100 in Table 1 under the Sec. 4.4 definitions? Please give per-cell numerators, denominators, weights and scorer version for both columns, and regenerate Table 1 and Fig. 2 together.
  2. [clarification] What evidence establishes that adjudication was completed for the 63 confirmatory worlds, and by whom, given App. G.1?
  3. [clarification] Split Table 28 by cell status (contract failure, empty, parsed). What share of omissions survives on contract-passing cells?
  4. [clarification] A per-row reconciliation of archived vs current scorer on the Table 1 cells, split into alignment vs contract. Can alignment precision, not only false absence, be audited?
  5. [clarification] What does this protocol measure that KNOWPLAN's closure certificate does not?
Reviewer_djoKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary
  • This paper proposes CurriculumWorldBench. The authors argue that university curricula are a kind of distributed rule system. For this task, an agent must be able to find appropriate sources on the web and retrieve the necessary information. It must also explore and compose obligations and ultimately reconstruct them into executable rules. On this basis, the authors claim that the benchmark can measure web agents' "ability to reconstruct an executable domain model from distributed sources." The benchmark evaluates agent capability along four aspects, obligation-level coverage, semantic checks, program-level behavior, and assessment of completion claims, in two environments: controlled layouts and snapshots of real frozen websites. The dataset spans 111 institutions and 450 worlds, of which 63 are closure-complete.
  • Experiments were conducted on Claude Sonnet 5, Claude Opus 5, GPT-5.6, GPT-6, Qwen3.8-27B, and Gemma-3-27B-it, across several scaffolds (one-shot, BM25+LLM, Graph-RAG, ReAct, Structured ledger). The results report the performance gap between finding entities and finding obligations. The best configuration was Opus 5 with direct exploration. The open-source models showed lower performance than the frontier models, partly due to output contract violations. Most of the models' failures fundamentally stem from failing to find the rules. The paper also reports performance on completion claims, noting that models' completion claims often correspond to reconstructions that are in fact incomplete.
Critical strengths
  • The paper decomposes agent capability into four metrics, obligation-level coverage, semantic accuracy of the rules, program-level behavior, and completion claims, and measures each separately. In addition, Entity F1 is used to examine whether the agent correctly found the course names.
  • The authors rendered the websites into multiple versions that change only the site structure. This allows them to measure agent performance even when the way information is organized differs.
  • In Section 5, the authors attempt to validate the reliability of the benchmark itself, dividing the analysis into annotation reliability, metric validity, contamination & leakage, and sensitivity analysis. In particular, the metric validity analysis in Section 5.2 rigorously validates the evaluator through a mutation test (22,914 mutants) and an alignment audit.
Critical weaknesses

W1. Performing CurriculumWorldBench requires the model to retrieve appropriate sources, identify the correct individual rules from them, and reconstruct those rules in an executable form. I therefore have no objection that the benchmark measures a comprehensive set of model capabilities. However, the domain is too narrow to call this a distinct agent capability. Even if rules differ across websites, it is hard to see their basic format differing very substantially. More convincing evidence is needed on how this benchmark differs from other benchmarks that evaluate diverse tasks on the web, and on whether this capability transfers to other domains.

W2. Since 180 worlds carry diagnostic references, only the 63 closure-complete references have actually completed human review. The main results in the paper are therefore based on only 63 worlds. With a single domain and such a small sample, the benchmark may be limited in its ability to measure overall agent capability. Moreover, as the authors acknowledge, strict WorldPass is attainable on only 18 of 243 worlds due to a limitation of the scorer. Given this, it is difficult to accept that the benchmark can rigorously measure whether an entire curriculum has been reconstructed. Although BA is a program-level metric, it measures agreement on a fixed record suite, and an empty program already achieves 0.351. It therefore cannot be regarded as a measure of whole-curriculum reconstruction accuracy. The confidence intervals in Table 1 are very wide, which makes it difficult to trust the rankings among models or scaffolds. Since seeds and sampling were not controlled, the actual uncertainty may be even larger.

W3. My biggest concern is that, although this is a benchmark paper, the construction process is not well described in the main text. For a benchmark paper, how the benchmark was built should be clearly presented in the main text. The main text does not describe what the student records look like, which institutions were selected, or how the renderings were produced. The structured output must contain a great deal of content, yet it is unclear how this was explained to the agent. I would like the authors to provide the prompts, at least in the appendix. The constraints of the scaffolds are not described in the main text and appear only in the appendix. From the main text, it is not possible to understand how many pages are given or what the ledger actually does.

Note to authors and AC: The weaknesses have been condensed and grouped for clarity. This is an editorial change only — the content and my overall evaluation remain the same as in the original version.

Questions for authors

Q1. The main text does not present an example of the overall flow of the benchmark. It would be helpful to include one.

  • In particular, the main text shows neither what the rule language (DSL) looks like nor any examples of it. It seems necessary to understand how difficult it is to correctly produce these rules. Figure 1 shows A AND (B OR C), but a more detailed example is needed.

Q2. On p. 4, omission is defined as "one minus the lower of course recall and the fraction of reference-decidable obligations recovered." Could you explain why it was defined this way? Is this course recall the same as the entity measured by Entity F1?

Q3. In the experimental design, most of the analyses are concentrated on Claude. Is there a reason why other frontier models, such as GPT-5.6 and GPT-6, were not evaluated on the frozen original-website condition?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.