- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CurriculumWorldBench, a benchmark designed to evaluate autonomous web agents on the task of reconstructing source-grounded and executable university curriculum rule systems directly from distributed institutional websites. Unlike conventional web navigation or information extraction benchmarks that focus on retrieving facts and completing transactional tasks, this benchmark frames curriculum synthesis as a distributed program synthesis and rule-recovery problem. Given only an institutional entry page, catalog year, and curriculum scope, an agent must autonomously discover governing source pages, extract and formalize degree requirements, prerequisites, and policies into an executable rule representation with source provenance, declare unresolved obligations as explicit UNKNOWNs, and produce a meta-cognitive completion claim.
Critical strengths
The authors frame curriculum reconstruction as an autonomous web-agent task requiring the production of persistent and executable domain models. The authors have made efforts to bridge three previously disconnected research areas that encompass web-agent benchmarks (e.g., WebArena, Mind2Web, τ-bench) which focus on transactional interactions; rule extraction and “rules-as-code” benchmarks (e.g., Merigoux et al., Lorenzo et al.), which supply the governing text in-context; and execution-driven program synthesis benchmarks (e.g., SWE-bench, AppWorld), which evaluate consequences of generated artifacts. The positioning appears to be strong because university curricula constitute an ideal compound-capability testbed. They are distributed across heterogeneous web pages, hierarchically nested, version-sensitive (catalog years), logic-heavy (nested AND/OR/choose-k/credit-pool constraints), and strictly executable against student records. The paper’s central insight, that recovering vocabulary (courses) is a fundamentally different capability from recovering governing rules, is empirically validated and reframes how the community should think about agentic knowledge-base construction tasks. The explicit modeling of unresolved content via UNKNOWN declarations and completion claims as selective predictions is a methodologically mature design decision that distinguishes this benchmark from leaderboard-style evaluations.
Principled and Novel Task Formulation: The framing of autonomous executable curriculum reconstruction seems to be a new agentic capability axis that is well-motivated, cleanly defined via the tuple e = (r, y, s) and world w = (e, D⋆, C⋆, V ), and consequential for real-world applications. • Rigorous Behavioral Evaluation Protocol: Rather than relying on brittle syntactic overlap metrics (BLEU, ROUGE, AST-matching), the authors have implemented execution-based behavioral equivalence via automated counterexample search ∃x ∈ X : q(x) ̸= ˆ q(x) with independent counterexample replay, providing semantically sound rule correctness verification. • Controlled Experimental Design (Content vs. Topography Decoupling): Holding governing content D⋆ and reference rules C⋆ fixed while systematically varying website view v ∈ V across three controlled layouts (Departmental, Paginated, Deep Navigation) alongside frozen original institutional sites is a methodologically commendable decision that is able to isolates web-navigation capability from semantic parsing capability. • Multi-Model, Multi-Scaffold Breadth: The matrix of 6 LLMs × 5 scaffolds × 3 layouts × 243 worlds yields substantial empirical breadth and demonstrates that the observed “Course vs. Rule Gap” evidences as architecturally fundamental rather than scaffold- or model-specific. • Statistical Rigor: The use of institution-clustered non-parametric bootstrapping (B = 10,000) for all confidence intervals, paired-by-world comparisons, and empirical run-to-run variance thresholds for Claude model repetitions may show that the manuscript paid attention to clustered-data hierarchies and LLM API stochasticity. • Proactive Evaluator and Metric Validation: The authors audit their own evaluation tooling via (i) 22,914 synthetic rule mutants (99.8% meaning-change detection, 0 false positives on 14,265 meaning-preserving rewrites), (ii) human dual-annotator audits on 300 absent-flagged obligations (95.0% agreement), and (iii) closed-book memorization controls bounding pretraining contamination. • Deep Diagnostic Failure Decomposition: The reviewer feels that the paper is able to dissect failures into source-access deficits vs. synthesis omission vs. present-but-wrong rules vs. miscalibrated completion claims and therefore provides a clear forward research agenda. • Candid Limitations Disclosure: The limitations section (Section 8) and ethics statement admit sample size constraints, alignment brittleness, redistribution concerns, and even a specific crawler concurrency incident on 2026-09-07.
Critical weaknesses
Limited Confirmatory Tier Sample Size: All headline claims are anchored on 63 closure-complete worlds across only 18 institutions. This looks to yield wide bootstrap confidence intervals (e.g.,Sonnet 5 BM25 Executable Recall: 25.0 [5.8, 40.8]) that make fine-grained model/scaffold comparisons statistically underpowered.
Suggestion: The authors should explicitly report power analyses or clarify which pairwise comparisons achieve statistical separation after multiplicity correction. The reviewer believes that expanding the confirmatory tier via further annotation effort would also substantially increase confidence in headline numbers.
• Alignment Step as an Uncontrolled Source of Noise: The 7.0% false-absent rate (with 13/21 of those being actually equivalent) implies that Executable Recall may be understated by up to ∼3 percentage points, a non-trivial margin when comparing close configurations (e.g., Sonnet One-shot 28.8% vs. BM25 25.0%).
Suggestion: The reviewer believes that the authors may explore semantic embedding-based alignment and LLM-assisted soft-alignment and report sensitivity bounds. • Rule-Language Expressivity and UNKNOWN Fraction: Approximately 30% of enumerated obligations are declared non-decidable and excluded from executable scoring. This risks survivor bias: are the simplest, cleanest rules being evaluated while the hardest real-world policies are structurally excluded?
Suggestion: The reviewer’s humble recommendation would be to provide a categorical taxonomy of what types of policies populate the UNKNOWN pool (e.g., residency rules, GPA floors, transfer caps) and report its distribution.
• Output Schema Brittleness Confounds Model Comparisons: The incident where GPT-6 scored 0.0% under the original strict two-space indentation contract (recoverable through a whitespacetolerant parser) may raise a concern that similar schema compliance issues may partially explain the extremely low scores of open-weight models (Gemma 2.3%–8.4% Exec. Recall). Suggestion: The reviewer believes that the authors may report a per-model schema compliance rate and conduct ablations with per-model schema-tuned parsers to isolate interface compliance from semantic reasoning deficits. • Limited Repetition for Non-Claude Models: It appears that only Sonnet 5 and Opus 5 received repeated executions for variance estimation. GPT-5.6, GPT-6, Qwen3.8-27B, and Gemma-3-27B-it ran only once per cell, meaning their reported numbers lack empirical noise floors. Suggestion: The revewier believes that if compute permits, at least one repeated run across all models may strengthen the breadth comparisons. • Lack of Chance-Corrected Agreement Statistics: Human annotation reliability is reported as raw percentage agreement (87.2%–92.1%) rather than chance-corrected measures such as Cohen’s κ or Krippendorff’s α. Suggestion: For a benchmark whose ground truth depends on expert adjudication, chance corrected statistics would be more informative. • Redistribution and Longevity Risk: The ethics statement acknowledges that per-institution redistribution authorization was not obtained for the frozen website snapshots that constitute the core benchmark artifact. Suggestion: The reviewer construes that the authors may proactively develop either a synthetic replica environment, a self-hosted mock institutional site, or an opt-in redistribution protocol to future-proof the benchmark against takedown requests. • Strict WORLDPASS Metric is Instrument-Bounded: The authors acknowledge that strict WORLDPASS is attainable in only 18 of 243 worlds due to reference-level credit-contract constraints, demoting it to a diagnostic. This limits headline interpretability. Suggestion: The reviewer opines that the authors should consider defining a graded closure metric (e.g., “closure at 90% obligation threshold”) as a middle-ground primary metric.
Questions for authors
See under weaknesses.