- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper formulates state-dependent scientific route selection for natural-hazard analysis agents and introduces HazardWeaver together with the Hazard Weaver Benchmark (HWB). HazardWeaver comprises (1) the Hazard Knowledge Compiler (HKC), which extracts applicability conditions from scientific literature into structured Route Cards (SATISFIED, VIOLATED, UNRESOLVED), (2) the Hazard Capability Graph (HCG), which checks 8-field typed input/output compatibility and reachability across registered models, datasets, and adapters, and (3) the Hazard Weaver Agent (HWA), which selects and revises routes from the intersection of scientifically applicable and capability-reachable workflows () or abstains when no valid route remains reachable within budget. HWB contains 141 sealed instances across 7 single-hazard domains and 4 multi-hazard cascades (45 single-route, 96 multi-route), scored by Decision-Constrained Accuracy (DCA), the product of artifact validity (), benchmark route validity (), provenance (), and execution/cutoff/budget validity (). The paper reports that HazardWeaver achieves 89.4% DCA on Llama-3.3-70B compared with 41.1% for the best baseline (Plan-and-Solve), and transfers across five open-weight backbones (60.3%–89.4% DCA).
Critical strengths
- Conceptual separation of scientific applicability and typed capability reachability: Formalizing scientific workflow selection as the intersection of literature-grounded applicability conditions () and typed artifact-transfer reachability (), together with temporal cutoff enforcement () and recovery-bounded abstention (), addresses a real problem in composing heterogeneous domain models.
Critical weaknesses
- Severe experimental invalidity and hardcoded baseline handicapping in the released code (
faithful_route_scoring_v1.py,registry.yaml,headline_g1_autoroute_v1.py):- Tautological shared predicate + non-LLM autocommit in HWA: In Section 3.4 (Eq. 5) and Appendix B.1, the benchmark's route-validity grader applies the exact same HKC and HCG rules over the frozen Route Cards that HazardWeaver uses at test time. When HKC or HCG is ablated on the 96 multiple-route instances (Table F.1), Llama-3.3-70B drops from 93.8% to 45.8% (
HKC off) and 47.9% (HCG untyped)—the same range as the baselines. Furthermore, inspecting the authors' repository (src/hazardweaver/hwa/experiments/headline_g1_autoroute_v1.py, enabled by default viaHWA_HEADLINE_G1_AUTOCOMMIT=1and invoked insideScientificRouteController._apply_post_enumerate_autocommit) reveals that on theg1_v1profile, HWA reads a pre-storedmetadata["ds_cbr_route_id"](or the top admissible route), auto-proposes, auto-commits, auto-executes, and auto-submits in Python without an LLM decision turn ("After enumerate: propose+commit+execute G1 route once (no LLM parse)"/"submit_solution without another LLM turn"). - Artificially crippled baselines: While Section 5.1 and Appendix E claim that the 10 baselines in Table 1 (
AIDE,MLE-STAR,AutoML-Agent,R&D-Agent,DS-Agent,DisasterBench-ToT,Reflexion,Plan-and-Solve,Self-Consistency,ReAct) are faithful LLM agent adaptations,src/hazardweaver/baselines/registry.yamlmarks the baseline adapters asstatus: FIXTURE_SMOKE,implementation: pipeline_only,eligible_live: false, andsrc/hazardweaver/baselines/system/shared/faithful_route_scoring_v1.pyexplicitly states: "Each policy uses a *different* public signal so Llama T≈0 cannot collapse Table 1 rows to one CAP." Insidepolicy_fallback_pickand the ranking helpers in that file:AIDEis hardcoded to skip the top metric-ranked route and pick the second one (alts = [e for e in metric_rank if e != greedy ...]; return alts[0], commented "expand one node away from the metric-greedy idea");ReActis hardcoded to pickpair[1](the second-ranked route) instead ofpair[0];R&D-Agentreverses the metric ranking whenhypo_s == 0("opposite end of the metric order");AutoML-Agentstrips metric fields ("Metric fields are excluded on purpose") and breaks ties in reverse (tie_mode="edge_id_desc");Plan-and-Execute(plan_first_step_score) weights(1.0 - idx / n) * 0.75, favoring whichever route is listed at index0of the JSON array; andDS-Agentreturnscandidates[0]. This renders the headline Table 1 comparison (89.4% vs. 22.7%–41.1%) and the McNemar p-values in Figure F.1 scientifically invalid.
- Tautological shared predicate + non-LLM autocommit in HWA: In Section 3.4 (Eq. 5) and Appendix B.1, the benchmark's route-validity grader applies the exact same HKC and HCG rules over the frozen Route Cards that HazardWeaver uses at test time. When HKC or HCG is ablated on the 96 multiple-route instances (Table F.1), Llama-3.3-70B drops from 93.8% to 45.8% (
- Unreproducible agent execution, stub evaluation scripts, and internal data inconsistencies:
- The repository cited in the abstract (
https://github.com/HazardWeaver2027/Hazard_Weaver) was completely empty at the time of submission through October 3, 2026, and was only populated on October 4, 2026 (01:33–02:10 UTC). - In the code pushed on October 4,
scripts/run_experiment.pyis a non-functional stub that immediately exits with"error": "full_agent_rerun_not_packaged", andscripts/evaluate_runs.py(make evaluate-released) is a"Re-score stub"that performs no scientific or trajectory evaluation—it merely sums the pre-bakeddca_validboolean column inreleased_results/structured_trajectories/unified_141_per_instance_v1.jsonland checksvalid == 126. - In that released per-instance file (
unified_141_per_instance_v1.jsonl),artifact_validis identical todca_validon all 141 rows (126/141 = 89.4%), contradicting Table 3 of the paper, which reports Llama-3.3-70BArtifact validity (E_q) = 92.2%(130/141) alongsideDCA = 89.4%. Meanwhile, Table 1 never reports (artifact correctness) separately from for any of the 10 baselines.
- The repository cited in the abstract (
Questions for authors
- Baseline implementation in
faithful_route_scoring_v1.py: Why doessrc/hazardweaver/baselines/system/shared/faithful_route_scoring_v1.pyhardcode deterministic tie-breaking and ranking rules that explicitly force baselines likeAIDEandReActto pick the second-ranked route (alts[0],pair[1]), forceAutoML-Agentto exclude metric fields, and scorePlan-and-Executeby array index(1 - idx/n)? How much of Table 1 reflects actual multi-turn LLM decisions versus these deterministic heuristics? - Non-LLM autocommit in HazardWeaver (
headline_g1_autoroute_v1.pyandheadline_hcg_bind_autocommit_v1.py): On how many of the 141 benchmark instances didapply_g1_autocommit_if_eligibleorapply_hcg_bind_autocommit_if_eligiblepropose, commit, execute, and submit the route deterministically in Python (ds_cbr_route_id/routes[0]) without an LLM selection/submission turn? - Discrepancy in released artifacts and missing runner: Why does
scripts/run_experiment.pyreturn"full_agent_rerun_not_packaged", why was the repository empty until October 4, 2026, and why doesunified_141_per_instance_v1.jsonlrecordartifact_valid == dca_valid(126/141) when Table 3 reports (130/141)?
2 comments
Response to the rebuttal
Thank you for the response and repository update (438cb06). Inspecting the revised PDF and updated code leaves the core validity and reproducibility issues unresolved and surfaces additional ones:
-
Deterministic baseline lock-in (
Table E.2): Table E.2 discloses a0.85weight on the deterministicplan-step score(llm_weight = 0.15) inblend_ranked_order(faithful_route_scoring_v1.py) forPlan-and-Solve. Evaluated across all96multi-route instances in the releasedbenchmark/public/stratum_M.jsonland taskpacks,0.85 * 0.75 / nstrictly exceeds the maximum possible LLM swing0.15 * (n - 1) / nfor all , lockingPlan-and-Solvetoallowed_edge_ids[0]on 96/96 instances regardless of the LLM's ranking (32/96where gold is at index0, plus26/45single-route =58/141). Neither baseline runner scripts nor baseline transcripts are included to verify the claimed unblended choices. -
Single-gold grading and gold-metadata leakage in
Full HWA: Contrary to the claim that multiple valid routes are credited, all96/96multi_eligiblecontracts inbenchmark/sealed_eval/evaluator_contracts/setroute_gold_required: truewith a singlegold_capability_id(len(accepted_witness_capability_ids) == 1), and0/141instances have abstention as the valid outcome (Appendix A.4). Moreover,apply_unified_benchmark_overlays(unified_benchmark_execution_v1.py) stampsunified_scenario_gold_capability_id(76/96multi-route rows) andablation_witness_capability_id(96/96rows) onto solvertask["metadata"], whichremsa_rank.pyboosts (+0.40/+0.12invalidation_utilityduringcontroller_enumerate_routes) andsubmit_solution(unified_tools.py) uses viamaybe_redirect_unified_submit_to_gold_registry. -
Non-runnable release and missing trajectories:
scripts/run_experiment.py(still cited indocs/reproduction.md) exits with"full_agent_rerun_not_packaged",scripts/run_headline_benchmark.pycrashes on import due to missing subpackages (data_adapters,hazardweaver.hcg.runtime,hazardweaver.hwb.run) and an invalid default arm (llama70bvs.llama70b_b200), and no raw trajectories are included—evaluate_runs.pyonly checks pre-populated booleans inunified_141_per_instance_v1.jsonl.
I maintain my score of 1.
Thank you for the detailed code inspection. We audited the sealed outputs behind every reported number; the findings and the corresponding revisions follow. All changes are marked in blue in the revised PDF.
Response:
Answer. The deterministic signals changed none of the reported results: scoring each adapter's own choice before signal combination gives the same DCA as Table 1 for all ten methods.
Evidence.
Each signal encodes a decision cue of the source method on public route metadata, such as plan-step order for Plan-and-Solve and metric fit for MLE-STAR. The registry status labels you cite do not describe the reported runs.
Revision. Appendix E documents the signals, their weights, and the fallback rule, and Table E.2 reports, per method, the DCA of the model's own choice.
Response:
Answer. None: 0 of 141 headline instances triggered either auto-commit path.
Evidence. The G1 path requires the g1_v1 route profile, an internal experiment excluded from the paper, and no headline run uses it. The HCG-bind path requires near-miss mechanism metadata that no headline instance carries. No instance metadata, prompt, transcript, or tool log contains ds_cbr_route_id. All 141 instances have LLM transcripts, including all 126 successes.
Revision. Appendix I states the headline run configuration. The release sets both auto-commit flags to off by default.
Response:
Answer. Table 3 is correct, and the updated release adds a runnable entry point and a corrected per-instance file.
Evidence. The repository was populated on October 4 with precomputed per-instance records and a runner entry point that the updated release completes. In the per-instance file, the artifact_valid column was exported from the DCA field. Recomputing from the sealed outputs gives DCA = 126/141, E_q = 130/141, and V_q = 134/141, matching Table 3.
Revision. The updated release contains the runnable entry point for the headline configuration, an evaluator that rescores from raw trajectories, and the corrected per-instance file.
Response:
Answer. A fixed-order control without a language model on the same harness reaches 48/141, against 126/141 for HazardWeaver, so the result depends on the model's decisions within the shared criteria.
Evidence. The criteria were frozen before all runs, checked by a stratified human review of 86 route–condition pairs (Appendix G.3), and validated on 160 controlled fixtures (Appendix G.1).
Revision. New Appendix F.5 (Table F.2) reports the no-LLM controls.
Response:
Answer. Two auxiliary verification scripts in the released code contained hard-coded cluster paths, and the updated release removes them.
Evidence. We found the paths only in these two path-checking utilities, which sit outside the method and evaluation code. We have reported the issue to the AC and will follow the AC's guidance on any further remediation of the repository.
Response:
Answer. All reported numbers come from live runs, as documented in our responses to Q1–Q3 above.