acceptodds
Desk rejected at ICLR 2027

HazardWeaver: Scientific Route Selection for Hazard Analysis Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding and assessing natural hazards is essential for disaster preparedness and risk reduction. Recent advances in large language models have spurred growing interest in AI agents for hazard analysis, particularly their ability to integrate scientific data, models, and tools into automated workflows. However, effective automation requires agents to determine which scientific methods are appropriate for a given event and executable with the available data and tools. As new evidence and execution results become available, these conditions can change, requiring agents to reconsider their choices. We formulate this problem as state-dependent scientific route selection and introduce HazardWeaver. Specifically, HazardWeaver first leverages the Hazard Knowledge Compiler to extract evidence-linked conditions governing scientific applicability, then its Hazard Capability Graph represents executable scientific capabilities and checks compatibility between their inputs and outputs. Using these complementary representations, the Hazard Weaver Agent component selects applicable and executable routes, carries out their workflows, and revises its decisions as the analysis state changes. To evaluate both the scientific outputs and the decisions that produce them, we introduce the Hazard Weaver Benchmark, comprising 143 instances across seven single-hazard domains and four multi-hazard interaction classes. The benchmark accommodates multiple valid scientific routes and evaluates output correctness, route validity, and justified abstention. Extensive experiments on this benchmark show that HazardWeaver outperforms existing agent systems, with the largest gains on tasks with multiple eligible scientific routes.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI agents, AI for science, natural hazards, scientific workflows, tool-using agents, multi-hazard analysis, scientific reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EPq4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper formulates state-dependent scientific route selection for natural-hazard analysis agents and introduces HazardWeaver together with the Hazard Weaver Benchmark (HWB). HazardWeaver comprises (1) the Hazard Knowledge Compiler (HKC), which extracts applicability conditions from scientific literature into structured Route Cards (SATISFIED, VIOLATED, UNRESOLVED), (2) the Hazard Capability Graph (HCG), which checks 8-field typed input/output compatibility and reachability across registered models, datasets, and adapters, and (3) the Hazard Weaver Agent (HWA), which selects and revises routes from the intersection of scientifically applicable and capability-reachable workflows (Rtelig\mathcal{R}_t^{\text{elig}}) or abstains when no valid route remains reachable within budget. HWB contains 141 sealed instances across 7 single-hazard domains and 4 multi-hazard cascades (45 single-route, 96 multi-route), scored by Decision-Constrained Accuracy (DCA), the product of artifact validity (EqE_q), benchmark route validity (VrouteBV_{\text{route}}^B), provenance (VprovV_{\text{prov}}), and execution/cutoff/budget validity (VexecV_{\text{exec}}). The paper reports that HazardWeaver achieves 89.4% DCA on Llama-3.3-70B compared with 41.1% for the best baseline (Plan-and-Solve), and transfers across five open-weight backbones (60.3%–89.4% DCA).

Critical strengths
  • Conceptual separation of scientific applicability and typed capability reachability: Formalizing scientific workflow selection as the intersection of literature-grounded applicability conditions (AsciA_{\text{sci}}) and typed artifact-transfer reachability (AcapA_{\text{cap}}), together with temporal cutoff enforcement (VcausalV_{\text{causal}}) and recovery-bounded abstention (VabsBV_{\text{abs}}^B), addresses a real problem in composing heterogeneous domain models.
Critical weaknesses
  • Severe experimental invalidity and hardcoded baseline handicapping in the released code (faithful_route_scoring_v1.py, registry.yaml, headline_g1_autoroute_v1.py):
    1. Tautological shared predicate + non-LLM autocommit in HWA: In Section 3.4 (Eq. 5) and Appendix B.1, the benchmark's route-validity grader Rtelig,B(q,sk)\mathcal{R}_t^{\text{elig},B}(q, s_k) applies the exact same HKC and HCG rules over the frozen Route Cards that HazardWeaver uses at test time. When HKC or HCG is ablated on the 96 multiple-route instances (Table F.1), Llama-3.3-70B drops from 93.8% to 45.8% (HKC off) and 47.9% (HCG untyped)—the same range as the baselines. Furthermore, inspecting the authors' repository (src/hazardweaver/hwa/experiments/headline_g1_autoroute_v1.py, enabled by default via HWA_HEADLINE_G1_AUTOCOMMIT=1 and invoked inside ScientificRouteController._apply_post_enumerate_autocommit) reveals that on the g1_v1 profile, HWA reads a pre-stored metadata["ds_cbr_route_id"] (or the top admissible route), auto-proposes, auto-commits, auto-executes, and auto-submits in Python without an LLM decision turn ("After enumerate: propose+commit+execute G1 route once (no LLM parse)" / "submit_solution without another LLM turn").
    2. Artificially crippled baselines: While Section 5.1 and Appendix E claim that the 10 baselines in Table 1 (AIDE, MLE-STAR, AutoML-Agent, R&D-Agent, DS-Agent, DisasterBench-ToT, Reflexion, Plan-and-Solve, Self-Consistency, ReAct) are faithful LLM agent adaptations, src/hazardweaver/baselines/registry.yaml marks the baseline adapters as status: FIXTURE_SMOKE, implementation: pipeline_only, eligible_live: false, and src/hazardweaver/baselines/system/shared/faithful_route_scoring_v1.py explicitly states: "Each policy uses a *different* public signal so Llama T≈0 cannot collapse Table 1 rows to one CAP." Inside policy_fallback_pick and the ranking helpers in that file:
      • AIDE is hardcoded to skip the top metric-ranked route and pick the second one (alts = [e for e in metric_rank if e != greedy ...]; return alts[0], commented "expand one node away from the metric-greedy idea");
      • ReAct is hardcoded to pick pair[1] (the second-ranked route) instead of pair[0];
      • R&D-Agent reverses the metric ranking when hypo_s == 0 ("opposite end of the metric order");
      • AutoML-Agent strips metric fields ("Metric fields are excluded on purpose") and breaks ties in reverse (tie_mode="edge_id_desc");
      • Plan-and-Execute (plan_first_step_score) weights (1.0 - idx / n) * 0.75, favoring whichever route is listed at index 0 of the JSON array; and DS-Agent returns candidates[0]. This renders the headline Table 1 comparison (89.4% vs. 22.7%–41.1%) and the McNemar p-values in Figure F.1 scientifically invalid.
  • Unreproducible agent execution, stub evaluation scripts, and internal data inconsistencies:
    1. The repository cited in the abstract (https://github.com/HazardWeaver2027/Hazard_Weaver) was completely empty at the time of submission through October 3, 2026, and was only populated on October 4, 2026 (01:33–02:10 UTC).
    2. In the code pushed on October 4, scripts/run_experiment.py is a non-functional stub that immediately exits with "error": "full_agent_rerun_not_packaged", and scripts/evaluate_runs.py (make evaluate-released) is a "Re-score stub" that performs no scientific or trajectory evaluation—it merely sums the pre-baked dca_valid boolean column in released_results/structured_trajectories/unified_141_per_instance_v1.jsonl and checks valid == 126.
    3. In that released per-instance file (unified_141_per_instance_v1.jsonl), artifact_valid is identical to dca_valid on all 141 rows (126/141 = 89.4%), contradicting Table 3 of the paper, which reports Llama-3.3-70B Artifact validity (E_q) = 92.2% (130/141) alongside DCA = 89.4%. Meanwhile, Table 1 never reports EqE_q (artifact correctness) separately from VrouteBV_{\text{route}}^B for any of the 10 baselines.
Questions for authors
  1. Baseline implementation in faithful_route_scoring_v1.py: Why does src/hazardweaver/baselines/system/shared/faithful_route_scoring_v1.py hardcode deterministic tie-breaking and ranking rules that explicitly force baselines like AIDE and ReAct to pick the second-ranked route (alts[0], pair[1]), force AutoML-Agent to exclude metric fields, and score Plan-and-Execute by array index (1 - idx/n)? How much of Table 1 reflects actual multi-turn LLM decisions versus these deterministic heuristics?
  2. Non-LLM autocommit in HazardWeaver (headline_g1_autoroute_v1.py and headline_hcg_bind_autocommit_v1.py): On how many of the 141 benchmark instances did apply_g1_autocommit_if_eligible or apply_hcg_bind_autocommit_if_eligible propose, commit, execute, and submit the route deterministically in Python (ds_cbr_route_id / routes[0]) without an LLM selection/submission turn?
  3. Discrepancy in released artifacts and missing runner: Why does scripts/run_experiment.py return "full_agent_rerun_not_packaged", why was the repository empty until October 4, 2026, and why does unified_141_per_instance_v1.jsonl record artifact_valid == dca_valid (126/141) when Table 3 reports Eq=92.2E_q = 92.2\\% (130/141)?
2 comments

Thank you for the detailed code inspection. We audited the sealed outputs behind every reported number; the findings and the corresponding revisions follow. All changes are marked in blue in the revised PDF.

Q1. Deterministic ranking rules in the baselines, and how much of Table 1 reflects LLM decisions.

Response:

Answer. The deterministic signals changed none of the reported results: scoring each adapter's own choice before signal combination gives the same DCA as Table 1 for all ten methods.

Evidence.

  1. On every instance with a recorded model choice, the submitted route equals the model's own choice, except four AIDE instances where the last refinement round failed JSON parsing and the submitted route stayed on the previous candidate; all four fail under either route.
  2. Every logged baseline decision records a live Llama-3.3-70B call, and the parse-failure fallback fired at most twice per method. The second-ranked selections you cite sit in this fallback.
  3. The deterministic rules alone reach 7/141 for both AIDE and ReAct, against 32/141 and 42/141 in Table 1.

Each signal encodes a decision cue of the source method on public route metadata, such as plan-step order for Plan-and-Solve and metric fit for MLE-STAR. The registry status labels you cite do not describe the reported runs.

Revision. Appendix E documents the signals, their weights, and the fallback rule, and Table E.2 reports, per method, the DCA of the model's own choice.

Q2. Auto-commit in HazardWeaver.

Response:

Answer. None: 0 of 141 headline instances triggered either auto-commit path.

Evidence. The G1 path requires the g1_v1 route profile, an internal experiment excluded from the paper, and no headline run uses it. The HCG-bind path requires near-miss mechanism metadata that no headline instance carries. No instance metadata, prompt, transcript, or tool log contains ds_cbr_route_id. All 141 instances have LLM transcripts, including all 126 successes.

Revision. Appendix I states the headline run configuration. The release sets both auto-commit flags to off by default.

Q3. Runner, repository timeline, and the JSONL discrepancy.

Response:

Answer. Table 3 is correct, and the updated release adds a runnable entry point and a corrected per-instance file.

Evidence. The repository was populated on October 4 with precomputed per-instance records and a runner entry point that the updated release completes. In the per-instance file, the artifact_valid column was exported from the DCA field. Recomputing from the sealed outputs gives DCA = 126/141, E_q = 130/141, and V_q = 134/141, matching Table 3.

Revision. The updated release contains the runnable entry point for the headline configuration, an evaluator that rescores from raw trajectories, and the corrected per-instance file.

W. Shared criteria between method and evaluator.

Response:

Answer. A fixed-order control without a language model on the same harness reaches 48/141, against 126/141 for HazardWeaver, so the result depends on the model's decisions within the shared criteria.

Evidence. The criteria were frozen before all runs, checked by a stratified human review of 86 route–condition pairs (Appendix G.3), and validated on 160 controlled fixtures (Appendix G.1).

Revision. New Appendix F.5 (Table F.2) reports the no-LLM controls.

Ethics concerns: anonymity.

Response:

Answer. Two auxiliary verification scripts in the released code contained hard-coded cluster paths, and the updated release removes them.

Evidence. We found the paths only in these two path-checking utilities, which sit outside the method and evaluation code. We have reported the issue to the AC and will follow the AC's guidance on any further remediation of the repository.

Ethics concerns: research integrity.

Response:

Answer. All reported numbers come from live runs, as documented in our responses to Q1–Q3 above.

Reviewer_EPq4OpenReview ↗

Response to the rebuttal

Thank you for the response and repository update (438cb06). Inspecting the revised PDF and updated code leaves the core validity and reproducibility issues unresolved and surfaces additional ones:

  1. Deterministic baseline lock-in (Table E.2): Table E.2 discloses a 0.85 weight on the deterministic plan-step score (llm_weight = 0.15) in blend_ranked_order (faithful_route_scoring_v1.py) for Plan-and-Solve. Evaluated across all 96 multi-route instances in the released benchmark/public/stratum_M.jsonl and taskpacks, 0.85 * 0.75 / n strictly exceeds the maximum possible LLM swing 0.15 * (n - 1) / n for all n∈{2,3,4}n \in \{2, 3, 4\}, locking Plan-and-Solve to allowed_edge_ids[0] on 96/96 instances regardless of the LLM's ranking (32/96 where gold is at index 0, plus 26/45 single-route = 58/141). Neither baseline runner scripts nor baseline transcripts are included to verify the claimed unblended choices.

  2. Single-gold grading and gold-metadata leakage in Full HWA: Contrary to the claim that multiple valid routes are credited, all 96/96 multi_eligible contracts in benchmark/sealed_eval/evaluator_contracts/ set route_gold_required: true with a single gold_capability_id (len(accepted_witness_capability_ids) == 1), and 0/141 instances have abstention as the valid outcome (Appendix A.4). Moreover, apply_unified_benchmark_overlays (unified_benchmark_execution_v1.py) stamps unified_scenario_gold_capability_id (76/96 multi-route rows) and ablation_witness_capability_id (96/96 rows) onto solver task["metadata"], which remsa_rank.py boosts (+0.40 / +0.12 in validation_utility during controller_enumerate_routes) and submit_solution (unified_tools.py) uses via maybe_redirect_unified_submit_to_gold_registry.

  3. Non-runnable release and missing trajectories: scripts/run_experiment.py (still cited in docs/reproduction.md) exits with "full_agent_rerun_not_packaged", scripts/run_headline_benchmark.py crashes on import due to missing subpackages (data_adapters, hazardweaver.hcg.runtime, hazardweaver.hwb.run) and an invalid default arm (llama70b vs. llama70b_b200), and no raw trajectories are included—evaluate_runs.py only checks pre-populated booleans in unified_141_per_instance_v1.jsonl.

I maintain my score of 1.

Reviewer_Mu8nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies scientific route selection for hazard analysis as event evidence, data, and tool availability change. The system compiles method applicability conditions from the literature, checks whether scientific capabilities can be composed through a typed graph, and selects, executes, and revises the resulting workflows. The proposed benchmark contains 141 instances across seven single-hazard domains and four multi-hazard interaction classes, evaluating scientific artifacts, route validity, and justified abstention. With Llama-3.3-70B, the complete system achieves 89.4% overall decision-constrained accuracy, compared with 41.1% for the strongest adapted baseline. Component ablations and cross-backbone experiments further examine the proposed mechanisms.

Critical strengths
  1. The problem setting is practically meaningful. The same hazard-analysis target can be addressed through multiple scientific routes, whose applicability and executability change with event evidence and tool availability. Bringing route selection, revision, and justified abstention into a common framework addresses an important challenge for agents that must adapt their analyses to changing scientific conditions.

  2. The benchmark has broad coverage and useful evaluation principles. It draws on eleven public scientific datasets, operational products, and existing benchmarks, covering both individual hazards and multi-hazard interactions. Assessing route validity, information cutoffs, and the justification for abstention alongside final artifacts provides a valuable basis for checking whether the process supporting an analysis is trustworthy.

Critical weaknesses
  1. Missing comparisons with frontier models and complete general-purpose agents limit the evidence for the specialized architecture. The main baseline comparison uses only Llama-3.3-70B. Although system and ablation results are reported for other backbones, the paper does not compare the complete system, key ablations, ReAct, and Plan-and-Solve on representative frontier Claude or GPT models. It therefore remains unclear whether the benefits of explicit route constraints persist with stronger model-based planning and reflection. Comparisons with complete agents such as Claude Code and Codex are also missing, leaving the additional value of the specialized architecture insufficiently established. It is further unclear whether the adapted baselines have equivalent access to information requests, workflow repair, route revision, and abstention, with matched recovery budgets.

  2. The evaluation closely mirrors the encoded rules, leaving the additional contribution of scientific route decisions insufficiently validated. Method-side route filtering and benchmark-side validity assessment use the same types of applicability and compatibility criteria, while scientific conditions, capabilities, and candidate routes are organized in advance. The results support explicit constraint enforcement, but do not sufficiently separate the contributions of rule-based filtering and language-model decisions. Two critical controls are missing: random selection from the system's own computed eligible set, and an executor using the same rules and execution mechanisms without language-model route decisions. The paper also needs to establish the size and nature of the scientific decision space remaining after filtering, to clarify how much the benchmark captures beyond executing predefined constraints.

Questions for authors
  1. Are matched comparisons available on frontier Claude or GPT models and with complete agents such as Claude Code and Codex? Does the proposed architecture retain its advantage with equivalent scientific information, tool permissions, action and recovery budgets, and evaluator-only information withheld?

  2. Are results available for random selection from the system's computed eligible set and an executor without language-model route decisions? With the same knowledge, state updates, execution and recovery mechanisms, and budgets, how much additional benefit comes from language-model decisions?

  3. What are the per-instance distributions of candidate and eligible routes, and the proportion of instances requiring abstention? Can an existing instance illustrate which consequential scientific judgments remain for the language model after the encoded rules have been applied?

2 comments

Thank you for the review. All changes are marked in blue in the revised PDF.

Q1a. Frontier Claude or GPT backbones.

Response:

Answer. Yes for the complete system: with GPT-6 Luna as the backbone, HazardWeaver reaches 123/141 (87.2%) DCA on the same 141 instances.

Evidence. The run uses the same controller, capability catalog, evaluator, and seed as the headline run, and solves 79/90 single-hazard and 44/51 multi-hazard instances (Wilson 95% interval [80.7, 91.8]). Across the five backbones in Table 3, removing HKC conditions or HCG typing lowers DCA for every backbone (Figure 3).

Revision. Appendix D reports the GPT-6 Luna run.

Q1b. Matched information, tools, recovery, and budgets.

Response:

Answer. Within HazardWeaver's executor, each ablation removes one constraint layer and keeps everything else fixed, and DCA drops for every backbone.

Evidence. The HKC-off, HCG-untyped, and fixed-eligibility variants share the controller executor, budgets, evaluator, and withheld evaluator-only information (Appendix G.2); they differ only in the component under test. On the 96 multiple-route instances, Llama falls from 93.8% to 45.8% and 47.9% (Figure 3, Table F.1). Disabling route updating lowers pooled DCA from 41.5% to 16.0% on the 40 instances where eligibility changes (Table 2). Among the Table 1 baselines, which commit one terminal route per instance (Appendix E), Reflexion revises its choice for up to three rounds from execution feedback and reaches 31.9%.

Revision. Section 5.2 states that the ablations share the executor, budgets, and evaluator. Appendix E states the single-commitment protocol of the adapted baselines.

Q1c. Complete coding agents.

Response:

Answer. In a one-instance-per-track pilot, Codex (GPT-6 Luna) given only the task and its data returned a prediction on 7 of 11 instances, and two matched the reference under substitute scoring; HazardWeaver passes 10 of the same 11.

Evidence. The agent had no capability catalog, route summaries, evaluator, or reference. The matches are drought outlook (normalized ordinal skill) and tropical-cyclone trajectory (midpoint window). Four predictions fell outside tolerance (dynamic flood inundation, rainfall-triggered landslide, coastal compound flooding, active-fire spread), and four instances produced none (medium-range heatwave, post-fire debris-flow cascade, earthquake-triggered landslide, multi-peril impact). Seismic phase picking scores 1.0 only on three labeled development anchors and is excluded. The pilot is a single run without Claude Code.

W2 / Q2. Separating rule-based filtering from language-model decisions.

Response:

Answer. Language-model decisions add 78 to 95 solved instances beyond two no-LLM controls: the controls reach 31/141 and 48/141 DCA, against 126/141 for HazardWeaver.

Evidence. Both controls use the same 141 instances, capability bindings, execution stack, budget, and evaluator, without a language model.

SettingRoute decisionDCA
Uniform controluniform choice among the instance's candidate routes31/141
Fixed-order controlcandidate routes tried in a fixed utility order until success or budget exhaustion48/141
HazardWeaver (Llama-3.3-70B)language-model decisions within the eligible set126/141

The uniform control samples from the candidate routes, which contain the eligible set. The shared criteria were frozen before all runs, checked by a stratified human review of 86 route–condition pairs (Appendix G.3), and validated on 160 controlled fixtures (Appendix G.1).

Revision New Appendix F.5 (Table F.2) reports both controls; Section 5.2 summarizes the result.

Q3. Decision space, abstention, and a concrete example.

Response:

Answer. Each multiple-route instance offers 2 to 4 candidate routes (median 3), of which 1 to 4 (median 2) pass the applicability check; abstention is the benchmark-valid outcome in none of the 141 instances.

Evidence. Across all 141 instances, candidate counts range from 1 to 4 (median 2), and the 45 single-route instances have one candidate each. Every instance has a construction-time workflow that produces the required artifact (Appendix A.3), so every instance is solvable at the initial state.

Example. In a seismic phase-picking instance, several routes with different method chains and output contracts remain after filtering. The model compares them against the phase-picking requirement and commits to a route that satisfies it, and the artifact passes DCA. Appendix H.2 shows the failure that follows when a post-fire analysis commits to a route covering only the volume stage.

Revision. New Appendix A.4 (Table A.2) gives per-track distributions of candidate and applicable routes.

Thank you for the additional experiments and clarifications. Two distinct concerns about the comparisons remain insufficiently addressed.

  1. Does the architectural advantage persist with a strong backbone? The original paper compares HazardWeaver with ReAct and Plan-and-Solve using Llama-3.3-70B. However, the added GPT-6 Luna experiment reports only the complete HazardWeaver system, without corresponding baseline results using the same backbone. It therefore remains unclear whether stronger model-based planning and reflection would narrow the performance gap. Please provide a matched comparison of HazardWeaver, ReAct, and Plan-and-Solve using a representative flagship-capability model, such as GPT-5.6 Sol.

  2. How much does the complete HazardWeaver agent outperform a complete general-purpose agent such as Codex? The Codex is relevant preliminary evidence, but one instance per track and a single run provide limited support for estimating this advantage. A broader end-to-end comparison against Codex with a strong model configuration is needed to establish the practical value of the specialized system.

Both comparisons should provide comparable scientific information and tool access, matched action and recovery budgets, and a common evaluation protocol.

Reviewer_iHiiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates automated natural-hazard analysis as a state-dependent scientific route selection problem and introduces HazardWeaver, an agent framework that determines which scientific workflows are both theoretically applicable and computationally executable as an event unfolds. The architecture comprises three core modules: the Hazard Knowledge Compiler, which extracts evidence-linked applicability conditions from scientific literature; the Hazard Capability Graph, which models tools, datasets, and simulators as a typed directed graph to enforce input-output compatibility; and the Hazard Weaver Agent, which selects, executes, and dynamically revises routes as intermediate results or failures alter the analysis state. To evaluate the system, the authors introduce the Hazard Weaver Benchmark, a 141-instance benchmark spanning seven single-hazard domains and four multi-hazard interaction classes, evaluated via Decision-Constrained Accuracy, which is a metric that jointly requires output artifact correctness, route validity, provenance, temporal causality, and justified abstention.

Critical strengths
  1. Well-Motivated Problem Formulation Grounded in Scientific Reality
  2. Principled Separation of Domain Applicability and Typed Execution
  3. High-Quality Multi-Hazard BenchmarkGrounded in Real Operational Products
  4. Comprehensive Empirical Validation and Clear Ablation Studies
Critical weaknesses
  1. Insufficient Technical Detail on the Hazard Knowledge Compiler (HKC) In the Abstract and Section 3.1, the HKC is presented as an active pipeline that extracts, canonicalizes, and grounds applicability conditions from raw scientific papers, reports, and guidelines. However, Section 3.1 is very brief and does not introduce the concrete NLP/LLM extraction prompts, schema definitions, entity canonicalization algorithms, and conflict-resolution strategies used when papers disagree, making it hard to understand the method.

  2. Potential Architectural Asymmetry in Baseline Adaptations In Table 1, HazardWeaver outperforms the best baseline by nearly 50 percentage points (89.4% vs. 41.1%). As explained in Appendix E, the adapted baselines (such as AIDE, DS-Agent, MLE-STAR, and Plan-and-Solve) appear to be configured primarily to inspect candidate Route Cards, rank or refine them for a few rounds, and commit to a route upfront. It is unclear whether these baselines were given direct access to the pre-computed symbolic outputs of A_sci (from HKC) and A_cap (from HCG), or if they had to infer applicability and 8-tuple port compatibility purely via raw LLM prompting over text descriptions.

  3. Closed-World Assumption Limits Open-Ended Scientific Discovery As acknowledged briefly in Appendix J, HazardWeaver considers a sealed, pre-registered catalog of 82 capability specifications and pre-compiled Route Cards. Although this ensures high reproducibility, it reduces the task to discrete symbolic graph routing and configuration selection rather than open-ended scientific problem solving. Unlike code-generating agents (such as AIDE) that can write custom data-wrangling scripts or novel numerical adapters on the fly, HazardWeaver cannot synthesize a new adapter if a needed transformation is missing from the 82 pre-registered capabilities in HCG.

Questions for authors

See the comments above.

1 comment

Thank you for the review. All changes are marked in blue in the revised PDF.

W1. Insufficient technical detail on the Hazard Knowledge Compiler (HKC).

Response to W1:

Answer. HKC compiles conditions offline in two stages, LLM-assisted extraction from the literature followed by verification and freezing; at evaluation time it performs no extraction and resolves the frozen conditions deterministically.

Evidence.

  1. Extraction. HKC processes about 28,000 full-text papers. Llama-3.3-70B proposes route-relevant slots under a strict JSON prompt that requires an exact evidence quote for every slot and instructs the model to omit unsupported slots. Each slot is grounded against its quoted span and admitted only above a fixed threshold (τ = 0.55).
  2. Schema. Each benchmark condition records an identifier, a checkable claim, the context fields it reads, an optional workspace check (field, operator, expected value), and the action taken when context is missing or mismatched.
  3. Canonicalization and conflicts. The same backbone drafts the benchmark conditions from the admitted evidence. Automatic schema checks enforce a field whitelist and consistent handles across routes; the authors then merge equivalent conditions and resolve conflicting sources before freezing, keeping the source row of every condition. No model arbitrates evidence during evaluation.
  4. Evaluation-time use. The frozen registry holds 44 conditions over 22 capabilities and is reused across instances. Each condition is resolved by a deterministic check against the instance context (Eq. 2). A stratified human review of 86 route–condition pairs supports the registry (Appendix G.3).

Revision. New Appendix B.4 gives the extraction prompt, admission rule, condition schema, canonicalization and conflict procedure, and registry statistics.

W2. Whether baselines received the outputs of A_sci and A_cap.

Response to W2:

Answer. No. Baselines never received the resolved A_sci or A_cap outputs; they judged applicability and compatibility from text.

Evidence. Each baseline received the task context, a natural-language summary of every candidate route, the candidate capability identifiers, and the public output contract. Resolved applicability statuses and typed-port compatibility checks exist only inside HazardWeaver's controller. The RQ2 ablations isolate this information: with the same backbone and executor, removing HKC conditions or HCG typing lowers Llama's DCA on the 96 multiple-route instances from 93.8% to 45.8% and 47.9% (Figure 3).

Revision. Appendix E lists what each baseline receives and states that resolved A_sci and A_cap outputs are withheld from all baselines.

W3. The closed-world assumption limits open-ended discovery.

Response to W3:

Answer. The registered catalog is a deliberate design choice that keeps every artifact auditable and every route verifiable, and it is extensible.

Evidence. Every artifact traces to a registered, validated capability (Eq. 3 and provenance validity), while a capability written at run time carries no validation record. The benchmark registers every valid route and credits any of them (Eq. 5). Extension requires registering a capability's typed ports and validation metadata, after which HCG composes it with existing nodes; the Repair workflow operation (Table B.2) already inserts registered transformations during execution.

Revision. Appendix J describes how a synthesized adapter could enter HCG once it passes the same type and validation checks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.