acceptodds
Under review as a conference paper at ICLR 2027

Can AI Scientists Coordinate at Runtime?

Abstract

Multi-agent AI Scientists have shown improving performance across a diverse range of different tasks. Yet a common approach is design-time agentic orchestration, which typically relies on fixed workflows. In contrast, human scientists coordinate and adjust their division of labor at runtime. We therefore ask: can AI Scientists also coordinate at runtime? To this end, we introduce Runtime Agent Coordination (\rac), which selects agents from existing AI Scientist hosts during execution, assigns scoped work contracts, and provides artifact-grounded verification. Verification informs subsequent agents without blocking transitions or discarding artifacts. We evaluate one shared implementation across ARK, Agent Laboratory, and EvoScientist on ResearchClawBench under matched models, tools, permissions, and run-level budgets. Four cumulative conditions separate native execution, runtime communication, runtime selection, and the combined addition of contracts and verification. Our results highlight the potential of runtime coordination to improve research quality, while revealing host-dependent trade-offs between coordination overhead and performance under matched budgets.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Multi-agent systems, AI scientists, Runtime coordination, Agentic workflows, Large language models, Automated scientific discovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_kMvcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies an iterative way of using AI-scientist systems, called Runtime Agent Coordination (RAC). It allows existing agents to choose the next agent based on the current research state, with task contracts and output verification. The authors evaluate RAC on three AI-scientist systems.

Critical strengths

Studying cooperation among agents in AI-scientist systems is a meaningful direction. Research tasks often require several rounds of work, and understanding how agents should coordinate could help improve these systems.

Critical weaknesses
  1. The contribution beyond existing coordination methods is not clear enough. The experiments compare RAC with each system’s native workflow and RAC’s own variants. They do not directly compare it with other established multi-agent coordination methods applied to the same AI-scientist systems.

  2. The RAC framework does not consistently improve performance across the three systems. For some systems, estimated costs increase substantially while performance improves only slightly or remains almost unchanged.

Questions for authors
  1. What does RAC add beyond existing dynamic delegation methods?

  2. Is there a direct comparison with another coordination method under the same experimental setup?

2 comments

Thank you for raising novelty, external baselines and cost-effectiveness.

Contribution relative to dynamic delegation. RAC provides a shared coordination intervention across existing AI-scientist hosts and an experimental separation of communication from successor selection. R1 adds communication with the native successor; R2 changes selection under the same communication; R3 adds contracts and advisory verification. Keeping native capabilities, tools and permissions fixed within each host makes the R1–R2 contrast informative about changing who acts next.

AutoGen supports flexible multi-agent conversations, Magentic-One orchestrates specialists with progress tracking and replanning, and MetaGPT organizes collaboration around operating procedures. RAC builds on these broad coordination ideas. Its contribution is the cross-host intervention and comparison, rather than any one delegation or feedback component in isolation.

External baseline. We have not completed a comparison with an external coordinator under the same host, tasks, model and budget. This remains a gap. The current experiments answer whether changing coordination improves each host relative to its native workflow and communication-only condition; they do not establish superiority over other coordination methods.

Additional executions. With GPT-6 Luna (medium), R2 again exceeds R1 on average in all three hosts. The additional study covers 10 tasks each for Agent Laboratory (AL) and EvoScientist (Evo), and five for ARK. Scores are on a 0–100 scale.

HostN0R1R2R3R2−R1
AL3.9456.7456.8955.640+0.150
Evo10.9957.2158.3606.920+1.145
ARK8.2309.86012.4309.870+2.570

R2 beats R1 on 5/10, 6/10 and 4/5 tasks, respectively. For Δt=sR2,t−sR1,t\Delta_t=s_{R2,t}-s_{R1,t}, 100,000 within-host task-pair bootstrap resamples (seed 20261117) give 95% percentile intervals of [−2.015, 2.450], [−2.410, 4.860] and [−0.790, 5.840]. All include zero. The positive mean direction recurs across the original DeepSeek V4 Pro and additional Luna cohorts, but each additional cell has one selected execution; these intervals describe task-mix sensitivity, not fixed-model execution repeatability. Each score is the median of three judge evaluations.

Component ablations. We added contracts-only (C) and verification-only (V) conditions. Means are:

HostR2R2+CR2+VR3
AL, 10 tasks6.8958.2306.4505.640
Evo, 9 common tasks9.0678.7335.3337.350
ARK, all 5 tasks12.43011.32010.4109.870

Evo Math_003 R2+C exhausted its budget without a report; it is missing, not zero. The complete ARK ablations reinforce that neither added component improves R2's aggregate mean. Verifier-enabled error/completion handling also differs, so the ablations compare implemented component variants.

Resource use and realized routing. For the additional ten-task AL cohort, R1/R2/R3 mean input usage is 0.810/0.740/1.011M tokens, output usage is 74.63/76.60/69.04K, and wall time is 39.69/34.81/27.08 minutes. Evo's corresponding values are 2.232/2.331/2.263M, 35.34/34.38/36.42K and 15.14/15.90/15.42 minutes. The small R2−R1 gain accompanies lower input usage/runtime in AL but slightly higher usage/runtime in Evo. The resource benefit is therefore host-dependent. Subscription billing is unavailable; original dollar estimates use a standardized token tariff.

Routing needs both an actionable issue and an opportunity to act. AL R2 follows the native phase sequence, and none of the 30 AL R1–R3 traces selects a capability after report refinement. In ARK Physics_002, all 11 R1 post-review handoffs go to the planner, while R2 includes three direct handoffs to the coder. After one, the hop-18 coder reported parsing 4,610 JSON inputs, producing results for 550 eligible XEB instances and regenerating figures, while retaining unresolved scientific gaps. This shows how runtime selection can send a concrete review issue directly to a role that performs the corrective work.

The R3 cases further distinguish local checks from useful correction. AL Physics_002 V-only receives seven supported verdicts despite reporting no empirical estimates and using synthetic plotting demonstrations. ARK Neuroscience_002 R3 repeatedly records a writer read-access blocker rather than revising the manuscript. In these cases, local verification passed without target measurements, or the contract prevented a needed revision. Adding checks therefore did not ensure completion of the underlying research task.

Thank you for the response. However, my main concerns remain unresolved.

The proposed coordination mechanism should be compared with existing coordination methods in end-to-end AI-scientist workflows under comparable settings. I consider this comparison central to the paper’s motivation. It would help discuss which coordination mechanisms work well for AI scientists and provide a basis for further work. The ablations of RAC’s own components are useful, but their value is limited.

So I lean toward keeping my original score.

Reviewer_ENNgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies runtime coordination in existing multi-agent research systems. RAC preserves each host’s capabilities while changing communication, successor selection, and—in its fullest condition—contracts and advisory verification. Four cumulative conditions distinguish these interventions across Agent Laboratory, EvoScientist, and ARK. The evaluation uses subsets of ResearchClawBench and a smaller DiscoveryBench transfer experiment. Runtime selection alone has the highest observed mean for each host, while the full framework improves two hosts and leaves EvoScientist approximately unchanged. The main contribution is this cross-host empirical comparison.

Critical strengths

The within-host comparison addresses a meaningful empirical question. The design retains the host’s models, capabilities, tools, and permissions while varying coordination. In particular, comparing communication alone with runtime successor selection helps distinguish additional shared context from changing which capability acts next. Applying a common intervention across hosts with different native control structures makes this more informative than a single-system demonstration.

The results include useful negative findings and careful interpretation. Runtime selection has the highest observed mean across all three hosts, while adding contracts and advisory verification lowers those means. The authors acknowledge that these additional components are bundled and that overhead, distracting instructions, and inaccurate feedback remain alternative explanations. They also distinguish descriptive rankings from repeatability, illustrative traces from causal evidence, and internal consistency from scientific correctness. This restraint strengthens the empirical contribution.

Critical weaknesses
  1. The intervention and matched-budget protocol are insufficiently specified. The paper describes the information available to the selector and its intended behavior, but does not provide enough operational detail to reproduce the shared policy. Important missing details include the routing and verification instructions, checkpoint representation, capability descriptions, host revisions, and handling of invalid selections. Matched maximum budgets are central to the comparison, yet their numerical values, units, general hop limits, and condition-specific stopping rules are not supplied. Different realized token consumption is compatible with equal maximum budgets, so I do not infer unfair allocation. Nevertheless, the missing configuration prevents independently checking this central experimental control. Please also clarify scored-run and token-record coverage and report termination causes by condition. These details would make the resource comparisons more interpretable and help distinguish budget exhaustion, native completion, and ineffective repeated coordination.

  2. Task selection and execution variability limit the reliability of the empirical pattern. Each task–host–condition has one execution at seed 0. Agent Laboratory and EvoScientist use ten ResearchClawBench tasks, ARK uses five, and the transfer experiment uses four DiscoveryBench tasks. The criteria determining these subsets and exclusions are not explained. The runtime-selection signal is broader than a single favorable average, but task-level gains and losses vary considerably. Repeated judging of one output does not measure variability in the research executions themselves. The observed ranking therefore remains preliminary. The authors explicitly acknowledge this limitation, which reduces its severity. A bounded repeatability check on a fixed common subset would strengthen the evidence. DiscoveryBench task identifiers and per-task, per-host results would also make the transfer finding more assessable.

Questions for authors
  • I personally felt like a lot of AI scientist agents are not cited, see ScientistOne, pAI/MSc, and many others. Analogously, the choice of what to benchmark is particular and I do not think I understand the rationale behind it. Also, most papers cited are from 2025, while the topic exploded in 2026, so I believe that a more thorough literature review is necesary. The agent thinks otherwise, though :) Can you please comment on the above?

  • What exactly implements the shared policies and matched resource constraint? Please provide the operative routing and verification instructions, host revisions and capability mappings, numerical budgets, hop limits, and stopping/accounting rules.

  • How were tasks selected and exclusions handled? Please explain ARK’s smaller coverage, identify any attempted but omitted runs, clarify the denominators underlying token means, and provide the task-level DiscoveryBench results.

  • What evidence is available about execution and evaluation variability? Please clarify judge repetitions and aggregation throughout Table 4, whose upper block explicitly uses median evaluations. Any existing independent-execution results—or a small targeted replication of the central comparisons—would materially strengthen confidence in the ranking.

1 comment

Thank you for the detailed review.

Host choice and related work. To test how next-role handoffs affect work done, we retained native roles and tools across diverse baseline workflows: Agent Laboratory (fixed phases), EvoScientist (native specialist delegation), and ARK (review loopbacks). We chose ResearchClawBench because its paper tasks, runnable environments, and shared rubrics let execution logs trace concrete outcomes—e.g., whether a handoff gets a review-flagged missing experiment run and reported. An exploratory DiscoveryBench subset checks if routing changes carry to tasks where data-derived findings are scored against references. We will also discuss complementary 2026 work, including ScientistOne (claim–evidence audits) and pAI/MSc (human-guided modular research).

Policy and configuration. The selector filters for available capabilities and picks the open issue with the largest (attempts, issue_id) pair. It follows a required native role if available. Otherwise it maximizes required-tag overlap, breaking ties by fewer card tags, then the larger capability ID. With no match, it falls back to an available planner. R1 uses the native successor with the same shared communication.

Contracts specify issues, read/write paths, and required evidence. Verification checks artifact changes, types, sizes, and nonempty output. supported means those checks passed. Verdicts are advisory, and out-of-scope writes generate warnings. Enabling verification also changes how some nonfatal errors and completion signals are handled, so that difference is part of the R2–R3 comparison.

Budget controls. In the added runs, N0 measured native usage under a separate ceiling. We set matched R1–R3 limits within each task using host-level N0 maxima plus headroom:

  • AL: 1.4M input tokens (5M for Math_000), 125K output, 14 hops, 5,400 s.
  • Evo: 3M input, 35K output (50K for Math_001–003), 14 hops, 5,400 s.
  • ARK: 150M input, 2.7M output, 46 hops, 64,800 s.

A hop can make several model calls. The initial per-run request limits were 100,000 for AL and Evo and 1,800 for ARK. Budget checks occur between requests and at role boundaries, so the last completed work can cross a ceiling. External judging is accounted for separately.

AL exposes seven phases from literature review to report refinement. Evo exposes planner, research, code, debug, data analysis, writing, and review. ARK exposes researcher, experimenter, coder, writer, reviewer, and planner. AL's R2 follows the native phase sequence. Host revisions are AL d9017d9, Evo 418abca, and ARK 1e14a34. We record integration hashes per run.

Tasks and denominators. We chose Energy_001, Life_001, Neuroscience_000/002, Physics_002, Math_000–003, and Information_003 for domain coverage, with an emphasis on math and information science. ARK runs the first five because it takes more time and resources. The added AL/Evo set has 119 scores for 120 planned cells. Evo Math_003 C-only produced no report, so its score remains missing and the ablation uses the other nine Evo tasks. Original score denominators are 10/10/5 by host. Usage denominators in N0/R1/R2/R3 order are AL 10/10/10/9, Evo 10/8/10/10, and ARK 5/5/5/5.

DiscoveryBench. Scores are in N0/R1/R2/R3 order on a 0–100 scale.

  • nls_ses/metadata_0: AL 0/13.33/22.22/100, Evo 12.50/44.44/44.44/0.
  • requirements_engineering_for_ML_enabled_systems/metadata_0: AL 0/0/19.05/0.
  • worldbank_education_gdp/metadata_0: AL 0/11.43/0/0, Evo 0/missing/0/0.
  • worldbank_education_gdp_indicators/metadata_0: AL 16.67/0/0/0.

We average available within-host differences from N0 per task, then weight the four tasks equally. Using unrounded scores, the gains are +4.35/+7.37/+6.77 for R1/R2/R3. The missing Evo R1 changes which hosts enter that mean.

Execution versus judging variability. Table 4’s upper 60 entries report the median of three judgments per artifact. The median within-artifact range was 1.98 points. In 6 of 15 matched R1–R2 task comparisons, the winner changed across the nine ways to pair one recorded score per condition. The lower 40 entries report archived final scores. The available receipts do not let us reconstruct the round-level aggregation for all 40.

In 3 of those 15 comparisons, every pair of recorded scores favored one condition in the original batch and the other condition in the Luna batch. This establishes a reversal across batches for those artifacts. Repeatability under a fixed model and configuration remains untested. Across the 25 shared host–task pairs, the mean R2−R1 difference was positive in all three hosts in both batches.

Reviewer_5UiFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Runtime Agent Coordination (RAC), which allows the currently active agent to select the next authorized agent based on the research state, available artifacts, unresolved issues, and remaining budget. The full framework also adds scoped work contracts and advisory artifact verification.

The evaluation compares native execution, additional communication, runtime selection, and contracts plus verification across Agent Laboratory, EvoScientist, and ARK on ResearchClawBench, with a small transfer study on DiscoveryBench. Runtime selection achieves the highest observed mean score in all three hosts, while adding contracts and verification produces mixed outcomes. The paper uses these results and selected execution traces to examine the benefits and limitations of runtime coordination.

Critical strengths

1. The cumulative comparison is a useful experimental design.

The distinction between communication and next-agent selection is worth studying. R1 preserves native successors, while R2 changes selection under the same communication mechanism. Keeping the host agents, models, tools, and permissions fixed makes this comparison informative. Applying the same coordination interface to three hosts with different native control structures also gives the study practical relevance beyond a single system.

2. The mixed results provide useful evidence about coordination choices.

I appreciate the clear reporting that R2 has the highest observed mean in all three hosts, while the additional contracts and verification in R3 do not improve those means. The authors also distinguish illustrative failure cases from established explanations of aggregate performance. This makes the negative findings useful: the results identify a promising intervention while showing that additional coordination components require their own justification.

Critical weaknesses

1. The evidence is too limited to establish a reliable performance advantage.

There is one execution per task–host–condition, with ten tasks each for Agent Laboratory and EvoScientist and five for ARK. R2 improves the reported means, but the gains for EvoScientist and ARK are only about 2–3 points, and task-level outcomes vary substantially. With one trajectory per condition, it remains unclear whether these improvements persist across independent executions. Repeated judging of the same output cannot resolve variability in the research process itself. The authors appropriately describe the results as descriptive, but the central empirical conclusion remains preliminary.

2. The coordination policy and budget controls need a more reproducible specification.

Sections 3.1–3.2 describe the information available for selection and the shared interface, but do not provide the actual selection prompts or a sufficiently concrete decision procedure. The contract and verifier prompts and numerical run-budget settings are also missing from the provided description. These details define the intervention being evaluated.

The proposed explanation that R3 loses useful work to coordination overhead also remains unresolved. Matched-task records of completed steps, resource consumption, and stopping reasons are needed to assess that explanation; aggregate cost averages cannot establish it.

I would raise my score if the authors demonstrate that the runtime-selection gains persist across independent runs and provide a reproducible specification of the coordination policy and budget controls.

Questions for authors
  1. Do the improvements from R2 over N0 and R1 persist across independent executions? Additional runs on the same tasks, with uncertainty estimates for paired score differences, would materially affect my assessment.

  2. Can the authors provide the exact selection, contract-generation, and verifier prompts, together with the implementation version and numerical budget and stopping configurations used for each host?

  3. On matched tasks, does R3 actually complete fewer useful research steps or exhaust its budget earlier than R2? Per-run stopping reasons and resource records would help assess whether coordination overhead explains the observed performance differences.

1 comment

Thank you for these concrete questions.

Do the gains persist across executions? Yes. R2 again scored higher than R1 on average in all three hosts when we reran the same tasks with GPT-6 Luna. The table reports the Luna results on 10 tasks each for Agent Laboratory and EvoScientist and five for ARK.

HostN0R1R2R3R2−R1
AL3.9456.7456.8955.640+0.150
Evo10.9957.2158.3606.920+1.145
ARK8.2309.86012.4309.870+2.570

R2 beats R1 on 5/10, 6/10 and 4/5 tasks, respectively. For Δt=sR2,t−sR1,t\Delta_t=s_{R2,t}-s_{R1,t}, 100,000 within-host task-pair bootstrap resamples (seed 20261117) give 95% percentile intervals of [−2.015, 2.450], [−2.410, 4.860] and [−0.790, 5.840]. All include zero. These intervals describe sensitivity to the selected task mix, not variability across repeated executions of the same task. Another model also produces positive mean differences.

Relative to N0, R2 gains 2.950 points in AL and 4.200 in ARK, but loses 2.635 in Evo. Thus the positive R2−R1 mean recurs across all hosts, whereas improvement over native execution does not.

Budget controls. N0 received a generous ceiling to measure native usage. R1–R3 limits were calibrated from host-level N0 maxima with approximate headroom, with a separate AL Math_000 input allowance. Final limits match across R1–R3 within each task:

  • AL: 1.4M input tokens (5M for Math_000), 125K output, 14 hops, 5,400 s.
  • Evo: 3M input, 35K output (50K for Math_001–003), 14 hops, 5,400 s.
  • ARK: 150M input, 2.7M output, 46 hops, 64,800 s.

A hop can contain multiple model requests. The initial per-run request limits were 100,000 for AL and Evo and 1,800 for ARK. Checks between requests and at capability boundaries can allow completed work to cross a ceiling before subsequent work stops. External judging is excluded. N0 and R1–R3 do not have identical configured ceilings; R1–R3 provide the matched intervention comparison.

Selection and templates. The evaluated selector is programmatic: filter available capabilities; choose the unresolved issue with the largest (attempts, issue_id) pair; follow an available native-required role if that issue specifies one; otherwise maximize required-tag overlap. Ties favor fewer card tags, then the lexicographically larger capability ID; an available planner is the no-match fallback. R1 uses the native successor under the same shared communication.

Contracts specify scoped issues, readable/writable paths and required persisted evidence. Verification checks artifact changes, types, sizes and nonempty output; supported is not a scientific-validity verdict. Out-of-scope writes are warnings. Verdicts enter later context but do not themselves trigger rollback, retry or termination.

There are no separate LLM selector/contract-generator/verifier prompts. The execution template is Objective: {objective}. Perform the {capability_id} step using only the current workspace. Persist all work before returning. Contracts add the scope and evidence fields above. SharedNet adds background context. Adapters pass this request to the selected native role.

Recorded upstream commits are AL d9017d9, Evo 418abca, ARK 1e14a34; episode metadata retains integration source-tree hashes. Judging uses Azure GPT-5.5, API 2025-04-01-preview, serial evaluation and three-round medians.

Does R3 do less work? Yes. Per-run records confirm both phenomena: R3 can exhaust its budget earlier and complete fewer useful research steps due to coordination overhead.

First, R3 can exhaust budgets prematurely. In ARK Life_001, R3 halted at hop 34 after hitting the 1,800-request cap, whereas R2 completed 46 hops using only 1,311 requests before reaching the hard hop limit.

Second, even when hop counts match, R3 completes fewer productive steps. Across 25 paired runs, R3’s deficit is largest under equal hops (n = 10, mean Δ = −3.265, lower in 8/10). In ARK Neuroscience_002, both systems reached 46 hops and invoked the experimenter 10 times (scores: 6.55 vs. 0.25). While R2 produced four classifier baselines, R3 executed no experiments. Its tag selector repeatedly assigned a manuscript-editing issue to the experimenter, whose declared write contract excluded report/main.tex. Because validation accepted blocker reports, R3 wasted 24 requests across hops 6, 11, and 15 trapped in an unresolvable reporting loop.

Therefore, coordination overhead and contract mismatches directly displace productive empirical work, explaining the observed performance gap.

Reviewer_uHTtOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether a shared runtime agent-selection mechanism can improve existing AI-scientist systems. Its motivation is that different research states require different corrective actions: a missing experiment, broken bibliography, and inconsistent figure need not trigger the same successor. Runtime Agent Coordination (RAC) lets the current agent select an authorized host capability using saved artifacts, unresolved issues, execution history, and remaining budget. The full configuration adds work contracts specifying objectives and outputs, followed by artifact-grounded verification. Verification supplies advisory feedback rather than directly triggering retries, rollback, or stopping.

The evaluation compares native execution, additional communication, runtime selection, and selection with contracts plus verification across Agent Laboratory, EvoScientist, and ARK. The same agents and tools are retained within each host. Runtime selection without the final contracts-and-verification bundle has the highest observed mean score in all three hosts, but the experiment uses only one run per task–host–condition.

Critical strengths

The experimental design separates information sharing from successor selection. The communication-only condition retains native successors, while the next condition changes selection under the same communication mechanism. Models, native role prompts, tools, permissions, inputs, and maximum budgets remain fixed within each host. This provides a useful comparison of routing policies without replacing the entire agent system.

Critical weaknesses

R3 performs worse than R2 in every host, but contracts and verification are added together. Their individual effects, interaction, and budget cost cannot be separated. Separate ablations and measurements of completed research steps would help explain the decline.

In the experiment, each task–host–condition is executed only once, which cannot show whether the reported gains persist across independent executions.

The paper lacks detailed descriptions on the coordination/routing procedure.

Questions for authors

Do R2’s gains persist across independent runs? Results from several seeds, with paired comparisons on the same tasks, would help distinguish a reliable improvement from execution variability.

Why does R3 perform worse than R2?

1 comment

Thank you for recognizing the communication-versus-selection comparison. We address repeatability, the R3 decline and the missing procedure details below.

Do the gains persist across executions? Yes, the average R2–R1 gain appeared again in all three hosts when we reran the same tasks with GPT-6 Luna. The table reports the selected Luna runs for Agent Laboratory and EvoScientist (10 tasks each) and ARK (5 tasks), on a 0–100 scale.

HostN0R1R2R3R2−R1
AL3.9456.7456.8955.640+0.150
Evo10.9957.2158.3606.920+1.145
ARK8.2309.86012.4309.870+2.570

R2 beats R1 on 5/10, 6/10 and 4/5 tasks, respectively. For Δt=sR2,t−sR1,t\Delta_t=s_{R2,t}-s_{R1,t}, 100,000 within-host task-pair bootstrap resamples (seed 20261117) give 95% percentile intervals of [−2.015, 2.450], [−2.410, 4.860] and [−0.790, 5.840]. All include zero. These intervals describe sensitivity to the selected task mix, not variability across repeated executions of the same task. Another model also produces positive mean differences, but same-model, same-configuration repeatability remains unresolved. Each score is the median of three judge evaluations.

Why R3 can underperform R2. The traces expose two failures: verification can accept intermediate artifacts without obtaining target measurements, and a restrictive contract can block a necessary revision. Advisory feedback does not itself repair either problem. We tested the added components separately and examined the corresponding traces.

Component ablations. We added contracts-only (C) and verification-only (V) conditions. Means are:

HostR2R2+CR2+VR3
AL, 10 tasks6.8958.2306.4505.640
Evo, 9 common tasks9.0678.7335.3337.350
ARK, all 5 tasks12.43011.32010.4109.870

Evo Math_003 R2+C exhausted its budget without a report; it is missing, not zero. The complete ARK ablations reinforce that neither added component improves R2's aggregate mean. In the inspected implementation, verifier-enabled execution also changes handling of some nonfatal errors and completion signals. These comparisons therefore cannot isolate contracts and verification from continuation behavior.

The traces identify specific design gaps. On AL Physics_002, contracts-only scored 22.70 and its report describes 500 validated XEB estimates. Verification-only scored 3.35, reported no compatible empirical estimates and used synthetic values for plotting demonstrations; all seven stages nevertheless received supported. Artifact checks can pass without obtaining target measurements.

On ARK Neuroscience_002, R3's writer repeatedly reported that its contract permitted writes under report/** without the read access needed to revise the manuscript. At hops 7, 12 and 20 it recorded the blocker instead of revising. Advisory verification did not itself repair that scope mismatch. R2/R3 scored 6.55/0.25. These are observed failure mechanisms, not a decomposition of the aggregate score gap.

Coordination and interpretation. The evaluated selector is programmatic: filter available capabilities; choose the unresolved issue with the largest (attempts, issue_id) pair; follow an available native-required role if that issue specifies one; otherwise maximize required-tag overlap. Ties favor fewer card tags, then the lexicographically larger capability ID; an available planner is the no-match fallback. R1 uses the native successor under the same shared communication.

Routing needs both an actionable issue and an opportunity to act. AL R2 follows the native phase sequence, and none of the 30 AL R1–R3 traces selects a capability after report refinement. Its +0.150-point difference therefore does not demonstrate a benefit from changed routing and may reflect execution variability. In ARK Physics_002, all 11 R1 post-review handoffs go to the planner, while R2 includes three direct handoffs to the coder. After one, the hop-18 coder reported parsing 4,610 JSON inputs, producing results for 550 eligible XEB instances and regenerating figures, while retaining unresolved scientific gaps. This shows how runtime selection can send a concrete review issue directly to a role that performs the corrective work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.