acceptodds
Under review as a conference paper at ICLR 2027

Async-RBench: Benchmarking Asynchronous Task Execution in LLM Agents

Abstract

Large language model (LLM) agents increasingly execute tasks asynchronously: a main agent continues working while its subagents run, and the task state may change before their results arrive. The main agent must decide which work to update or preserve and which checks to repeat, while avoiding prohibited changes. Final task success alone does not reveal whether these results were handled correctly, and natural asynchronous arrivals are hard to reproduce. We introduce Async-RBench, a benchmark for asynchronous task execution in LLM agents, with 200 executable tasks covering eight scenarios. A scenario controller regulates result delivery using predefined task-state conditions, while an Event Response Contract (ERC) specifies what each response must change, preserve, avoid, and re-verify. The Dynamic Replanning Score (DRS) weights the response process and outcome equally and is reported separately from task success. We evaluate nine agents and find DRS scores of 19.2–45.1, with the lowest scores on duplicate reports and subagent failures. Under our execution protocol, all nine agents obtain lower task success with asynchronous than with batched delivery (26.0% vs. 29.7% on average). For three agents, prompting the main agent to rebuild its plan after every result improves DRS by only 2.6–3.5 points. Together, these findings highlight the challenges of reliable asynchronous task execution and the need to evaluate how agents handle incoming results alongside end-to-end task success. We release our code at https://anonymous.4open.science/r/Async-RBench-723A

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent benchmark, multi-agent collaboration, asynchronous interruptions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8aXdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Async-RBench, a benchmark for evaluating how LLM agents handle asynchronously arriving subagent results during ongoing task execution. It contains 200 executable tasks adapted from five existing benchmarks and organized into eight asynchronous scenarios. A scenario controller regulates result delivery, while Event Response Contracts (ERCs) specify required changes, preserved work, prohibited actions, verification requirements, and expected outcomes. The proposed Dynamic Replanning Score (DRS) evaluates event-level response quality separately from end-to-end task success. The authors evaluate nine agents under asynchronous and batched result delivery, with three repetitions per task and mode. They report DRS scores ranging from 19.2 to 45.1, lower task-success point estimates under asynchronous delivery, and modest improvements from mandatory plan rebuilding. They also conduct Oracle/Frozen comparisons, sampled human audits, and reference-solution and verifier checks to support the evaluation design.

Critical strengths
  1. A meaningful and practically relevant evaluation problem. The paper identifies an important gap in current agent evaluation. Final task success may fail to capture whether an agent correctly incorporates late-arriving information, preserves unaffected work, avoids unnecessary side effects, and revalidates previously completed steps. The eight asynchronous scenarios cover several meaningful situations in multi-agent execution. The distinction between event-level response quality and final task success is well motivated and potentially useful for future research.
  2. A substantial executable evaluation framework with encouraging validation efforts. The authors construct 200 executable tasks, provide a controlled result-delivery mechanism, and evaluate nine agents under a relatively consistent execution protocol. The released task metadata provides identifiable upstream sources, and the paper reports reference executions, equivalent-solution checks, invalid variants, human auditing, and Oracle/Frozen comparisons. These are relevant components of a rigorous benchmark-validation pipeline and provide a useful foundation for further evaluation.
Critical weaknesses
  1. Insufficiently justified benchmark curation and concerns about construct validity. For a benchmark paper, I consider the scientific rigor of dataset curation and validation to be a fundamental requirement. In particular, a reliable benchmark should establish clear provenance for every instance, justify why each task and scenario was constructed, ensure that the intended capability is actually being measured, and validate the resulting tasks before using them to evaluate other systems. While Sections 3.3 and A describe a general construction pipeline, the instance-level documentation I inspected provides limited justification for individual design decisions. The released provenance records identify the original benchmarks and task IDs, which is useful, but they do not sufficiently explain why each source task was selected, why a particular asynchronous event was introduced, how its timing and dependencies were justified, or how the corresponding ERC requirements were determined. More importantly, I found concrete examples of participant-visible instructions that explicitly disclose the expected decision. In mab-conflicting-specialist-results-ce3ff5b928, adapted from MultiAgentBench, the task extension directly instructs the agent to select INSERT_LARGE_DATA as the final database diagnosis. In mab-dependency-unblock-1c96d4414d, adapted from Gaia2, the instructions specify which revision should be adopted, which late receipt should be rejected, and the exact replacement scheduling actions. These cases raise a substantive construct-validity concern. If the expected decision is already provided in the task instruction, success may primarily reflect compliance with an explicitly specified resolution, rather than the ability to independently determine the validity, authority, and consequences of asynchronously arriving evidence. Such guidance may be acceptable for some execution-oriented tasks, but its compatibility with the paper's claimed evaluation objective needs to be established. I would also like clarification about development-time exposure of evaluation cases. The released registry retains legacy calibration and development labels for instances included in the final 200-task evaluation set. Although the paper reports a separate construction set, the exposure and revision history of the final cases is not sufficiently clear from the available documentation. These concerns matter because executable task validity alone does not establish benchmark validity. The authors should demonstrate that the curated cases are correctly specified, representative of the intended capability, sufficiently diverse, and free from answer-revealing instructions that compromise the intended evaluation.
  2. Insufficient per-instance Oracle and verifier validation, limiting confidence in score reliability. A high-quality executable benchmark should demonstrate that every included task has a feasible reference execution, that correct outcomes and alternative valid solutions are accepted, and that invalid behaviors are reliably rejected. For agent benchmarks, this requires checking both final artifacts and the relevant execution trajectories, including the evidence supporting changes, preservation, prohibited operations, and revalidation. The paper makes useful efforts in this direction, including reference and equivalent solutions, invalid variants, and a human audit reporting 96.6% initial agreement on scored items. However, the evidence is primarily aggregated, and the paper does not provide a complete per-instance validation report establishing the acceptance behavior of all 200 tasks. The Oracle results in Figure 3 are particularly important. Despite executing reference recovery programs with access to the event requirements, the Oracle achieves DRS values of 96.4–98.8 rather than 100. These aggregate values do not by themselves establish the presence or number of broken tasks, since DRS is a continuous process-and-outcome metric. Nevertheless, every unexplained Oracle score deficit deserves investigation. The authors should identify which cases and scoring components account for the missing credit, distinguish reference-execution limitations from task or verifier defects, and establish whether every included instance has a reference execution that satisfies its complete acceptance criteria. The discriminative validity of DRS also needs stronger support. Section 3.2 assigns zero when the main agent terminates before the target result is presented, which combines failure to encounter an event with incorrect handling after receiving it. Differences in source-task difficulty, event exposure, and the amount of guidance in task instructions may therefore influence the reported scenario scores and model rankings. Oracle/Frozen separation is encouraging, but it is insufficient on its own to establish that the benchmark specifically measures asynchronous response competence. Finally, the released public cases do not include the original paper's complete ERC truth and hidden verification materials, and the documented newer runtime judge does not directly reproduce the historical paper scoring. A reproducible route for validating the original results is important if the benchmark is intended to become a reliable community evaluation resource. My concern is the downstream consequence of potential benchmark defects. If a task contains an incorrect ground truth, an overly restrictive verifier, or answer-revealing instructions, researchers may attribute misleading evaluation outcomes to model capability. Such issues should be identified, corrected, and revalidated before the benchmark's scores are treated as reliable evidence.
Questions for authors

Q1. Per-instance curation and provenance.

Could the authors provide a complete construction and validation record for all 200 tasks? Ideally, this would include the original source task, the transformation performed, the rationale for the chosen asynchronous scenario and release conditions, the ERC requirements, and the acceptance results. How were the scenarios selected, and how was the correctness of each transformed task independently established? Please also clarify which evaluation cases were accessed or revised during development.

Q2. Answer-revealing instructions and construct validity.

For cases such as mab-conflicting-specialist-results-ce3ff5b928 and mab-dependency-unblock-1c96d4414d, why do the participant-visible instructions explicitly specify the expected decision? How many cases contain similar outcome-specific guidance? Can the authors audit the complete benchmark for this issue and demonstrate, for example through an ablation using more neutral instructions, that the benchmark still measures independent asynchronous evidence handling?

Q3. Complete Oracle and verifier validation.

Can the authors report the per-instance Oracle validation results across all 200 tasks, including full task-success status, ERC component scores, and the exact causes of any failures or missing credit? In particular, what accounts for the Oracle DRS values below 100 in Figure 3? Were all tasks independently verified to have at least one execution that passes every final acceptance test and applicable event-response requirement? How many equivalent correct solutions and intentionally invalid variants were tested, and what were their acceptance and rejection rates?

Q4. Reliability and discriminative validity of the evaluation.

Can the authors report target-event presentation rates, DRS conditional on successful presentation, and the C/R/F/V/O score breakdown by model and scenario? How stable are model rankings when controlling for source-task difficulty and task-specific guidance? Can independent researchers reproduce or validate the exact scoring criteria used for the reported results? These details are important for determining whether the benchmark reliably measures asynchronous response competence.

11 comments

Dear Reviewer 8aXd,

Thank you for your rigorous review. Your W1/W2 and Q1–Q4 point to the same body of evidence, so we answer them together below.

We have carefully addressed all of your questions (W1/W2 and Q1–Q4), and added per-task evidence and the corresponding paper revisions. Your rigorous comments made our construction rationale, solvability evidence and scoring design clearer and easier to verify. If these responses and revisions have resolved your main concerns, we would be grateful if you would consider re-evaluating your assessment of our paper; if any questions remain, we would be glad to discuss them further.

W1 & Q1 & Q2. Construction rationale, source selection and instruction wording

Q1. Could the authors provide a complete construction and validation record for all 200 tasks? ... How were the scenarios selected, and how was the correctness of each transformed task independently established? Please also clarify which evaluation cases were accessed or revised during development.

Q2. For cases such as mab-conflicting-specialist-results-ce3ff5b928 and mab-dependency-unblock-1c96d4414d, why do the participant-visible instructions explicitly specify the expected decision? How many cases contain similar outcome-specific guidance? Can the authors audit the complete benchmark for this issue and demonstrate, for example through an ablation using more neutral instructions, that the benchmark still measures independent asynchronous evidence handling?

(1) Source selection and transformation method. We chose Terminal-Bench, Gaia2, OSWorld, SWE-bench and MultiAgentBench not as an arbitrary combination, but because together they satisfy the four necessary conditions for constructing asynchronous replanning tasks. First, the tasks are complex enough to split naturally into multiple parallel, interdependent sub-workflows — the premise for results arriving while work is in progress; source tasks that are too simple finish before the event arrives, so the interruption cannot occur. Second, the initial environment is executable and reproducible, so the states before and after the event can be genuinely recorded. Third, the acceptance criteria are public and executable, so “what was changed, what was preserved, and whether it was reverified” can be checked rather than judged subjectively. Fourth, source versions and task IDs are traceable, allowing others to check the correspondence before and after transformation. The transformation follows the principle of “keep the source task, add only the collaboration and event layer”: the source task's goal, initial environment and acceptance tests are unchanged, and what is added are the sub-agent workflows, result version relations, release conditions and the ERC. This preserves ecological validity while isolating asynchronous replanning, avoiding mixing GUI operation or basic coding ability into the measured ability.

We have built a per-case construction and acceptance record for all 200 tasks, with fixed fields: source task and version, transformation, scenario rationale, release conditions and timing, ERC requirements, workflows and artifacts, hidden checks and reverification anchors, reference and equivalent-solution results, and execution and detection of invalid variants. A metadata table without answers or reference solutions will be released with the public repository (paper_artifacts/construction-records-200.csv); the full records, including reference solutions, hidden checks and invalid variants, are provided as supplementary material, with each of the 200 cases having a reference solution, a verifier, an invalid variant and an equivalent solution.

(2) Why the instructions state the event. We confirm that 28 cases (14%) have participant-visible instructions that reveal the expected handling or outcome. The reason is that these cases come from open-ended tasks (especially multi-agent and database-diagnosis types) whose source tasks allow multiple reasonable forms of delivery. Without stating the event payload, the expected terminal state and the delivery path, a run often cannot even enter the intended setting — the main agent may choose another equally reasonable but incomparable approach, or produce no checkable artifact, so the event can be neither observed nor scored. These statements were originally intended to make the tasks executable, the events observable and the acceptance decidable, not to lower the difficulty. We run a paired before/after comparison on these 28 cases: the same systems and the same number of repetitions, once with the original instructions and once with neutral instructions, reporting the change in DRS and task success for each affected case.

(3) Relationship between the construction set and the evaluation set. The construction set and the evaluation set are two separate batches of tasks: the 47 construction tasks do not overlap with the 200 evaluation tasks and do not participate in any reported result. The calibration/development/test labels in the registry are historical labels from the old repository policy, recording whether a case already had runs or paired evidence in the early repository; they are kept only for compatibility and optional diagnostic filtering, do not define the paper's data split, and all 200 cases participate in the evaluation.

W2 & Q3 & Q4. Solvability of the 200 tasks, rationale for the DRS design, and reproducibility

Q3. Can the authors report the per-instance Oracle validation results across all 200 tasks ...? What accounts for the Oracle DRS values below 100 in Figure 3? Were all tasks independently verified to have at least one execution that passes every final acceptance test ...?

Q4. Can the authors report target-event presentation rates, DRS conditional on successful presentation, and the C/R/F/V/O score breakdown by model and scenario? ... Can independent researchers reproduce or validate the exact scoring criteria used for the reported results?

(1) All 200 tasks are solvable. Here we do not discuss whether the acceptance records are complete, but answer the solvability question directly: for all 200 tasks we apply the same criterion — at most 4 runs, and if any one run passes all terminal-state and event requirements, the task is judged solvable (pass@4 = 1). 186 tasks were passed on the first execution of the reference solution, with the declared invalid variants failing as expected; of the other 14 supplementary tasks, 11 were completed by a model within 4 runs, and 3 were passed by the scripted reference solution and an independent equivalent solution. Hence 200/200 meet this criterion, and there is no case of “passes acceptance but is unsolvable”.

Table 5. Solvability of the 200 tasks (pass@4)

Task setPassedpass@4
186 tasks with an existing reference execution186186/186
14 supplementary tasks1414/14
All 200 evaluation tasks200200/200

The revised version will release a per-case table listing the pass@4 result, reference solution, equivalent solution, invalid variants, terminal-state points and trace points.(2) Why the Oracle scores of 96.4–98.8 fall short of 100. The Oracle is not a “perfect trajectory”, but a reference recovery program written by humans at design time; it is executed in real runs and scored on the actual execution result, not given full marks automatically because of its “reference” status. It starts from each model's own pre-event state, can only perform the operations that state allows, and does not automatically produce the evidence that some components require (for example, an explicit post-event reverification record), so scores below 100 are normal. Precisely because the starting points differ, the same reference program obtains different Oracle scores on the three models (98.8 / 98.5 / 96.4) — which shows that DRS evaluates “what was done in this actual state” rather than assigning a fixed score to a script; this is itself evidence that DRS is discriminative. Solvability remains defined by the pass@4 criterion in (1), which all 200 tasks satisfy.(3) Why DRS is valid, and how to reproduce it. DRS splits event response into C/R/F/V (the response process) and O (the terminal state within the event's scope of impact): P is the mean of the first four, Q is O, and the two are weighted equally because they answer the two halves of the same contract; S is reported separately because it requires the whole task to be completed. This design is not arbitrary, and the experiments above give direct evidence:- Insensitivity to weights. When the P:Q weight is varied, the full ranking is unchanged for w∈[0.2,0.8]w\in[0.2,0.8] (τ=1.00\tau=1.00), and under extreme weights τ\tau stays at or above 0.79 (Table 1).

  • P and Q are separable. The audited runs contain both “high Q, low P” and “high P, low Q” cases, showing that the two are not two measurements of the same quantity (see W1 and Q1–Q2 above).
  • Cross-scenario differences come mainly from difficulty, not metric noise. The variance decomposition of the 9×8 DRS matrix is 77.8% scenario, 12.6% system and 9.6% interaction; after removing the scenario main effect the overall ranking is unchanged (τ=1.00\tau=1.00, Table 4). This also addresses the question of whether the ranking is affected by the task mix.
  • The zero-scoring rule avoids rewarding avoidance. A run is scored zero if the target result is never presented, because without the event there is no response to evaluate; we also report the presentation rate and conditional DRS so that “did not encounter” and “handled wrongly” can be distinguished.
  • The instruction issue does not affect the ranking. The paired before/after comparison on the 28 cases with hint-bearing instructions reports the change in DRS and task success for each affected case (see point (2) of W1 and Q1–Q2 above). For reproducibility we provide two paths. First, the public repository contains the framework, the 200 participant-visible task packages, the registry and the experiment manifest (binding the model identifier, protocol version, evaluation-contract digest and judge digest), so any researcher can run a system under test locally with the same framework. Second, the public site shows the paper's results, the task catalog and a community leaderboard, and provides a submission entry: independent researchers who complete a run submit a run receipt through the site, which, after review by maintainers under a fixed procedure, enters the community leaderboard; paper results and community results are displayed separately and never overwrite each other. The ERC ground truth, the hidden verifiers and the historical judge are not distributed with the public package, because they contain answers and judging logic, and releasing them would contaminate the benchmark and invalidate future evaluations; teams that need to reproduce can obtain a version-bound judge through an application channel, and the site's submission records and release-manifest digests are used to check version consistency.A pass@4 solvability conclusion has been added to §4.3.
Reviewer_7Vb3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Async-RBench to evaluate the capability of LLM agents when the main agent keeps executing its task while sub-agent results arrive asynchronously and the task state may evolve over time. The work designs eight categories of asynchronous scenarios, introduces the Event Response Contract (ERC) to specify the mandatory responses for each event, and proposes the Dynamic Replanning Score (DRS). It also reports asynchronous execution results from nine agents. The topic is practically meaningful. In particular, modern multi-agent systems commonly adopt concurrent subtasks, background tool calls and delayed result delivery, yet existing benchmarks mostly only evaluate whether the final task succeeds and cannot examine whether agents correctly handle stale, duplicate, partially completed or conflicting results.

Critical strengths
  1. The problem is important and relatively well-formulated, closely resembling real-world multi-agent settings.
  2. The categorization of scenarios is practically valuable. The eight scenarios cover many common failure modes, and the examples in the appendix are more concrete than abstract descriptions, helping readers understand each type of asynchronous error.
  3. The evaluation metric emphasizes process-level correctness and can expose issues that traditional task success metrics fail to capture.
Critical weaknesses
  1. The experiments mainly control for result delivery timing, rather than asynchronous execution in the full sense. It is recommended to adopt more precise description throughout the paper, and avoid generalizing findings from this delivery protocol to the entire asynchronous execution system.
  2. The experimental section lacks in-depth analysis. For instance, It is observed that model rankings shift across scenarios and diverge from the task-success-based ranking, but the underlying causes are not investigated.
  3. Although the performance of a wide range of LLMs is evaluated, deeper comparative analyses are missing. Examples include comparisons between open-weight and closed-source models, thinking vs non-thinking models, and models of different sizes.
Questions for authors

Please refer to weakness.

6 comments

Dear Reviewer 7Vb3,

Thank you for your detailed review. We respond to each point below.

We have carefully addressed all of your questions (W1–W3) and added the corresponding experimental data and paper revisions. Your comments made our description of the experimental scope and conclusions more precise. If these responses and revisions have resolved your main concerns, we would be grateful if you would consider re-evaluating your assessment of our paper; if any questions remain, we would be glad to discuss them further.

W1. The experiments control only the timing of result delivery, not asynchronous execution in the full sense

To clarify: “results arrive while work is in progress, forcing the main agent to judge and replan” is exactly the phenomenon this benchmark is designed to measure, not a simplification of a full asynchronous system.

The setting we study is this: multiple Children execute concurrently, some results arrive after the main agent has already started working, and the main agent must judge their validity and scope of impact and decide what to keep, modify or reverify. To reproduce this setting under controlled conditions, we make “when and in what order qualifying results are presented” the experimental variable, and use synchronous joining only as a control baseline — the baseline is for comparison, while the benchmark characterizes incremental delivery itself. This is how asynchronous interruption manifests in real multi-agent systems. A full asynchronous system additionally involves scheduling, preemption and shared-state concurrency, which is a separate research question.

We will adopt more precise terminology throughout — asynchronous result delivery / asynchronous interruption — and add an explicit scope statement. Specifically: the residual “asynchronous execution” in the §4.4 cost-analysis paragraph and its heading and in Appendix C will be changed to “asynchronous result delivery”; a scope paragraph will be added to §4.1 stating that the comparison fixes the delivery protocol; the abstract and conclusion will keep the result-delivery wording.

[Added] A scope statement has been added to §3.1; the Table 3 caption and Appendix C now use “asynchronous result delivery”.

W2. Rankings drift across scenarios and diverge from the task-success ranking

That rankings vary across scenarios and differ from the task-success ranking is an expected and desired result, not a gap in the experimental design. The benchmark contains eight failure modes; if they all produced exactly the same ordering, that would indicate the scenarios lack discriminative power. The revision adds a decomposition of this phenomenon rather than eliminating it.

First, scenario rankings differ because different scenarios require different response behaviors. In Table 3 of the paper, Qwen3.7 Plus scores 6.9 on Child failure and 85.9 on Resource stragglers; Claude Sonnet 5 scores 15.5 on Duplication and replay, lower than GPT-5.6 Terra's 18.5. The mean pairwise Kendall τ\tau between scenario rankings is 0.42, indicating that they are not repeated measurements of the same ability.

Second, DRS and task success rank systems differently because they measure different things: DRS evaluates the correctness of event response, whereas S also requires the whole task to be completed, including goals unrelated to the event.

Table 3. DRS vs. task success (source: main results table of the paper)

SystemDRS (rank)S Async (rank)
Claude Sonnet 545.1 (1)38.2 (2)
GPT-5.6 Terra39.7 (2)38.8 (1)
Qwen3.7 Plus38.2 (3)18.7 (7)
GPT-5.6 Luna31.1 (5)34.3 (4)
DeepSeek V4 Pro30.6 (6)34.7 (3)

Qwen3.7 Plus ranks 3rd on DRS but 7th on Async success; DeepSeek V4 Pro is the opposite (6th on DRS, 3rd on success). If the two rankings agreed, DRS would have no independent value. This divergence is precisely why we report DRS and S separately, not evidence of inconsistent scoring.

Third, two kinds of causes need to be distinguished. One is measurement scope: a system may finish before the target result is presented and is then scored zero under a predefined rule — this is “did not encounter the event”, not “handled it wrongly”; we separate the two with the presentation rate and conditional DRS. The other is capability structure: different scenarios probe change, preserve, forbid and verify, and systems differ in their component strengths, so rankings naturally vary across scenarios — this is not noise but the capability difference DRS is meant to capture.

We also quantified where the cross-scenario differences come from. Treating the 9×8 DRS matrix in Table 3 of the paper as a two-way system × scenario layout, we decompose the total variance:

Table 4. Variance decomposition of the 9×8 DRS matrix

ComponentShare of variance
Scenario (task difficulty)77.8%
System12.6%
System × scenario interaction9.6%

Scenario means range from 8.6 to 60.7 DRS — Duplication and replay is simply much harder than Resource stragglers — so the apparent “ranking drift” comes mainly from scenario difficulty rather than metric instability. After removing the scenario main effect, the system ranking is completely unchanged (Kendall τ\tau = 1.00): Sonnet > Terra > Qwen3.7 > GLM > Luna > DeepSeek > Qwen3.5 > Kimi > Gemini. The real capability structure lies in the 9.6% interaction term, which is exactly what the scenario-level table reports (e.g., Qwen3.7 Plus at 85.9 on stragglers and 6.9 on child failure).

We do not treat this phenomenon as a bias to be eliminated. Differences in ordering across scenarios and across metrics are precisely the benchmark's discriminant validity, and the basis for our claim that DRS cannot be replaced by S.This variance decomposition has been added to §4.3.

W3. Missing comparisons across open- vs closed-source, thinking vs non-thinking, and different model sizes

Thank you for the suggestion. We have added grouped data for 9 configurations, shown in Tables A–C.

Open vs. closed source (Table A). The 5 closed-source configurations have DRS of 54.9, 47.6, 46.9, 36.1 and 15.9, with a mean of 40.3 and a mean of 61.9 calls; the 4 open-source configurations have 35.5, 7.6, 3.2 and 0.6, with a mean of 11.7 and a mean of 38.3 calls.

Thinking (Table B). Sonnet 5.5: thinking on 54.9, off 46.9, a difference of −8.0; calls 51.4 and 45.5, a difference of −5.9. GPT-6 Sol: reasoning high 47.6, none 36.1, a difference of −11.5; calls 55.7 and 37.5, a difference of −18.2.

Parameter count (Table C). 8B scores 0.6, 27B 35.5, 70B 3.2 and 72B 7.6; the corresponding calls are 44.1, 73.3, 7.0 and 28.8.

Comparison with the main table of the paper. In the paper, the 5 closed-source systems have DRS of 45.1, 39.7, 38.2, 31.1 and 19.2, with a mean of 34.7; the 4 open-source systems have 33.4, 30.6, 29.8 and 26.1, with a mean of 30.0. The thinking setting and parameter count are not reported per system in the main table of the paper.

Table A. Open vs. closed source

ConfigurationDelayStaleConflictReplayPartialFail.ScopeStrag.DRSCalls
Sonnet 5.5 adaptive (closed)58.334.987.514.850.518.886.987.554.951.4
GPT-6 Sol high (closed)41.729.265.212.535.718.884.493.847.655.7
Sonnet 5.5 thinking-off (closed)45.023.055.023.546.815.079.787.546.945.5
GPT-6 Sol none (closed)19.811.544.318.827.410.063.593.836.137.5
Gemini 3.8-flash (closed)34.419.613.36.27.15.029.212.515.9119.3
qwen3.8-27B (open)35.910.251.225.035.712.080.233.335.573.3
qwen2.5-72B-AWQ (open)4.24.66.718.80.06.010.010.37.628.8
llama3.3-70B (open)0.06.30.012.50.04.02.50.03.27.0
llama3.1-8B (open)0.00.05.00.00.00.00.00.00.644.1

Table B. Thinking vs. non-thinking

PairConfigurationDRSΔ\DeltaDRSCallsΔ\DeltaCalls
Sonnet 5.5thinking on54.9—51.4—
thinking off46.9−8.045.5−5.9
GPT-6 Solreasoning high47.6—55.7—
reasoning none36.1−11.537.5−18.2

Table C. Parameter count (open-weight)

ConfigurationParametersDRSCalls
llama3.1-8B8B0.644.1
qwen3.8-27B27B35.573.3
llama3.3-70B70B3.27.0
qwen2.5-72B-AWQ72B7.628.8
AuthorsOpenReview ↗

Follow-Up: Seeking Further Feedback

Dear Reviewer 7Vb3,

We sincerely appreciate the time and effort you have devoted to reviewing our submission. We have carefully addressed all the points you raised in our rebuttal. As the discussion period is drawing to a close, we would be grateful if you could kindly let us know whether our responses have resolved your concerns, or if there are any remaining questions we can clarify.

We are eager to continue the discussion and will respond promptly to any further comments.

Thank you again for your valuable input.

Best regards,

Authors of Submission 21225

Reviewer_TbRAOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark designed to evaluate how LLM agents handle asynchronous subagent results while continuing task execution. Unlike existing benchmarks that focus primarily on final task success, Async-RBench assesses whether agents correctly perform correct action when facing asynchronous events, including plan updating, valid work preservation, unnecessary actions avoidance etc.

The benchmark consists of 200 executable tasks across eight asynchronous scenarios. The tasks are sourced from different existing benchmarks including Terminal-Bench, Gaia2, OSWorld, SWE-bench, and MultiAgentBench. It introduces a metric, Dynamic Replanning Score (DRS) to evaluate both response processes and outcomes. The paper shows that agents perform poorly on asynchronous coordination, achieving DRS scores of 19.2–45.1 (out of 100), particularly struggling with duplicate reports and subagent failures.

Critical strengths
  • The paper identified an underexplored challenge in LLM agent swarming mechanism, handling asynchronously arriving subagent results while task execution continues. Unlike existing benchmarks that primarily focus on final task success, Async-RBench is more fine-grained and explicitly evaluates the exact agent actions on different asynchronous events.

  • The paper introduces a controller to manage the event delivery, creating asynchronism in a verifiable and deterministic way. The experiments across nine LLM agents provide useful insights into current limitations in asynchronous task execution.

Critical weaknesses
  • The core metric, DRS, which is defined as DRS=100×P+Q2DRS=100\times\frac{P+Q}{2}, weighs the same on process score and final state score. The two should have high correlation by intuition (more correctly handled steps, higher chance the final state is correct). Also, the final state often correlates with the task success rate. The above three scores (P, Q, task SR) should be reported separately with a few correlation analysis and case studies.
Questions for authors
  • The tasks are sourced from other benchmarks including OSWorld. GUI tasks have particularly complicated environment state and executor (VMs). How the progress/state on different subagents (separate workspaces according to the paper) was merged into main deliverables/agent? A few case studies would help.
  • The paper mentions using invalid variants to validate the verifiers but provides limited details on how these variants are constructed. Are they manually designed or automatically generated?
  • The implementation of the task-specific verifiers is not sufficiently clear. While the paper describes ERC-based scoring and reports high agreement with human judgments, it remains unclear how individual scoring points are reached. Are these checks implemented using deterministic rules, execution-based tests, or LLM-based judgments?
8 comments

Dear Reviewer TbRA,

We sincerely thank you for the time and effort you devoted to reviewing our work, and for your valuable comments. We respond to each point below.

We have carefully addressed all of your questions (W1, Q1–Q3) and indicated the corresponding changes to the paper in each response. Your comments helped us present the scoring design and the sub-agent merging mechanism more clearly. If these responses and revisions have resolved your main concerns, we would be grateful if you would consider re-evaluating your assessment of our paper; if any questions remain, we would be glad to discuss them further.

W1. Process and outcome scores are equally weighted; are P, Q and task success redundant?

We disagree that P and Q are redundant, and we show that the ranking is insensitive to the weights. Q is not a second kind of “task success” measure; it is the fifth requirement of the same event-response contract Γi=(Ci,Ri,Fi,Vi,Oi)\Gamma_i=(C_i,R_i,F_i,V_i,O_i) (§3.2): C/R/F/V are evaluated over the response interval, and O is evaluated on the terminal state within the event's scope of impact. S is reported separately because it measures end-to-end success, which answers a different question.

The two components can separate in two directions:

  • High Q, low P: the terminal state is correct, but the process overwrote work that should have been preserved or performed a forbidden rollback.
  • High P, low Q: the response actions were reasonable, but the required terminal state was not reached.

Our additional ablation confirms this separation. We vary w(Q)w(Q) in (1−w)P+wQ(1-w)P + wQ and recompute the full ranking (audited Async runs, baseline w=0.5w=0.5):

Table 1. P:Q weight sweep (audited Async runs, baseline w=0.5w = 0.5)

w(Q)w(Q)Kendall τ\tauRank 1Max rank change
0.00.79GPT-5.6 Terra2
0.10.93GLM5.3 Flash1
0.21.00Claude Sonnet 50
0.31.00Claude Sonnet 50
0.41.00Claude Sonnet 50
0.51.00Claude Sonnet 50
0.61.00Claude Sonnet 50
0.71.00Claude Sonnet 50
0.81.00Claude Sonnet 50
0.90.93Claude Sonnet 51
1.00.93Claude Sonnet 51

For w∈[0.2,0.8]w\in[0.2,0.8] the full ranking is unchanged (τ=1\tau=1). Even at the extremes, τ\tau stays at or above 0.79, no system moves by more than two places, and Claude Sonnet 5 ranks first for all w≥0.2w\ge0.2.

Q1. For GUI/OSWorld tasks, how is sub-agent progress merged into the main deliverable?

To clarify: tasks derived from OSWorld are modeled as executable desktop-state contracts, not live GUI sessions. This is a deliberate control: the benchmark tests whether the main agent can judge and replan after asynchronous results arrive, not GUI operation skill. Bringing a GUI executor in would mix differences in operation ability into the comparison and make them impossible to attribute. The native desktop path is used only for independent environment qualification.

Sub-agent workspaces are isolated from one another, and results enter the main agent through two channels only; the merge is performed explicitly by the main agent.

Table 2. The two channels through which sub-agent results enter the main agent

StepMechanismDecision maker
IsolationEach Child gets a new, independent container created from a snapshot of the main containerFramework
Result intakeAfter gateway validation, the controller releases results according to the planController
File mergeAfter the main agent confirms, it calls promote_child_path to copyMain agent
Artifact commitcommit_artifact, with lineage containing only accepted resultsMain agent
ClosureRun verify_current_state before finishMain agent
ScoringScore after freezing the main container snapshotEvaluator

Every step leaves an auditable record: release time and sequence number, promotion record and file hash, commit lineage and verification record. There is no automatic merge or three-way merge. Promotion is an overwrite, and conflicts must be handled explicitly: each case declares in advance which results are “superseded” and which are “authoritative”; when an authoritative result arrives it invalidates the related artifacts, and the main agent must recommit and reverify, otherwise the lineage check loses points.

Case 1: osw-cross-app-artifact-19cdaed8e8 (AWS invoice archiving and statistics).

  • Sub-agent output: three workers operate in isolated containers — locating the email and attachment, checking the filename pattern and statistics columns, and extracting invoice fields; only the authoritative worker receives the event payload.
  • Merge the main agent must perform: accept the authoritative result, promote the artifact into the main workspace (or regenerate it from its content), rebuild the receipt and the statistics row, bind the receipt hash in the decision manifest, then commit and reverify.
  • Verifier checks: the receipt path matches the archive directory, the amount and fields are correct, the receipt hash in the manifest matches the artifact, and closure_complete is true.
  • Mechanism illustrated: sub-agent artifacts do not appear in the main workspace automatically. If the main agent finishes its work only inside the child container without promoting or redoing it, the terminal-state check fails — this is precisely what “merge” means in this benchmark.

Case 2: osw-cross-app-artifact-8e188f4a86 (row-wise merge of two tables).

  • Sub-agent output: four workers respectively extract the columns of the first table, extract the columns of the second table, perform the row-wise merge, and open Calc to cross-check; the high-priority worker is authoritative for the merged result and overrides the low-priority draft.
  • Merge the main agent must perform: recognize the supersession relation between the two results, promote the authoritative result, and recheck the row count, boundary values and row order.
  • Verifier checks: the output has 5001 rows, boundary values and row order are correct, the output path follows the convention, and opened_in_calc = true.
  • Mechanism illustrated: when a low-priority result arrives first and has already been written, the authoritative result must replace it; conflict handling is the main agent's responsibility, constrained jointly by lineage and terminal-state checks.

Q2. Are the invalid variants designed manually or generated automatically?

They are designed manually by the case author; scripts only apply the designs. This is deliberate: only minimal semantic errors that target a specific requirement make the point of failure attributable. Random or templated mutations often change only formatting or irrelevant fields, so even when they are rejected they do not show that the verifier actually checks the corresponding requirement.

Construction has three steps: list the ERC requirements; write, for each requirement, a minimal error that changes only one thing; and let the build script apply it to the terminal state of the reference solution (Oracle) and declare the scoring points that must fail. Taking gaia2-stockholm-moveout as an example:

  • corrupt-event-stream-receipt replaces stream_revision with 64 zeros and must make sm.eval.monitor_report_exists fail, testing whether the verifier binds to the real authoritative version rather than accepting any non-empty string.
  • break-saved-list-closure deletes the last listing in the list and must make sm.closure.final_saved_closure fail, testing whether the verifier checks semantic completeness and not merely file existence.

A variant counts as detected only if all declared scoring points fail; under the same verifier version, the reference solution and an independent equivalent solution must pass. Both variants of this case are detected (2/2). Such positive/negative contrast also exposes scoring points that “only reward positive examples without penalizing negative ones” — which is exactly how we test the discriminant validity of DRS. The manual design of the invalid variants is now documented in the appendix.Appendix E.1 gives the three-step construction procedure, two complete examples and the detection criterion.

Q3. Are scoring points produced by deterministic rules, executable tests, or LLM judgment?

The publicly released verifiers are fully deterministic and do not rely on model judgment. The computation has three layers.

1. Evidence collection. At the end of a run, the participant container is committed as a frozen snapshot, and the evaluator retains the event log, which records every presentation, confirmation, promotion, commit and reverification.

2. Scoring-point judgment. Two kinds of deterministic judgment operate on the above evidence:

  • Semantic points run hidden pytest inside the snapshot. The tests re-derive the expected state from the public inputs and compare content and SHA-256; a point passes only on exact agreement.
  • Control points are predicates over the event trace. For example, wait_for_authority requires: (i) the authoritative completion ID appears in Main's consumed set; (ii) the first commit touching the affected artifact occurs after that presentation sequence number; (iii) the commit lineage contains that completion ID.

3. Aggregation. The applicable Change / Preserve / Forbid / Verify points are averaged to give P; O is evaluated once on the terminal snapshot; DRS = 100·(P+Q)/2; S is computed separately from terminal acceptance.

Complete example (gaia2-stockholm-moveout). The ERC requires the final list to equal the corrected initial list plus all matching late additions. The semantic point sm.closure.final_saved_closure rebuilds the expected list from /app/task_file/ and compares it with /app/output_data/saved_list_final.json; deleting the last entry makes it fail — this is exactly the break-saved-list-closure variant. On the dynamic side, sm.cf.wait_authority fails any run that resubmits the list before the authoritative result is presented, even if the final file happens to be correct, because the commit sequence number and lineage do not satisfy the predicate.

The 96.6% figure comes from a sampled audit of these automatic judgments (405 runs drawn from the 200 tasks, judged item by item by humans), which checks for systematic misjudgment rather than being the scoring mechanism itself. All disagreements were traced to infrastructure failures, and no task-design problems were found.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Manuscript Revisions and New Experiments

We sincerely thank the Area Chairs and all three reviewers (TbRA, 7Vb3, 8aXd) for their constructive feedback. We are encouraged that reviewers recognized the importance of evaluating how agents handle asynchronously arriving sub-agent results, the practical value of the eight scenarios and the ERC-based process-level evaluation, and the executable, controlled evaluation framework of Async-RBench.

To thoroughly address all shared questions and suggestions, we have uploaded a revised manuscript (all updates highlighted in blue), featuring six major additions:

  1. Metric Validity: P/Q Separability and Weight Robustness (TbRA, 8aXd): We show that P (response process: C/R/F/V) and Q (terminal state O) are separable (high-Q/low-P and high-P/low-Q runs both exist) and that rankings are insensitive to the P:Q weight. Sweeping the weight from 0 to 1, the full ranking is unchanged (Kendall τ = 1.00) over the intermediate range 0.2–0.8. Even at the extremes τ ≥ 0.79, no system moves more than 2 places, and Claude Sonnet 5 ranks first for almost all weights (Table 1 in our response to TbRA).
  2. Ranking Drift Across Scenarios and vs. Task Success (7Vb3, 8aXd): A two-way variance decomposition of the 9×8 DRS matrix attributes 77.8% of variance to scenario difficulty, 12.6% to system, and 9.6% to system×scenario interaction. After removing the scenario main effect, the system ranking is unchanged (τ = 1.00). The mean pairwise τ between scenario rankings is 0.42, and DRS vs. S rankings diverge (e.g., Qwen3.7 Plus is 3rd on DRS but 7th on async success), supporting that DRS and S measure different things (Section 4.3; Tables 3–4 in our response to 7Vb3).
  3. Open vs. Closed Source, Thinking vs. Non-thinking, and Model Size (7Vb3, 8aXd): We add grouped results for 9 additional configurations. Closed-source configurations average DRS 40.3 vs. 11.7 for open-source. Disabling thinking lowers DRS by 8.0 (Sonnet 5.5) and 11.5 (GPT-6 Sol), with fewer tool calls. Among open-weight models, DRS is not monotonic in parameter count (8B: 0.6, 27B: 35.5, 70B: 3.2, 72B: 7.6). We also report per-scenario DRS and mean tool calls, showing that DRS does not reward activity (Tables A–C and S1 in our responses to 7Vb3 and 8aXd).
  4. Scope of the "Asynchronous" Claims (7Vb3): We clarify that the benchmark controls and studies asynchronous result delivery (synchronous joining is only a control baseline), not full asynchronous execution with scheduling, preemption or shared-state concurrency. We added a scope statement in Section 3.1, and the Table 3 caption, Section 4.4 and Appendix C now use "asynchronous result delivery" (plus a scope paragraph in Section 4.1).
  5. Task Construction, Answer-Revealing Instructions, and Solvability (8aXd): We confirm that 28/200 (14%) tasks had instructions that state the expected decision, and report a paired original-vs-neutral comparison on these cases. Removing the stated decision leaves task success unchanged (5/28 vs. 5/28) and changes mean DRS from 0.120 to 0.074, while all 28 cases remain delivered and scored; the revision removes only the expected decision. We clarify that the 47-task construction set is disjoint from the 200 evaluation tasks, and that registry labels are legacy labels. All 200/200 tasks are solvable under pass@4 (186 via existing reference executions, 14 supplementary), and Oracle scores of 96.4–98.8 are explained (Section 4.3; per-case records construction-records-200.csv).
  6. Verifier Transparency, Sub-agent Merging, and Reproducibility (TbRA, 8aXd): We document that the verifiers are fully deterministic (hidden pytest on frozen snapshots for semantic points; trace predicates for control points), with the 96.6% human audit serving only as a check for systematic misjudgment. Invalid variants are manually designed (three-step construction, two worked examples, detection criterion; Appendix E.1). For OSWorld-derived tasks, we explain the explicit promote/commit/verify merge mechanism with two case studies (Tables 2 in our response to TbRA). We describe two reproducibility paths (public framework, 200 task packages and leaderboard; version-bound judge via application).

Please see our individual responses below for detailed results and discussions.

Dear Reviewers,

I hope this message finds you well.

I would like to remind you that we are currently in the rebuttal phase of the paper review process. During this stage, authors may respond to the comments and feedback you provided. If you receive any responses from the authors, please consider providing your feedback to ensure a smooth review process.

Thank you for your hard work and dedication throughout this review process. Your insights are invaluable in enhancing the quality of the submissions, and we truly appreciate your involvement.

If you have any questions or need further assistance, please do not hesitate to reach out.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.