- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Async-RBench, a benchmark for evaluating how LLM agents handle asynchronously arriving subagent results during ongoing task execution. It contains 200 executable tasks adapted from five existing benchmarks and organized into eight asynchronous scenarios. A scenario controller regulates result delivery, while Event Response Contracts (ERCs) specify required changes, preserved work, prohibited actions, verification requirements, and expected outcomes. The proposed Dynamic Replanning Score (DRS) evaluates event-level response quality separately from end-to-end task success. The authors evaluate nine agents under asynchronous and batched result delivery, with three repetitions per task and mode. They report DRS scores ranging from 19.2 to 45.1, lower task-success point estimates under asynchronous delivery, and modest improvements from mandatory plan rebuilding. They also conduct Oracle/Frozen comparisons, sampled human audits, and reference-solution and verifier checks to support the evaluation design.
Critical strengths
- A meaningful and practically relevant evaluation problem. The paper identifies an important gap in current agent evaluation. Final task success may fail to capture whether an agent correctly incorporates late-arriving information, preserves unaffected work, avoids unnecessary side effects, and revalidates previously completed steps. The eight asynchronous scenarios cover several meaningful situations in multi-agent execution. The distinction between event-level response quality and final task success is well motivated and potentially useful for future research.
- A substantial executable evaluation framework with encouraging validation efforts. The authors construct 200 executable tasks, provide a controlled result-delivery mechanism, and evaluate nine agents under a relatively consistent execution protocol. The released task metadata provides identifiable upstream sources, and the paper reports reference executions, equivalent-solution checks, invalid variants, human auditing, and Oracle/Frozen comparisons. These are relevant components of a rigorous benchmark-validation pipeline and provide a useful foundation for further evaluation.
Critical weaknesses
- Insufficiently justified benchmark curation and concerns about construct validity. For a benchmark paper, I consider the scientific rigor of dataset curation and validation to be a fundamental requirement. In particular, a reliable benchmark should establish clear provenance for every instance, justify why each task and scenario was constructed, ensure that the intended capability is actually being measured, and validate the resulting tasks before using them to evaluate other systems. While Sections 3.3 and A describe a general construction pipeline, the instance-level documentation I inspected provides limited justification for individual design decisions. The released provenance records identify the original benchmarks and task IDs, which is useful, but they do not sufficiently explain why each source task was selected, why a particular asynchronous event was introduced, how its timing and dependencies were justified, or how the corresponding ERC requirements were determined. More importantly, I found concrete examples of participant-visible instructions that explicitly disclose the expected decision. In mab-conflicting-specialist-results-ce3ff5b928, adapted from MultiAgentBench, the task extension directly instructs the agent to select INSERT_LARGE_DATA as the final database diagnosis. In mab-dependency-unblock-1c96d4414d, adapted from Gaia2, the instructions specify which revision should be adopted, which late receipt should be rejected, and the exact replacement scheduling actions. These cases raise a substantive construct-validity concern. If the expected decision is already provided in the task instruction, success may primarily reflect compliance with an explicitly specified resolution, rather than the ability to independently determine the validity, authority, and consequences of asynchronously arriving evidence. Such guidance may be acceptable for some execution-oriented tasks, but its compatibility with the paper's claimed evaluation objective needs to be established. I would also like clarification about development-time exposure of evaluation cases. The released registry retains legacy calibration and development labels for instances included in the final 200-task evaluation set. Although the paper reports a separate construction set, the exposure and revision history of the final cases is not sufficiently clear from the available documentation. These concerns matter because executable task validity alone does not establish benchmark validity. The authors should demonstrate that the curated cases are correctly specified, representative of the intended capability, sufficiently diverse, and free from answer-revealing instructions that compromise the intended evaluation.
- Insufficient per-instance Oracle and verifier validation, limiting confidence in score reliability. A high-quality executable benchmark should demonstrate that every included task has a feasible reference execution, that correct outcomes and alternative valid solutions are accepted, and that invalid behaviors are reliably rejected. For agent benchmarks, this requires checking both final artifacts and the relevant execution trajectories, including the evidence supporting changes, preservation, prohibited operations, and revalidation. The paper makes useful efforts in this direction, including reference and equivalent solutions, invalid variants, and a human audit reporting 96.6% initial agreement on scored items. However, the evidence is primarily aggregated, and the paper does not provide a complete per-instance validation report establishing the acceptance behavior of all 200 tasks. The Oracle results in Figure 3 are particularly important. Despite executing reference recovery programs with access to the event requirements, the Oracle achieves DRS values of 96.4–98.8 rather than 100. These aggregate values do not by themselves establish the presence or number of broken tasks, since DRS is a continuous process-and-outcome metric. Nevertheless, every unexplained Oracle score deficit deserves investigation. The authors should identify which cases and scoring components account for the missing credit, distinguish reference-execution limitations from task or verifier defects, and establish whether every included instance has a reference execution that satisfies its complete acceptance criteria. The discriminative validity of DRS also needs stronger support. Section 3.2 assigns zero when the main agent terminates before the target result is presented, which combines failure to encounter an event with incorrect handling after receiving it. Differences in source-task difficulty, event exposure, and the amount of guidance in task instructions may therefore influence the reported scenario scores and model rankings. Oracle/Frozen separation is encouraging, but it is insufficient on its own to establish that the benchmark specifically measures asynchronous response competence. Finally, the released public cases do not include the original paper's complete ERC truth and hidden verification materials, and the documented newer runtime judge does not directly reproduce the historical paper scoring. A reproducible route for validating the original results is important if the benchmark is intended to become a reliable community evaluation resource. My concern is the downstream consequence of potential benchmark defects. If a task contains an incorrect ground truth, an overly restrictive verifier, or answer-revealing instructions, researchers may attribute misleading evaluation outcomes to model capability. Such issues should be identified, corrected, and revalidated before the benchmark's scores are treated as reliable evidence.
Questions for authors
Q1. Per-instance curation and provenance.
Could the authors provide a complete construction and validation record for all 200 tasks? Ideally, this would include the original source task, the transformation performed, the rationale for the chosen asynchronous scenario and release conditions, the ERC requirements, and the acceptance results. How were the scenarios selected, and how was the correctness of each transformed task independently established? Please also clarify which evaluation cases were accessed or revised during development.
Q2. Answer-revealing instructions and construct validity.
For cases such as mab-conflicting-specialist-results-ce3ff5b928 and mab-dependency-unblock-1c96d4414d, why do the participant-visible instructions explicitly specify the expected decision? How many cases contain similar outcome-specific guidance? Can the authors audit the complete benchmark for this issue and demonstrate, for example through an ablation using more neutral instructions, that the benchmark still measures independent asynchronous evidence handling?
Q3. Complete Oracle and verifier validation.
Can the authors report the per-instance Oracle validation results across all 200 tasks, including full task-success status, ERC component scores, and the exact causes of any failures or missing credit? In particular, what accounts for the Oracle DRS values below 100 in Figure 3? Were all tasks independently verified to have at least one execution that passes every final acceptance test and applicable event-response requirement? How many equivalent correct solutions and intentionally invalid variants were tested, and what were their acceptance and rejection rates?
Q4. Reliability and discriminative validity of the evaluation.
Can the authors report target-event presentation rates, DRS conditional on successful presentation, and the C/R/F/V/O score breakdown by model and scenario? How stable are model rankings when controlling for source-task difficulty and task-specific guidance? Can independent researchers reproduce or validate the exact scoring criteria used for the reported results? These details are important for determining whether the benchmark reliably measures asynchronous response competence.
11 comments
(3) Relationship between the construction set and the evaluation set. The construction set and the evaluation set are two separate batches of tasks: the 47 construction tasks do not overlap with the 200 evaluation tasks and do not participate in any reported result. The calibration/development/test labels in the registry are historical labels from the old repository policy, recording whether a case already had runs or paired evidence in the early repository; they are kept only for compatibility and optional diagnostic filtering, do not define the paper's data split, and all 200 cases participate in the evaluation.
W2 & Q3 & Q4. Solvability of the 200 tasks, rationale for the DRS design, and reproducibility
Q3. Can the authors report the per-instance Oracle validation results across all 200 tasks ...? What accounts for the Oracle DRS values below 100 in Figure 3? Were all tasks independently verified to have at least one execution that passes every final acceptance test ...?
Q4. Can the authors report target-event presentation rates, DRS conditional on successful presentation, and the C/R/F/V/O score breakdown by model and scenario? ... Can independent researchers reproduce or validate the exact scoring criteria used for the reported results?
(1) All 200 tasks are solvable. Here we do not discuss whether the acceptance records are complete, but answer the solvability question directly: for all 200 tasks we apply the same criterion — at most 4 runs, and if any one run passes all terminal-state and event requirements, the task is judged solvable (pass@4 = 1). 186 tasks were passed on the first execution of the reference solution, with the declared invalid variants failing as expected; of the other 14 supplementary tasks, 11 were completed by a model within 4 runs, and 3 were passed by the scripted reference solution and an independent equivalent solution. Hence 200/200 meet this criterion, and there is no case of “passes acceptance but is unsolvable”.
Table 5. Solvability of the 200 tasks (pass@4)
| Task set | Passed | pass@4 |
|---|---|---|
| 186 tasks with an existing reference execution | 186 | 186/186 |
| 14 supplementary tasks | 14 | 14/14 |
| All 200 evaluation tasks | 200 | 200/200 |
The revised version will release a per-case table listing the pass@4 result, reference solution, equivalent solution, invalid variants, terminal-state points and trace points.(2) Why the Oracle scores of 96.4–98.8 fall short of 100. The Oracle is not a “perfect trajectory”, but a reference recovery program written by humans at design time; it is executed in real runs and scored on the actual execution result, not given full marks automatically because of its “reference” status. It starts from each model's own pre-event state, can only perform the operations that state allows, and does not automatically produce the evidence that some components require (for example, an explicit post-event reverification record), so scores below 100 are normal. Precisely because the starting points differ, the same reference program obtains different Oracle scores on the three models (98.8 / 98.5 / 96.4) — which shows that DRS evaluates “what was done in this actual state” rather than assigning a fixed score to a script; this is itself evidence that DRS is discriminative. Solvability remains defined by the pass@4 criterion in (1), which all 200 tasks satisfy.(3) Why DRS is valid, and how to reproduce it. DRS splits event response into C/R/F/V (the response process) and O (the terminal state within the event's scope of impact): P is the mean of the first four, Q is O, and the two are weighted equally because they answer the two halves of the same contract; S is reported separately because it requires the whole task to be completed. This design is not arbitrary, and the experiments above give direct evidence:- Insensitivity to weights. When the P:Q weight is varied, the full ranking is unchanged for (), and under extreme weights stays at or above 0.79 (Table 1).
- P and Q are separable. The audited runs contain both “high Q, low P” and “high P, low Q” cases, showing that the two are not two measurements of the same quantity (see W1 and Q1–Q2 above).
- Cross-scenario differences come mainly from difficulty, not metric noise. The variance decomposition of the 9×8 DRS matrix is 77.8% scenario, 12.6% system and 9.6% interaction; after removing the scenario main effect the overall ranking is unchanged (, Table 4). This also addresses the question of whether the ranking is affected by the task mix.
- The zero-scoring rule avoids rewarding avoidance. A run is scored zero if the target result is never presented, because without the event there is no response to evaluate; we also report the presentation rate and conditional DRS so that “did not encounter” and “handled wrongly” can be distinguished.
- The instruction issue does not affect the ranking. The paired before/after comparison on the 28 cases with hint-bearing instructions reports the change in DRS and task success for each affected case (see point (2) of W1 and Q1–Q2 above). For reproducibility we provide two paths. First, the public repository contains the framework, the 200 participant-visible task packages, the registry and the experiment manifest (binding the model identifier, protocol version, evaluation-contract digest and judge digest), so any researcher can run a system under test locally with the same framework. Second, the public site shows the paper's results, the task catalog and a community leaderboard, and provides a submission entry: independent researchers who complete a run submit a run receipt through the site, which, after review by maintainers under a fixed procedure, enters the community leaderboard; paper results and community results are displayed separately and never overwrite each other. The ERC ground truth, the hidden verifiers and the historical judge are not distributed with the public package, because they contain answers and judging logic, and releasing them would contaminate the benchmark and invalidate future evaluations; teams that need to reproduce can obtain a version-bound judge through an application channel, and the site's submission records and release-manifest digests are used to check version consistency.A pass@4 solvability conclusion has been added to §4.3.
Dear Reviewer 8aXd,
Thank you for your rigorous review. Your W1/W2 and Q1–Q4 point to the same body of evidence, so we answer them together below.
We have carefully addressed all of your questions (W1/W2 and Q1–Q4), and added per-task evidence and the corresponding paper revisions. Your rigorous comments made our construction rationale, solvability evidence and scoring design clearer and easier to verify. If these responses and revisions have resolved your main concerns, we would be grateful if you would consider re-evaluating your assessment of our paper; if any questions remain, we would be glad to discuss them further.
(1) Source selection and transformation method. We chose Terminal-Bench, Gaia2, OSWorld, SWE-bench and MultiAgentBench not as an arbitrary combination, but because together they satisfy the four necessary conditions for constructing asynchronous replanning tasks. First, the tasks are complex enough to split naturally into multiple parallel, interdependent sub-workflows — the premise for results arriving while work is in progress; source tasks that are too simple finish before the event arrives, so the interruption cannot occur. Second, the initial environment is executable and reproducible, so the states before and after the event can be genuinely recorded. Third, the acceptance criteria are public and executable, so “what was changed, what was preserved, and whether it was reverified” can be checked rather than judged subjectively. Fourth, source versions and task IDs are traceable, allowing others to check the correspondence before and after transformation. The transformation follows the principle of “keep the source task, add only the collaboration and event layer”: the source task's goal, initial environment and acceptance tests are unchanged, and what is added are the sub-agent workflows, result version relations, release conditions and the ERC. This preserves ecological validity while isolating asynchronous replanning, avoiding mixing GUI operation or basic coding ability into the measured ability.
We have built a per-case construction and acceptance record for all 200 tasks, with fixed fields: source task and version, transformation, scenario rationale, release conditions and timing, ERC requirements, workflows and artifacts, hidden checks and reverification anchors, reference and equivalent-solution results, and execution and detection of invalid variants. A metadata table without answers or reference solutions will be released with the public repository (paper_artifacts/construction-records-200.csv); the full records, including reference solutions, hidden checks and invalid variants, are provided as supplementary material, with each of the 200 cases having a reference solution, a verifier, an invalid variant and an equivalent solution.
(2) Why the instructions state the event. We confirm that 28 cases (14%) have participant-visible instructions that reveal the expected handling or outcome. The reason is that these cases come from open-ended tasks (especially multi-agent and database-diagnosis types) whose source tasks allow multiple reasonable forms of delivery. Without stating the event payload, the expected terminal state and the delivery path, a run often cannot even enter the intended setting — the main agent may choose another equally reasonable but incomparable approach, or produce no checkable artifact, so the event can be neither observed nor scored. These statements were originally intended to make the tasks executable, the events observable and the acceptance decidable, not to lower the difficulty. We run a paired before/after comparison on these 28 cases: the same systems and the same number of repetitions, once with the original instructions and once with neutral instructions, reporting the change in DRS and task success for each affected case.