acceptodds
Under review as a conference paper at ICLR 2027

Communicate While Thinking with Inspected Peer Assistance During Ongoing Multiagent Reasoning

Abstract

Multi-agent collaboration can improve the reasoning of large language models (LLMs) through external insights. When peer feedback arrives only after an agent completes its local chain-of-thought (CoT) reasoning, intermediate errors may already have shaped later work, and correction relies on post-hoc revision. We propose CoTIE (CoT-State Inter-Agent Exchange), a collaboration mechanism that brings targeted peer assistance into unfinished local reasoning. In CoTIE, requests are formed from the agent's current state, replies are inspected in a separate model call, and only accepted information is added before the agent continues the same local reasoning. Naive CoTIE leaves peer requests and reply assessment to ordinary tool calls. Full CoTIE adds dedicated procedures for request triggering, state disclosure, peer selection, and record admission. Across six task scenarios with DeepSeek-V3.2 and Qwen3-235B-A22B, Full CoTIE achieves the highest mean score among the compared methods. Full CoTIE improves scores over Naive CoTIE by 1.8–7.1 points at 17.5–40.0% more tokens. Full CoTIE uses fewer tokens than the native workflow of MultiAgentBench (MAB) in all MAB settings and more tokens than output-level message passing in five of six general reasoning settings. In a controlled MAB Coding experiment with DeepSeek-V3.2 and fixed accepted peer information, delivery during unfinished reasoning scores 12.6 points higher than delivery at the start of revision, after candidate submission or the drafting limit, under matched stage budgets. Early delivery also uses 19.2% fewer tokens from the shared checkpoint through final submission. Regenerating the peer reply at revision narrows the score gap to 11.3 points. Manual review of continuations from 50 matched timing checkpoints finds fewer code edits based on incorrect assumptions and less error-related rework for early than late delivery. The code is available at https://anonymous.4open.science/r/CoTIE-review-ICLR/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Multi-Agent Reasoning, Inter-Agent Communication, Communication Timing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hg2SOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CoTIE, a mechanism for obtaining peer assistance before an agent completes its local solution. Full CoTIE adds explicit request triggering, state disclosure, peer selection, and isolated inspection, admitting only selected reply records into the caller’s continuation. Evaluation spans six task scenarios and two backbones, with quality–cost comparisons and component ablations. A controlled coding experiment delivers identical accepted information either during drafting or at revision. Earlier delivery improves the composite score by 12.6 points while reducing subsequent token use, and independent manual annotation finds less erroneous work and rework.

Critical strengths

The timing experiment is the strongest contribution. Freezing the assistance packet, restoring shared checkpoints, disabling further exchanges, and matching stage budgets substantially reduces confounding between information quality and delivery time. The regenerated late-feedback condition further tests whether the result merely reflects stale advice. Independent, condition-blinded annotation of matched continuations provides complementary process evidence, making the explanation in terms of reduced rework more convincing than an accuracy comparison alone. The main evaluation also covers materially different activities, including coding, question answering, research, and database work, with explicit accounting for assistance costs.

Critical weaknesses

The evaluation is not fully independent of configuration selection. Appendix E states that the requery limit was selected by comparing settings on the evaluation tasks, and Appendix J uses the reported MAB results to justify the default. Since this parameter materially changes both quality and token use, the main results partly reflect selection on the evaluation set. This weakens the evidence that the reported operating point transfers to unseen tasks.

The central timing finding has narrower support than the overall framework evaluation. It is tested only on DeepSeek-V3.2 MAB Coding, using checkpoints drawn from Full CoTIE trajectories. It therefore establishes a benefit conditional on states reached by that workflow and the specified drafting–revision policy. The second backbone and broader task suite do not independently validate this mechanism. Moreover, communication occurs between model calls, so the contribution concerns intervention during an unfinished multi-call workflow rather than interruption within a single generation.

Questions for authors

For clarification of evaluation independence, which parameters besides the requery limit were selected using evaluation outcomes? Is there an existing evaluation on tasks excluded from configuration selection, or evidence that the comparative conclusions remain stable across the tested limits?

For clarification of the timing result, how were checkpoints distributed across source tasks, and were they selected independently of assistance usefulness as well as branch outcomes? Does task-clustered uncertainty preserve the Early–Late difference? These questions concern the submitted evidence; extending the timing experiment to additional backbones and task types would be future work.

4 comments
AuthorsOpenReview ↗

Response to Reviewer hg2S (1/2)

We thank Reviewer hg2S for the thorough and constructive review and for recognizing the timing experiment as the strongest contribution. Your questions led directly to the new analyses below.

Q1. Evaluation independent of configuration selection

Parameters selected on evaluation outcomes. Only L. New Table 11 lists every setting. K = 5, L_S = 512, L_C = 8,192, the 512-token inspection summary, disclosure A, the Qwen3-8B inspector, token ceilings, and all baseline settings were fixed before the main runs. The inspector comparison (App. L) did not change the default, and all its paired CIs include zero.

Tasks outside selection. L was compared only on MAB. Full − Naive with task-cluster 95% CIs and Holm correction over 12 contrasts (Table 22):

MATHHumanEvalHotpotQA
DeepSeek4.27 [2.77, 5.81]1.83 [−0.81, 4.67]2.70 [1.50, 3.90]
Qwen32.67 [1.15, 4.21]2.64 [−1.02, 6.30]3.40 [2.19, 4.65]

The difference is positive in 6 of 6 settings and significant after Holm correction in 4. HumanEval is near ceiling.

Selection-aware estimate on MAB (App. J.1). L is chosen for each held-out fold on the other four folds.

Held-out (fixed L = 3)CodingResearchDatabase
DeepSeek3.02 (4.10)3.20 (3.20)7.06 (7.10)
Qwen31.82 (3.20)2.30 (2.30)4.85 (5.20)

All six estimates stay positive. Held-out estimates are 1.1 to 1.4 points lower for Coding and within 0.35 points elsewhere. For Qwen3 Coding and Research the task-cluster CIs include zero, which the Limitations now state.

Stability across limits. Full > Naive in all six settings at L = 3 and L = 5. At L = 2 the budget often runs out before clarification ends. 14.5% to 18.2% of invocations stop with INSPECT still asking for clarification, against 7.0% to 11.1% at L = 3, and Full stays below Naive in 5 of 6 settings (Table 25). The conclusion holds for L ≥ 3.

AuthorsOpenReview ↗

Response to Reviewer hg2S (2/2)

W2. Narrower support for the timing finding

We agree and have scoped the claim.

  • The contribution bullet names the setting.
  • The abstract says help enters between the model calls of a multi-call local solution.
  • The Limitations paragraph (Section 5) and App. N state that the result holds for DeepSeek-V3.2 MAB Coding, is conditional on states reached by Full CoTIE and on the drafting and revision policy, and is not tested independently by the second backbone or other tasks. CoTIE intervenes between calls, not inside a single generation.

Q2. Checkpoints and task-clustered uncertainty

Distribution (Table 8). The 150 checkpoints come from 73 distinct source tasks. 27 tasks contribute 1 checkpoint, 25 contribute 2, 12 contribute 3, 8 contribute 4, and 1 contributes 5. Per run there are 48, 51, and 51 checkpoints from 35, 32, and 32 tasks.

Selection. Eligible points are boundaries between caller calls after at least one valid iteration and before the first submit_candidate. We sample at most 4 per task and run without replacement (seed 2718), using only call boundaries, identifiers, and controller state. The rule, seed, and selected points were fixed before any timing request or reply was generated, so packet content and branch outcomes could not affect selection. 24 of 150 packets are empty because INSPECT rejected every record, and all of them are kept.

Empty-packet placebo. On these 24 checkpoints (21 tasks) Early and Late deliver no information. Early − Late = 0.29, 95% CI [−1.27, 1.95]. The gap therefore depends on the delivered information, not on the branching procedure.

Task-clustered uncertainty (Table 9). We resample source tasks with replacement (10,000 resamples) and also fit a mixed model with a random intercept per task.

Early minusΔ scorePaired-run CITask-cluster CIMixed modelΔ tokens, task-cluster CI
Late12.59[10.18, 15.01][10.48, 14.76]12.63 [10.43, 14.82]−24,727 [−37,812, −12,394]
No14.96[9.82, 20.10][12.95, 17.12]14.96 [12.69, 17.23]−35,418 [−46,193, −24,959]
Late-regen.11.32[10.64, 12.00][9.09, 13.67]11.38 [9.04, 13.72]−44,072 [−55,263, −33,245]

The score gap and the token reduction both hold under task clustering. Planning (+0.53) and Task (+0.73) both improve. Executed programs succeed in 61.3% of checkpoints for Early and 38.3% for Late, +23.0 points [11.0, 34.5] (App. F.3).

Extending the timing experiment to more backbones and task types is the next step. Thank you again for the careful review. If our answers resolve your questions, we would be grateful if you would consider raising your score.

Thank you for your response. I will keep my score unchanged.

Thank you for reading our response and for your positive assessment of the work. If any point would still benefit from clarification or an additional analysis, we would be glad to provide it during the discussion period.

Reviewer_pbshOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a multi-agent collaboration framework that enables agents to request peer assistance during ongoing reasoning. The framework introduces mechanisms for request triggering, state disclosure, peer selection, and response inspection. Experiments across multiple benchmarks show improvements in task performance compared to existing baselines, although with additional token costs.

Critical strengths
  • The paper addresses a relevant problem in multi-agent reasoning.
  • The evaluation covers multiple benchmarks and LLM backbones.
Critical weaknesses
  • Insufficient motivation and positioning: The introduction lacks sufficient motivation for the proposed approach. The authors should discuss relevant prior work that has addressed similar problems, identify the limitations of existing approaches, and clearly explain the research gap that this work aims to fill. Without such positioning, it is difficult to assess the novelty and significance of the proposed method. Simply sending the entire related work discussion to the appendix is inappropriate, especially when the paper does not clearly differentiate its contributions from existing approaches.

  • The main text should clearly explain the proposed workflow rather than relying primarily on Figure 1. Furthermore, the term "disclosure A" appears in the figure caption before it is defined, which may confuse readers. Overall polishing in writing is required.

  • There are unclear experimental contexts in the introduction. Lines 81-86 should be revised to provide sufficient context for the experimental setup and task being discussed. Alternatively, these details may be better suited to the experimental section rather than the introduction.

  • The proposed method introduces additional LLM calls for request generation, state disclosure, peer selection, and response inspection, which may incur substantial latency overhead. However, the paper neither reports end-to-end latency nor compares the proposed method with baselines under the same wall-clock time budget. This makes it difficult to assess the practical efficiency of the proposed approach.

Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to Reviewer pbsh (1/2)

We thank Reviewer pbsh for the detailed comments on motivation, presentation, and efficiency, and for noting that the paper addresses a relevant problem and covers multiple benchmarks and backbones. We revised the paper in response to each point.

W1. Motivation and positioning

New Section 2 (Related Work and Positioning) is in the main text. Appendix A keeps the extended version. The introduction now states the argument in three steps.

  • Problem. In multi-call reasoning such as coding, retrieval, or planning, later steps build on an early error, so a late correction must also undo dependent work.
  • Limitations of prior work. Debate, EoT, Self-Refine, and ChatDev review exchange information between completed outputs, so help can only trigger post-hoc revision. The receiver usually consumes peer messages directly, so unsupported advice enters its context. ChatDev's communicative dehallucination asks for details before a response, but not from a specific need in an unfinished state. No prior work tests whether the timing of fixed information matters.
  • Contribution. CoTIE builds requests from the unfinished state, checks replies in a separate call, and admits only selected records. A controlled experiment isolates delivery time.

New Table 1:

ApproachWhen info arrivesFrom caller stateSeparate checkUnit admittedTiming test
Debate, ReConcile, EoTafter completion××whole message×
Self-Refine, ChatDev reviewafter completion××whole artifact×
AgentDropoutV2after generation×✓whole output×
ChatDev dehallucinationbefore response∼×whole message×
CoTIEduring reasoning✓✓reply record✓

AgentDropoutV2 screens finished outputs before they reach other agents. CoTIE controls which requested information enters the caller's own context, record by record.

W2. Workflow and "disclosure A"

  • Section 3.1 now walks through Figure 1 in six steps: bounded steps, trigger, request with operation and help need and disclosure, routing and peer reply, separate INSPECT, and restored context with admitted records only. A requery repeats steps 3 to 5 at most L times.
  • The caption defines disclosure A as the caller's own state entries that it selects as relevant to the help need.
  • Algorithm 1 moved to App. B and Section 3 was tightened. The method is unchanged.

W3. Lines 81 to 86

The paragraph is now self-contained. It explains that MAB Coding is a multi-agent program implementation benchmark and that a checkpoint is a saved unfinished caller state. One peer reply is inspected once and frozen as a packet. Branches differ only in when the packet arrives, Early during candidate generation and Late at the start of revision. Further exchanges are disabled and stage budgets are matched. Details stay in Section 4.3.

AuthorsOpenReview ↗

Response to Reviewer pbsh (2/2)

W4. Latency and wall-clock comparison

New App. M (summary in Section 4.2) reports three measurements.

(a) Extra calls (Table 33). Full CoTIE is invoked 0.16 to 0.53 times per task on general reasoning and 0.62 to 2.62 times on MAB. Each invocation makes about 8 to 11 extra calls. This gives 1.3 to 4.4 extra calls per task on general reasoning and 6.5 to 26.7 on MAB.

(b) Serial wall-clock time (Table 34). Calls run one at a time on a local vLLM server, and time is taken from log timestamps from task start to the saved submission. Median seconds per task for DeepSeek-V3.2:

MATHHumanEvalHotpotQACodingResearchDatabase
Native (MP / MAB)14.68.013.2446.0204.1106.3
Fastest of SC, SR, EoT, AgentDropoutV224.414.627.4796.1405.3301.0
Naive CoTIE13.78.013.2319.5179.480.8
Full CoTIE19.18.819.6383.7220.289.2

Across both models Full takes 1.10 to 1.55 times the median time of Naive. It is slower than native message passing on general reasoning (1.10 to 1.54 times). It is faster than SC, SR, EoT, and AgentDropoutV2 in all 12 settings. It is faster than native MAB on Coding and Database and slower on Research (1.08 and 1.12 times). INSPECT takes 0.09% to 0.43% of serial time (Table 35).

(c) Common wall-clock deadline (Table 36). DeepSeek-V3.2 HumanEval and MAB Coding with one per-task deadline for all methods, 35 s for HumanEval and 700 s for Coding. At the deadline the current candidate is submitted.

HumanEvalMAB CodingDeadline hits, Coding
Native90.6546.0817.0%
Native + SC / + SR92.48 / 94.1141.03 / 40.7294.7% / 94.0%
EoT / AgentDropoutV290.04 / 93.9045.67 / 48.0182.7% / 63.0%
Naive CoTIE94.5160.790.7%
Full CoTIE95.7364.666.3%

Under equal time Full keeps the highest mean on both benchmarks.

The general response lists further soundness checks: selection-aware evaluation, Holm-corrected task-level tests, and execution-based scoring. Thank you again for these comments, which made the paper clearer. We believe the revisions address each of your four points. We would be grateful if you would reconsider your rating, together with the soundness and presentation scores, in light of them.

Thank you again for your review. Before the discussion period ends, we would like to check whether our response and the revised PDF address your concerns. In brief:

  • Motivation and positioning: a new Related Work and Positioning section with Table 1 in the main text, and a rewritten introduction that states the gap.
  • Workflow: a step-by-step walkthrough of Figure 1 in Section 3.1, and the caption now defines disclosure A.
  • Lines 81 to 86: rewritten with the full experimental context.
  • Latency: App. M reports extra calls, serial wall-clock time, and results under a common wall-clock deadline, where Full CoTIE keeps the highest score. We would be grateful for your feedback and, if these changes address your concerns, for a reconsideration of your rating.
Reviewer_wuuqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CoTIE, a protocol for obtaining peer assistance before an agent completes its current local solution. It combines request triggering, selective state disclosure, peer selection, and a separate inspector that admits selected reply records into the caller’s context.

Experiments cover six task scenarios and two language models. Full CoTIE achieves the highest reported mean scores, although it uses more tokens than its naive variant. A controlled MAB Coding experiment delivers the same accepted information at different stages, reporting a 12.6-point gain and 19.2% fewer tokens from the checkpoint to final submission with early delivery. Ablations and manually reviewed coding trajectories further examine selective admission, disclosure, and rework.

Critical strengths
  1. The timing experiment provides a useful controlled comparison.Early and Late receive the same accepted information from shared checkpoints, with further exchanges disabled. The improvement persists against Late-regenerated, and both reported score components improve.

  2. The additional controls support the admission mechanism.The All-accept ablation and Naive plus Self-Refine comparison show that Full's advantage cannot be explained solely by higher token use in these settings .

Critical weaknesses
  1. The selected configuration lacks independent evaluation.Appendices E and J explicitly state that the requery limit L=3L=3 was selected using the reported MAB evaluation tasks. Table 16 shows that this choice materially affects performance. The sensitivity analysis is transparent, but it does not establish that the selected configuration retains its advantage on tasks unused for selection. Comparable configuration-selection procedures for the baselines are also unclear .

  2. The MAB scoring procedure is insufficiently specified. Appendix F.1 defines the Planning–Task composite, and Tables 10–12 helpfully separate its components. However, the exact evaluator configuration, scoring rules, and grader inputs are not sufficiently documented. This matters because the central timing result relies on these scores, whose relationship to successful task execution remains difficult to assess from the paper.

Questions for authors
  1. Can the Full-versus-Naive comparison be confirmed on tasks not used to select LL, with the configuration fixed in advance? Please also clarify how baseline configurations were selected.

2.Which exact evaluator, scoring rules, and inputs produce the MAB Planning and Task scores? Please explain how the chosen composite relates to the original benchmark configuration.

4 comments
AuthorsOpenReview ↗

Response to Reviewer wuuq (1/2)

We thank Reviewer wuuq for the careful reading and for recognizing the controlled timing comparison and the admission controls. Both questions are precise, and we answer them with new analyses.

Q1. Does Full > Naive hold on tasks not used to select L? How were baselines configured?

L was compared on the MAB evaluation tasks and it matters (Table 24, formerly 16). We add three checks.

(a) Tasks not used to select L. L was compared only on MAB. MATH, HumanEval, and HotpotQA ran with L = 3 fixed in advance. Full − Naive with tasks resampled as clusters (100,000 resamples) and Holm correction over all 12 contrasts (Table 22):

SettingFull − NaiveTask-cluster 95% CIHolm p
DeepSeek MATH4.27[2.77, 5.81]0.0001
DeepSeek HumanEval1.83[−0.81, 4.67]0.38
DeepSeek HotpotQA2.70[1.50, 3.90]0.0001
Qwen3 MATH2.67[1.15, 4.21]0.005
Qwen3 HumanEval2.64[−1.02, 6.30]0.38
Qwen3 HotpotQA3.40[2.19, 4.65]0.0001

The difference is significant in 4 of 6 settings. HumanEval is near ceiling for both variants.

(b) Selection-aware estimate on MAB (App. J.1). Task IDs are sorted and shuffled with seed 1729 into 5 folds of 20 tasks, the same for both models, with all runs of a task in one fold. For each held-out fold, L ∈ {2, 3, 5} is chosen on the other four folds as the smallest L within 1.0 point of the best. The held-out fold is then scored.

SettingL per foldHeld-out Full − NaiveTask-cluster 95% CIFixed L = 3
DeepSeek Coding3,5,5,3,33.02[0.17, 7.11]4.10
DeepSeek Research3,3,3,3,33.20[0.82, 5.54]3.20
DeepSeek Database5,3,5,3,37.06[4.63, 10.57]7.10
Qwen3 Coding3,3,5,3,51.82[−0.79, 6.23]3.20
Qwen3 Research3,3,3,3,32.30[−0.44, 4.63]2.30
Qwen3 Database3,3,5,3,34.85[2.59, 7.75]5.20

The difference stays positive in all six settings. Held-out estimates are 1.1 to 1.4 points lower for Coding and within 0.35 points of the fixed-L values elsewhere. For Qwen3 Coding and Research the intervals include zero, which the paper now states.

(c) Why L matters (Table 25). At L = 2, 14.5% to 18.2% of invocations end with INSPECT still asking for clarification. At L = 3 this is 7.0% to 11.1%, and at L = 5 it is 1.5% to 3.4%. Full exceeds Naive in all six MAB settings at L = 3 and L = 5. At L = 2 the budget often runs out before clarification ends, and Full stays below Naive in five of six settings. The paper now documents this minimum budget.

AuthorsOpenReview ↗

Response to Reviewer wuuq (2/2)

(d) Baselines. New Table 11 lists how each setting was chosen. SC (5 samples), SR (at most 4 rounds), EoT (at most 3 rounds, Memory mode), and AgentDropoutV2 were fixed before the main runs and not tuned on evaluation outcomes. We also applied the held-out selection to the baselines (Table 29). The pool is native MAB, Naive CoTIE, SC with 3, 5, or 7 candidates, SR with 2, 3, or 4 rounds, EoT with 1, 2, or 3 rounds, and AgentDropoutV2 with 2, 3, or 4 correction rounds. On the training folds we take the cheapest configuration within 1.0 point of each family's best, then the family with the highest mean.

Selected Full − selected baselineCodingResearchDatabase
DeepSeek1.82 [−1.82, 6.76]4.84 [0.79, 7.21]7.12 [3.65, 11.42]
Qwen31.82 [−0.79, 6.36]3.93 [−1.28, 6.64]4.85 [2.62, 9.71]

The selected Full exceeds the selected baseline in all six settings. The interval excludes zero for DeepSeek Research and Database and Qwen3 Database.

Q2. Evaluator, scoring rules, inputs, and relation to the original MAB

Relation to the original MAB. The original reports Task Score (TS) and Coordination Score CS = (Communication + Planning)/2 separately, plus milestone KPIs. Our composite S = 10 × (Planning + Task) is our own aggregate. Sections 4.1 and F.1 now say this, and Table 17 reports the original scores.

Evaluator (App. F.3). All methods use the MARBLE pipeline with unmodified prompts and one judge, gpt-4o-mini-2024-07-18 at temperature 0 , with one final response per component. The judge does not see method names. CoTIE requests, replies, and admitted records enter the logs like native MAB messages. All three scores use a 5-point scale.

  • Planning rates task organization, role keeping, and strategy adaptation from the task description, agent profiles, and planning logs.
  • Communication rates the communication logs.
  • Task rates the final deliverable. Coding inputs are the task statement, functional requirements, final solution.py, and required files. Research inputs are the assignment and final proposal. Database inputs are the task statement, symptoms, final diagnosis, and reference root cause.

All components (Table 17), Full − Naive.

SettingPlanningTaskCommunication20·CS
DeepSeek Coding+0.14+0.27+0.03+1.74
DeepSeek Research+0.13+0.19+0.08+2.05
DeepSeek Database+0.36+0.35+0.21+5.67
Qwen3 Coding+0.19+0.13+0.10+2.90
Qwen3 Research+0.27−0.05+0.17+4.49
Qwen3 Database+0.28+0.24+0.19+4.64

Full is higher on every component in every setting except the Qwen3 Research Task Score. In the timing experiment Early is higher than all other conditions on every component (20·CS: Early 71.40, Late 63.11, No 59.26).

Execution check (Table 18). We ran the final MAB Coding programs on independent tests written from each task's specification before any solution was inspected. They cover execution, functionality, and edge cases and are the same for all methods. Success means passing all required tests within 30 seconds.

Success rateSuccesses
DeepSeek main, Naive → Full55.0% → 68.3%165 → 205 of 300
Qwen3 main, Naive → Full58.7% → 62.7%176 → 188 of 300
Timing, Early / Late / No / Late-regen.61.3 / 38.3 / 31.8 / 33.1%92 / 58 / 53 / 52 of 150

Early − Late in execution success is +23.0 points (95% CI [11.0, 34.5], paired percentile bootstrap over 73 source tasks, 10,000 resamples). The judge Task Score correlates with execution success at r = 0.50 to 0.52. The timing result therefore also holds for executed code.

Thank you again. We believe these analyses resolve both of your questions. If so, we would be grateful if you would consider raising your rating, together with the soundness and presentation scores. We are happy to provide further details.

Reviewer_wuuqOpenReview ↗

Response to the author

Thank you for responding to my questions. Your responses clarified some of my concerns. After considering the paper as a whole, I have decided to keep my original score.

Thank you for reading our response and for letting us know that it clarified some of your concerns. To make good use of the remaining discussion period, could you tell us which concerns remain? We would be glad to address them with further analyses or revisions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all three reviewers for their time and careful reading. Their comments led to new analyses and a clearer paper. The reviews raise four points.

  • Whether results depend on choosing the requery limit L on evaluation tasks (wuuq, hg2S).
  • How MAB scores are produced and how they relate to working code (wuuq).
  • Scope and statistical robustness of the timing experiment (hg2S).
  • Positioning, presentation, and latency (pbsh).

The revised PDF marks all changes in blue. The main text stays within 10 pages.

Changes

  1. New Section 2 with Table 1 positions CoTIE against output-level exchange, output screening, and help seeking. The introduction now states the limitations of prior work and the gap.
  2. Section 3.1 walks through Figure 1 step by step. The caption defines disclosure A. The introduction's timing paragraph is self-contained.
  3. Selection-aware evaluation (App. J.1). L is chosen on held-out task folds. Full − Naive stays positive in all six MAB settings (1.82 to 7.06, versus 2.30 to 7.10 at fixed L = 3). MATH, HumanEval, and HotpotQA were not used to choose L. There Full − Naive is positive in all six settings and significant after Holm correction in four (Table 22).
  4. MAB scoring (App. F.3). Evaluator and inputs, all judge components including the original Coordination Score, and an execution-based check.
  5. Timing robustness (App. D.1). Checkpoint distribution, bootstrap over source tasks, a mixed model, and an empty-packet placebo.
  6. Extra calls, serial latency, and a common wall-clock deadline (App. M, summary in Section 4.2).
  7. A Limitations paragraph (Section 5) that scopes the timing claim.

Corrections (App. N)

  • App. F.1 described our MAB composite as the original MultiAgentBench configuration. We corrected this. The composite is our own aggregate, and both original scores are now reported.
  • Per-run values in old Tables 8 to 12, 17, and 18 had been copied from a draft table. We replaced them with values from the final logs. Means in the main text do not change, and the Early − Late 95% CI becomes [10.18, 15.01] instead of [8.69, 16.51]. App. N lists all affected numbers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.