acceptodds
Under review as a conference paper at ICLR 2027

What Does Talking Add? The Value of Feedback in Multi-Agent Inference

Abstract

In multi-agent language model systems, a decision-maker combines reports from agents with different private evidence to answer a task question. Limited report space requires each agent to select which evidence to include. The information needed for this selection may be held by another agent, leaving the reporting agent uncertain about what to preserve. A follow-up protocol addresses this issue by allowing the decision-maker to ask questions after reading the initial reports. We compare the accuracy and cost of this protocol with stronger independent reporting and sequential reporting protocols. Stronger independent reporting improves report encoding and allows local revision, while sequential reporting sends selection information from one agent to another before the receiving agent prepares a report. Each agent’s initial evidence access and the total resource limits remain unchanged across protocols. Our analysis separates accuracy loss caused by information missing from reports from failures to use information already received. Before the sender knows which evidence is needed, a report must support multiple possible needs. Earlier access to selection information can reduce what the report must preserve. Controlled routing and document-selection experiments show that stronger reports reduce the estimated accuracy advantage of follow-up. With the direction of information dependence known, sequential reporting achieves high accuracy with fewer generated tokens than follow-up. The experiments also reveal errors in preparing and reading reports, even when the report budget is sufficient to include all required information.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, multi-agent collaboration, distributed information, communication budgets

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vGC3OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

The paper studies what adaptive follow-up communication adds when agents hold different private evidence and communicate under resource limits. It compares independent reporting, stronger reporting with local revision, a fixed A-to-B sequential exchange, and an optional follow-up protocol. A decision-theoretic analysis separates information lost during reporting from decision potential left unused by the model, and characterizes how learning a selection clue before reporting changes coverage requirements. Routing and synthetic document-selection experiments use development-selected reporting baselines, paired comparisons, fixed-record interventions, and deterministic decoders. Stronger reporting reduces the estimated follow-up advantage; with a known dependency direction, sequential exchange attains high accuracy with fewer generated tokens in the tested implementations. Further interventions show benefits from renewed private-evidence access and errors in using already transmitted information.

Critical strengths
  1. The study addresses weak-baseline and mechanism confounds directly. The independent reporting baseline is selected from a richer pool on separate development worlds, substantially improving its test accuracy. Fixed-record comparisons then distinguish another reply based only on the report from renewed access to private evidence, while fixed decoders test what can be recovered from the identical decision record. Together these controls provide more convincing evidence than aggregate protocol accuracy alone.

  2. The empirical accounting and uncertainty reporting are unusually careful. Communication includes transmitted copies and query/reply overhead, generation includes private revisions and unsuccessful work, and paired whole-world intervals accompany important comparisons. The paper reports that the main four-point Follow-up advantage over strengthened reporting has an interval crossing zero, while the fixed-record fresh-evidence and document-task gains are supported more clearly.

Critical weaknesses

The contribution's empirical scope is limited. Both routing and document selection are synthetic tasks with a known A-to-B selection dependency, and the tested models are two 14B members of the Qwen series. This is a useful setting for isolating clue timing, but it also favors a predetermined exchange order. It does not establish whether the same accuracy–cost relationship holds when information dependencies are unknown, bidirectional, or discovered over several rounds. The theoretical decomposition and coverage counts are largely foundational decision/communication arguments; the paper's added value lies mainly in the controlled empirical diagnosis. I view this as a limitation on significance and generality.

Questions for authors
  1. Could you compare Sequential and Follow-up with and without local revision, using matched report budgets? Please report both accuracy and generation cost.

  2. Could you repeat the private-evidence intervention on the main Follow-up runs, keeping the actual reports and query decisions fixed? How much does renewed evidence access improve accuracy?

  3. How do the results change when the information-dependency direction is unknown in advance? Please compare Follow-up with both fixed orders and a fixed bidirectional exchange under the same budgets.

  4. Could you further explore whether your findings also apply to powerful closed-source models, such as GPT-6-Sol and Claude-Opus-5?

8 comments

Thank you for your careful assessment and detailed questions. We appreciate your recognition of the reporting baselines, controlled diagnostics, and resource accounting. We understand that your main concern is how far the resulting insights extend beyond the tested workflows and models. Your questions also help identify the controls needed to interpret the differences between complete communication procedures.

We organize our response around these concerns. We examine report preparation under matched message limits, assess private-evidence access while preserving the main Follow-up reports and query decisions, and compare communication procedures when evidence holders vary. We also address the stronger-model question through a focused comparison with an additional model family within our access and funding constraints. These comparisons serve our broader aim of helping designers identify which improvements require better reports, better use of received information, or further evidence access. The closing discussion connects this guidance to a concrete troubleshooting workflow and clarifies the scope of the empirical findings.

Response to Q1:

We agree that matching both report preparation and report budgets makes the generation-cost comparison easier to interpret. Our original comparison evaluated the complete procedures selected for each protocol. To examine the contribution of report preparation more directly, we compare Sequential and Follow-up under both Report and Revise while fixing the first-report allocations.

The comparison uses the same 200 Permutation worlds at K=32 and b=64, with the same Qwen3-14B model, report encoding, and generation ceilings. A's first report has an eight-token ceiling, and B's first report has a 28-token ceiling. For Sequential, Report or Revise applies to A's report and B's report conditioned on A's clue. For Follow-up, the preparation choice applies to the two initial independent reports. The subsequent query, reply, and decision rules remain fixed across its preparation conditions. Communication charges every transmitted copy, including the two copies of A's report in Sequential.

ProtocolPreparationAccuracyMean generation
SequentialReport96.5%1.22k
SequentialRevise97.0%1.77k
Follow-upReport94.5%4.18k
Follow-upRevise96.0%5.00k

Follow-up uses an additional 2.96k generated tokens when both protocols use Report and 3.23k when both use Revise. Changing Report to Revise adds 0.55k tokens for Sequential and 0.82k for Follow-up.

The corresponding accuracy changes from revision are +0.5 percentage points with a pointwise 95% paired interval of [-1.5, 3.0] for Sequential and +1.5 points [-1.0, 4.0] for Follow-up. These intervals use 100,000 bootstrap replicates over the 200 paired worlds. The accuracy effects of revision remain uncertain at this sample size, while mean generation increases in both procedures.

The original manuscript reports Sequential at 97% accuracy and 1.20k generated tokens under its original conditional-report allowance. This supplementary comparison instead fixes B's first-report ceiling at 28 tokens in both protocols. A mean generation gap remains after preparation and first-report budgets are matched. The remaining difference concerns the complete procedures. Sequential prepares B's report after receiving the clue, while Follow-up prepares independent reports before deciding whether to request further evidence.

Response to Q2:

Restoring access to the responder's private evidence raises overall accuracy from 84% to 96% when the main Follow-up reports and query decisions are held fixed. We use this intervention to examine the benefit of renewed evidence access along the trajectories produced by the complete protocol.

The comparison uses the 200 main Permutation worlds at K=32 and b=64. Follow-up issues a query in 156 worlds and answers directly in 44 worlds. For each queried world, we retain the actual initial reports, query recipient, complete query text, and remaining reply allowance. The report-only responder receives the transmitted report, query, and common instructions. The fresh-evidence responder additionally receives the original private evidence.

Both conditions use the same responder template and non-private context. The private-evidence block is the controlled difference in the responder's input. The fresh-evidence condition retains the original reply and final answer, while the report-only condition uses a newly generated continuation under the same model version and sampling configuration. Each condition contributes one stochastic continuation per queried world. The final decision uses the same template and original decision context, together with the reply produced under the corresponding condition. The 44 direct-answer outcomes are retained identically in both conditions.

Evaluation groupReport-onlyFresh evidencePaired difference in percentage points
Queried worlds125/156, 80.13%149/156, 95.51%+15.38 [8.97, 22.44]
Direct-answer worlds43/44, 97.73%43/44, 97.73%0
All worlds168/200, 84.00%192/200, 96.00%+12.00 [7.00, 17.50]

Within the queried subset, both conditions answer correctly in 121 worlds. Fresh evidence alone succeeds in 28 worlds, report-only alone succeeds in four, and both fail in three. Renewed access therefore yields 24 additional correct answers across the fixed query trajectories.

The pairing is defined by each original world and its fixed query trajectory. We compute pointwise 95% percentile intervals using 100,000 paired bootstrap replicates, resampling 156 worlds for the queried-subset comparison and 200 worlds for the overall comparison. Each replicate retains the two corresponding outcomes within every sampled world. The reported intervals summarize variation across these paired world-level outcomes.

The queried-subset result measures the benefit of private-evidence access at the questions actually selected by Follow-up. The overall result incorporates that benefit while preserving the protocol's original decisions about whether to query. This extends the separate fixed-report diagnostic to the main Follow-up trajectories and provides direct evidence that renewed access improves the replies and subsequent decisions under the protocol's own reporting and query choices.

Response to Q3 (Part 1):

We agree that varying the evidence holders directly tests the dependence of Sequential on a known communication direction. We therefore exchange the clue and detailed-evidence packages between A and B within each base world while preserving the task and correct answer. The decision-maker receives the public task and transmitted messages, while the evidence allocation remains private to the agents.

We implement Sequential in both fixed orders, A-to-B and B-to-A. In each order, the first agent sends its report to the other agent and the decision-maker, followed by the second agent's conditional report. Fixed bidirectional exchange sends both initial reports to the other agent and the decision-maker, after which both agents prepare conditional reports.

We also include Rule as a deterministic scheduling comparison. When the received report labels identify one clue holder and one detailed-evidence holder, Rule forwards the clue report to the detailed-evidence holder and obtains a conditional reply. Other label combinations lead to a decision from the initial reports.

Each setting contains 200 base worlds evaluated under both evidence allocations, giving 400 outcomes and 200 independent sampling units. The two K=128 settings reuse the same worlds. All procedures use Qwen3-14B under the same total resource ceilings and initial evidence permissions. Every transmitted copy is charged. Each table entry gives accuracy in percent, mean communication tokens, and mean generation in thousands of tokens.

ProcedureK=32, b=64K=128, b=64K=128, b=128
Sequential A-to-B83.25 / 42.4 / 2.9750.75 / 43.1 / 4.4459.50 / 67.7 / 4.63
Sequential B-to-A82.75 / 42.5 / 2.9850.75 / 42.9 / 4.5560.00 / 67.7 / 4.54
Fixed bidirectional exchange94.50 / 61.8 / 5.1691.75 / 62.5 / 8.2497.00 / 121.7 / 8.48
Rule95.75 / 52.4 / 4.1893.25 / 53.1 / 6.2596.25 / 74.3 / 6.59
Follow-up95.25 / 55.4 / 4.7694.00 / 60.4 / 7.9896.75 / 89.2 / 8.05

Response to Q3 (Part 2):

The allocation-specific results show why the fixed orders lose accuracy. At K=128 and b=64, Sequential A-to-B achieves 92.0% when A holds the clue and 9.5% after the evidence packages are exchanged. Sequential B-to-A achieves 10.0% and 91.5% under the same two allocations. Each fixed order therefore averages to 50.75%. When the detailed-evidence holder reports first, its 24-token report must be prepared before the relevant route is known.

Under the original and exchanged allocations, bidirectional exchange achieves 92.5% and 91.0%, while Follow-up achieves 96.0% and 92.0%. Both procedures maintain high accuracy across the two allocations. Their pooled accuracies are 91.75% and 94.00%.

The Follow-up-minus-bidirectional accuracy differences across the three settings are +0.75 points [-1.00, 2.75], +2.25 points [0.25, 4.50], and -0.25 points [-2.25, 1.50]. These pointwise 95% intervals use 100,000 paired bootstrap replicates. Each replicate resamples 200 base worlds as clusters, retaining both evidence allocations and all compared procedures within each sampled world.

These results connect the useful communication schedule to the available dependency information. A fixed direction supports a single sequential exchange. When evidence holders vary, bidirectional exchange can cover both directions, while Rule and Follow-up select further communication using received messages. At K=128 and b=64, Rule reaches 93.25% with 6.25k generated tokens, compared with 94.00% and 7.98k for Follow-up. The paired accuracy difference is +0.75 points [-1.25, 2.75].

The extension therefore examines dependencies whose direction becomes identifiable from initial messages. Dependencies that emerge through several rounds of evidence interpretation remain a further setting for evaluating these communication procedures.

Response to Q4 (Part 1):

We agree that evaluating a stronger model from another family would help clarify how the findings depend on model capability. Our original study focused on understanding the sources of a follow-up improvement through controlled comparisons of reporting, clue timing, and evidence access. We held the model fixed within each comparison and examined these mechanisms using two Qwen models with different reasoning modes. Your suggestion adds an important question. A stronger model may construct more informative reports, use received evidence more effectively, or make better decisions about when and how to ask a follow-up question. Each improvement could change the accuracy and cost differences between the communication procedures.

We would therefore like to examine this question while preserving the controls that make the comparisons interpretable. However, our access to GPT-6-Sol and Claude-Opus-5 is currently constrained by the applicable API access policies. The financial requirements also affect the scale of the evaluation we can undertake. Each instance requires several complete communication procedures, with model calls for report preparation, revision, communication decisions, and final answers. Repeating these procedures across evaluation worlds and model families creates a cumulative requirement for API usage. These considerations led us to seek an accessible alternative that would allow us to retain the essential comparisons within our research budget.

We selected DeepSeek-V4-Pro for a focused supplementary evaluation because it introduces a flagship reasoning model from a different family. We retained the task definitions and controlled resource limits, and compared strengthened independent reporting, Sequential, and Follow-up. The two routing settings examine both a case where a complete representation fits the communication budget and a case with greater reporting demands.

The evaluation uses 50 worlds at each of two Permutation settings, K=32 and K=128, with communication budget b=64. All agents use DeepSeek-V4-Pro. The compared procedures use the same worlds, generation settings, and total resource ceilings. Reporting configurations are fixed using separate development worlds. Sequential and Follow-up both use private revision, with matched first-report ceilings of eight tokens for A and 28 for B.

Communication counts complete messages and every transmitted copy using the paper's reference tokenizer. Generation uses API-reported output usage across all calls, including reasoning and revision. The common reader is the same deterministic decoder used in the paper. API response fields are mapped to the existing record schema through predefined deterministic rules using only the recorded message fields. The decoder then applies its fixed parsing and recovery rules to the public task and messages available to D. Ground-truth labels are used for scoring.

Reviewer_74JtOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies when follow-up questions are useful in multi-agent reasoning, where agents have access to different information. It compares follow up with better independent reports and sequential communication, and distinguishes between missing information and failures to use information already received. The approaches are evaluated on constructed synthetic routing and document selection tasks, using different models, communication budgets, and task settings.

Critical strengths
  • The paper clearly analyzes what follow up communication adds, separating missing information from failures to use information already received. This distinction is useful for understanding and improving multi-agent systems.
  • The experiments are carefully controlled, with stronger reporting baselines and analyses of when clues are provided and whether agents can revisit their original evidence. These comparisons help explain where the gains come from.
Critical weaknesses
  • The generation cost comparison may partly depend on the reporting setup, as follow up includes local revision of its initial reports while Sequential uses Report. A comparison using the same local reporting procedure would make the cost difference easier to interpret.
  • The experiments assume a fixed A→B information dependency. I wonder how the comparison would change if the clue and record holders were swapped across instances, so that some cases require A→B communication and others require B→A. Could the authors provide an analysis or a small experiment comparing Sequential and Follow-up under the same budgets in this setting?
Questions for authors

How would the accuracy and generation costs of Sequential and Follow-up compare if both used Report or both used Revise? This would help clarify how much the cost difference depends on the reporting procedure.

5 comments

Thank you for your careful assessment and constructive suggestions. We appreciate your recognition of the distinction between missing information and failures to use received information, as well as the controlled comparisons supporting this distinction. Your questions identify two factors that deserve closer examination when interpreting the accuracy and generation costs of Sequential and Follow-up, namely report preparation and the allocation of private evidence.

We organize our response around these factors. We compare Sequential and Follow-up using both Report and Revise under matched initial report limits to clarify how report preparation affects the original cost comparison. We also examine changes in evidence holders through both fixed Sequential orders, bidirectional exchange, and message-based scheduling. These comparisons help assess when a fixed sequence can use available dependency information and when communication must accommodate a changing direction. We report accuracy and resource use together, with paired uncertainty estimates for the main contrasts.

Response to W1:

We agree that report preparation contributes to the generation cost of a complete communication procedure. To make this contribution easier to assess, we compare Sequential and Follow-up under both Report and Revise while also matching their first-report budgets.

The comparison uses the same 200 Permutation worlds at K=32 and b=64, with Qwen3-14B, the same report encoding, and the same generation ceilings. A's first report has an eight-token ceiling and B's first report has a 28-token ceiling. For Sequential, the preparation choice applies to A's report and B's report conditioned on A's clue. For Follow-up, the choice applies to the two initial independent reports. The subsequent query, reply, and decision rules remain fixed across its preparation conditions. Every transmitted copy is charged, including A's two copies in Sequential.

ProtocolPreparationAccuracyMean generation
SequentialReport96.5%1.22k
SequentialRevise97.0%1.77k
Follow-upReport94.5%4.18k
Follow-upRevise96.0%5.00k

Follow-up uses an additional 2.96k generated tokens when both procedures use Report and 3.23k when both use Revise. Revision adds 0.55k tokens for Sequential and 0.82k for Follow-up. The accuracy changes from revision are +0.5 percentage points [-1.5, 3.0] and +1.5 points [-1.0, 4.0], respectively. These pointwise 95% intervals use 100,000 paired bootstrap replicates over worlds. The accuracy effects remain uncertain, while mean generation increases in both procedures.

The original manuscript comparison uses Sequential's original conditional-report allowance and reports 97% accuracy with 1.20k generated tokens. The new comparison constrains B's first report to 28 tokens in both protocols. Under these matched allocations, a mean generation gap remains after report preparation is matched. This remaining difference concerns the complete procedures, including their reporting requirements and subsequent decision steps. Sequential prepares B's report after receiving the clue, while Follow-up prepares independent reports before deciding whether to request further evidence.

Response to W2:

We agree that exchanging the clue and record holders directly tests how Sequential depends on the evidence allocation. We therefore exchange the two private evidence packages within each base world while preserving the task and correct answer. We implement Sequential in both fixed orders, A-to-B and B-to-A, and compare these procedures with fixed bidirectional exchange and Follow-up under the same total resource limits.

Each fixed Sequential order sends the first report to the other agent and the decision-maker, followed by the second agent's conditional report. In fixed bidirectional exchange, both agents first send their reports to the other agent and the decision-maker, and then prepare conditional reports using the received message and their private evidence.

We also include Rule, which uses the generated report types to identify the useful direction. When one report identifies selection information and the other identifies detailed evidence, Rule forwards the selection report to the detailed-evidence holder and obtains a conditional reply. Other label combinations lead to an answer from the initial reports. Rule-matched applies this procedure to Follow-up's exact initial reports and includes their preparation costs.

The evaluation uses Qwen3-14B and 200 base worlds per setting, each evaluated under both evidence allocations. Each setting therefore contains 400 outcomes and 200 independent sampling units. The two K=128 settings reuse the same worlds. All transmitted copies are charged, including forwarding. Each table entry gives accuracy in percent, mean communication tokens, and mean generation in thousands of tokens.

ProcedureK=32, b=64K=128, b=64K=128, b=128
Sequential A-to-B83.25 / 42.4 / 2.9750.75 / 43.1 / 4.4459.50 / 67.7 / 4.63
Sequential B-to-A82.75 / 42.5 / 2.9850.75 / 42.9 / 4.5560.00 / 67.7 / 4.54
Fixed bidirectional exchange94.50 / 61.8 / 5.1691.75 / 62.5 / 8.2497.00 / 121.7 / 8.48
Rule95.75 / 52.4 / 4.1893.25 / 53.1 / 6.2596.25 / 74.3 / 6.59
Follow-up95.25 / 55.4 / 4.7694.00 / 60.4 / 7.9896.75 / 89.2 / 8.05
Rule-matched95.00 / 59.0 / 4.0492.75 / 60.4 / 6.8695.75 / 89.2 / 7.07

At K=128 and b=64, Sequential A-to-B achieves 92.0% when A holds the clue and 9.5% after the evidence packages are exchanged. Sequential B-to-A achieves 10.0% and 91.5% under the same two allocations. Both fixed orders average to 50.75%. When the detailed-evidence holder reports first, its 24-token report must be prepared before the relevant route is known. The later clue reaches the decision-maker after that report has been fixed.

Bidirectional exchange achieves 92.5% and 91.0% under the original and exchanged allocations, while Follow-up achieves 96.0% and 92.0%. Their pooled accuracies are 91.75% and 94.00%. Rule reaches 93.25% with lower mean generation than Follow-up.

The Follow-up-minus-bidirectional accuracy differences across the three settings are +0.75 points [-1.00, 2.75], +2.25 points [0.25, 4.50], and -0.25 points [-2.25, 1.50]. The corresponding Follow-up-minus-Rule-matched differences are +0.25 points [-1.00, 1.50], +1.25 points [-0.75, 3.50], and +1.00 points [-0.50, 2.50]. These pointwise 95% intervals use 100,000 bootstrap replicates over base worlds, retaining both allocations and all compared procedures within each sampled world.

At K=128 and b=64, Follow-up uses an additional 1.13k generated tokens [0.98k, 1.27k] over Rule-matched, calculated from the unrounded paired costs. Both mean communication costs round to 60.4 tokens, with a paired difference of 0.04 tokens [-0.92, 0.92]. The cost intervals use 30,000 paired bootstrap replicates. This matched-record comparison evaluates the subsequent communication procedures, including whether communication continues, query or forwarding content, and the resulting reply allowance.

These results show how the comparison changes when evidence holders vary. A single fixed Sequential order is sensitive to the allocation. Bidirectional exchange can cover both directions, while Rule and Follow-up use received messages to select further communication. The extension concerns dependencies identifiable from the initial reports. Dependencies discovered through several rounds of evidence interpretation remain a further setting for evaluation.

Response to Q1:

The matched comparison in W1 gives a mean Follow-up-minus-Sequential generation difference of 2.96k tokens when both procedures use Report and 3.23k when both use Revise. Both comparisons match report preparation and first-report budgets. The original comparison reports a 3.80k difference under its original preparation choices and report allocations.

W1 provides the four accuracy and generation-cost values, preparation rules, and paired uncertainty estimates. These comparisons clarify how report preparation affects generation cost, while a mean gap remains between the complete communication procedures after preparation and first-report budgets are matched.

Reviewer_GcuQOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the value of follow-up communication when agents hold different private evidence. It compares independent reporting, private report revision, fixed-order sequential reporting, and an optional follow-up query selected by a decision-maker. An analytical framework distinguishes information missing from transmitted reports from information that is available but not successfully used. Experiments on generated routing tasks and template-generated ticket/runbook tasks examine report quality, clue timing, renewed evidence access, and decoding of the received record.

Critical strengths

I did not identify a substantive strength sufficient to support acceptance in the current version. I acknowledge the controlled comparisons of reporting, clue timing, and renewed evidence access. My reservations concern the additional insight and practical relevance established by these comparisons, as explained below.

Critical weaknesses
  1. The additional theoretical and conceptual insight is limited in my assessment. The decomposition into reporting loss and unused decision potential, the distinguishability condition for report coverage, and the decoder-based bounds largely formalize familiar principles. The paper appropriately identifies the report-alphabet count as classical, and the framework does help organize the diagnostics. However, I did not find a sufficiently clear, non-obvious conclusion arising from the combination of these results and the experiments. The finding that a report can be more focused when its sender first learns which record is needed is reasonable, but I would appreciate a clearer account of what the analysis adds beyond this intuition.

  2. The justification and practical relevance of the synthetic evaluation are not yet convincing to me. Controlled synthetic tasks can isolate mechanisms, but the routing tasks and template-generated document task retain a known A-to-B information dependence. This makes a clue-first fixed sequence a particularly natural solution. Although the paper states this scope, it remains unclear how representative the setting is of practical multi-agent coordination and what broader design insight is established. My concern is the justification and significance of the evaluated setting, rather than a claim that synthetic evaluation is intrinsically invalid or that the reported results are incorrect.

I also found the presentation difficult to follow. A concise worked instance that traces the private inputs, protocol messages, budget charges, and final decision would help make the relationship between the main protocols and diagnostic interventions easier to understand.

My initial recommendation was clear rejection, primarily because I did not find sufficient additional insight and justification of significance. I regard the concerns about empirical support as minor or fixable; I have not identified a specific mathematical error.

Post-rebuttal update: Thank you for the detailed response, the worked example, and the additional experiments. Swapping the evidence holders directly addresses my concern about a fixed communication direction. The comparison with simple forwarding also gives a concrete result: when the initial reports reveal who has the clue and who has the detailed evidence, forwarding can achieve similar accuracy to autonomous follow-up at lower generation cost. These results clarify the contribution and lead me to revise my overall rating to weak acceptance (3). Testing workflows where the relevant dependency emerges over several exchanges would be a useful extension.

Questions for authors
  1. Could you explain more concretely which practical coordination settings the generated routing and document tasks are intended to represent, and justify the known A-to-B dependence? Which conclusions should readers expect to transfer when the useful information flow is not prescribed in advance?

  2. What is the principal additional insight supplied by the theoretical framework beyond the standard Bayes-accuracy and coverage arguments? Please identify how the empirical diagnostics substantiate that insight beyond the intuitive benefit of providing the relevant selection clue before reporting.

  3. Will the generated instances, data generators, development/evaluation splits, prompts, and evaluation harness be released so that others can evaluate additional models and communication protocols? I could not confirm an explicit release plan from the manuscript or current submission page. This is a request for clarification, not an assertion that the artifacts will be withheld.

8 comments

Thank you for your assessment and suggestions. We understand that your main concern is whether the theoretical framework and controlled experiments establish useful insights beyond the intuitive benefit of providing a selection clue before reporting. Our motivation is to help system designers understand an observed improvement after follow-up and identify which changes to reporting or communication could provide that improvement efficiently.

We address this question below. We explain how the analysis and diagnostics distinguish opportunities to improve reports, improve decisions from information already received, and obtain further private evidence. A network troubleshooting example makes these choices concrete. We also explain the practical role of known information dependencies and examine how the protocol comparison changes when evidence holders vary. Finally, we provide a worked instance tracing the messages and communication charges, together with the experimental details and release plan requested in your questions.

Response to W1:

In the main Permutation setting, stronger independent reports combined with a fixed decoder operating on the received record reach the Follow-up model's observed aggregate accuracy. A separate controlled intervention also shows a benefit from renewed access to private evidence. Together, these findings identify distinct opportunities to improve a system after observing a follow-up gain. We developed the framework to examine what the reports preserve, assess how the received information is used, and test the value of further evidence access.

Consider an illustrative network troubleshooting task. A monitoring agent knows that a failed request passed through node 7, while a diagnostic agent holds configuration and alarm records for many nodes. A coordinator combines their short reports. A follow-up about node 7 could disclose relevant evidence omitted from the initial report. When the reports already contain the relevant configuration, improving the coordinator's use of those reports offers another opportunity to reach the correct diagnosis. These possibilities suggest different priorities for system improvement.

Proposition 1 provides the analytical basis by separating reporting loss from unused decision potential using established Bayes principles. The coverage analysis specifies what a report must preserve across possible needs. For K routes with distinct components and a fixed active rule, the classical exact-coverage count gives K distinguishable reports for cyclic assignments and K! for arbitrary permutations before the sender learns the route, and K for either family afterward. Thus, the same conditional state space can impose different reporting requirements before the clue arrives. We examine the practical consequence through target coverage. At K=128 and b=64, earlier clues reduce target-entry omissions from 26 to one among 100 worlds (Section 4.3).

The framework also directs attention to construction and reading when a complete representation fits the budget, as at K=32 and b=64. Strengthening independent reporting raises Permutation accuracy from 80% to 92%, reducing the estimated Follow-up advantage from 16 to four percentage points. The remaining difference has a 95% interval of [-0.5, 8.5] points (Section 4.2).

We then hold those strengthened reports fixed and replace the model reader with a fixed decoder using only the record received by the decision-maker. Accuracy rises from 92% to 96%. Using the reported joint outcomes in Appendix I.4 and the paired bootstrap procedure in Appendix M, we obtain a four-point difference with a 95% interval of [1.0, 7.5]. By Equation (11), this attainable reader improvement provides a lower bound on the unused decision potential in the received record. Applying the same decoder to Follow-up records yields 99%, retaining a three-point recoverability advantage with a 95% interval of [0.5, 6.0] (Appendix I.8).

To assess what renewed evidence access adds, we use a separate set of fixed initial reports. Both reply conditions use the same query and reply allowance, while the fresh-evidence condition additionally provides the responder's private evidence. Accuracy rises from 81% to 96%, a paired gain of 15 points with a 95% interval of [9.5, 21.0] (Section 4.4 and Appendix I.9). This controlled intervention establishes the value of private-evidence access within that diagnostic and complements the decoder comparison on information already received.

In the troubleshooting example, these findings correspond to improving which node records are reported, improving the coordinator's use of those records, or consulting the diagnostic agent's private configuration again. The contribution is to connect protocol-level accuracy differences to evidence about these distinct design choices. Coverage analysis, decoder bounds, and controlled access interventions together help identify where further system improvement can be achieved.

Response to W2 (Part 1):

Our tasks represent workflows in which one agent identifies the relevant item and another holds the evidence needed to evaluate that item. The known A-to-B direction describes the relationship between these responsibilities. The particular item required for a decision still depends on the current observations. We chose this setting to examine how much follow-up adds when workflow knowledge can already guide communication.

For example, in software troubleshooting, a log agent identifies the path of a failed request, while a configuration agent holds records for many paths. The designer knows that request-path information guides configuration selection, while the agents must determine the path and retrieve the relevant evidence for each request. Routing captures this dependency directly. The document task expresses the same dependency through a ticket and configuration-specific instructions. Controlled generation also provides the correct target and evidence permissions, allowing us to distinguish target omission, incorrect disclosure, and recoverable information left unused by the decision-maker.

The original results establish the value of using this workflow knowledge. On Permutation at K=32 and b=64, Sequential reaches 97% accuracy with 1.20k generated tokens, while Follow-up reaches 96% with 5.00k. The paired Follow-up-minus-Sequential difference is -1 percentage point with a 95% interval of [-3.5, 1.5]. On the document task, Sequential improves over independent reporting by 8.5 points [3.5, 14.0], with mean generation of 2.90k and 4.10k tokens, respectively. These figures refer to the original manuscript configurations.

To examine dependence on fixed responsibilities, the extension exchanges the two evidence packages within each base world while preserving the task and answer. We compare six complete procedures and a matched-record diagnostic on Permutation with Qwen3-14B under the same initial evidence permissions, revision rules, and total resource ceilings. Sequential is implemented in both fixed orders. Each order sends the first report to the other agent and the decision-maker, followed by the second agent's conditional report. For fixed bidirectional exchange, each agent sends an initial report to the other agent and the decision-maker, followed by a conditional report using the received message and private evidence.

Rule reads the model-generated type labels in the initial reports. When one report identifies selection information and the other identifies detailed evidence, Rule forwards the complete selection report to the detailed-evidence holder and obtains a conditional reply. Other label combinations lead to an answer from the initial reports. Rule-matched applies this procedure to Follow-up's exact initial reports, including their preparation cost. The matched comparison therefore evaluates differences in the subsequent communication procedures, including whether and how communication continues, the content of a query or forwarded message, and the resulting reply allowance.

Each setting contains 200 base worlds with both evidence allocations, giving 400 outcomes and 200 independent sampling units. The two K=128 settings reuse the same worlds. Every transmitted copy is charged, including forwarding. Generation includes reasoning, drafts, revisions, queries, replies, repairs, and final answers. Each table entry gives accuracy in percent, mean communication tokens, and mean generation in thousands of tokens.

ProcedureK=32, b=64K=128, b=64K=128, b=128
Strengthened independent reporting91.25 / 38.5 / 3.2055.50 / 59.6 / 6.6386.75 / 112.8 / 6.46
Sequential A-to-B83.25 / 42.4 / 2.9750.75 / 43.1 / 4.4459.50 / 67.7 / 4.63
Sequential B-to-A82.75 / 42.5 / 2.9850.75 / 42.9 / 4.5560.00 / 67.7 / 4.54
Fixed bidirectional exchange94.50 / 61.8 / 5.1691.75 / 62.5 / 8.2497.00 / 121.7 / 8.48
Rule95.75 / 52.4 / 4.1893.25 / 53.1 / 6.2596.25 / 74.3 / 6.59
Follow-up95.25 / 55.4 / 4.7694.00 / 60.4 / 7.9896.75 / 89.2 / 8.05
Rule-matched95.00 / 59.0 / 4.0492.75 / 60.4 / 6.8695.75 / 89.2 / 7.07

At K=128 and b=64, Sequential A-to-B reaches 92.0% when A holds the clue and 9.5% after the evidence packages are exchanged. Sequential B-to-A reaches 10.0% and 91.5% under the same two allocations. Both average to 50.75%. In the reverse order, the mapping holder must prepare a 24-token report before learning the requested route. At b=128, the first mapping report has a 48-token allowance, and reverse-order accuracy rises to 23.0% and 23.5%. These contrasts show sensitivity to direction under the specified report allocations.

Response to W2 (Part 2):

Bidirectional exchange achieves 92.5% and 91.0% under the original and exchanged allocations, while Follow-up achieves 96.0% and 92.0%. Rule achieves 94.0% and 92.5%. Both a schedule covering the two directions and scheduling based on received messages therefore support high accuracy when evidence holders vary. Rule can use a short initial mapping report because its subsequent reply draws on private evidence after the relevant route becomes available.

The paired accuracy differences and pointwise 95% intervals are reported below in percentage points. The intervals use 100,000 bootstrap replicates. Each replicate resamples 200 base worlds as clusters, retaining both evidence allocations and all compared procedures within each sampled world. Invalid answers count as errors.

ComparisonK=32, b=64K=128, b=64K=128, b=128
Follow-up minus bidirectional exchange+0.75 [-1.00, 2.75]+2.25 [0.25, 4.50]-0.25 [-2.25, 1.50]
Follow-up minus Rule-0.50 [-2.25, 1.00]+0.75 [-1.25, 2.75]+0.50 [-1.50, 2.50]
Follow-up minus Rule-matched+0.25 [-1.00, 1.50]+1.25 [-0.75, 3.50]+1.00 [-0.50, 2.50]

The intervals comparing Follow-up with Rule or Rule-matched include zero. At K=128 and b=64, Follow-up uses an additional 1.13k generated tokens [0.98k, 1.27k] over Rule-matched, calculated from the unrounded paired costs. The paired communication difference is 0.04 tokens [-0.92, 0.92], and both mean communication costs round to 60.4 tokens. Cost intervals use 30,000 paired bootstrap replicates. Deterministic forwarding provides a high-accuracy comparison with lower generation cost in this setting, while the relative accuracy remains uncertain.

These comparisons suggest organizing communication around when the useful direction becomes identifiable. A known direction supports a fixed exchange. When evidence holders vary and report types identify the direction, deterministic scheduling provides a useful comparator for Follow-up. A further design implication is that autonomous questioning is particularly relevant when selecting the next useful information source requires interpreting evidence obtained during the interaction. Dependencies discovered through several rounds remain a further setting for evaluating these alternatives.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Contribution and Rebuttal

Dear Area Chair,

We thank you and the reviewers for assessing our work.

Our Contribution

Our paper examines how to obtain decision-relevant information efficiently when different agents hold the evidence and the clues needed to select that evidence. We compare autonomous follow-up with stronger independent reports and fixed-order communication under the same initial evidence access and resource ceilings. This comparison establishes how much additional accuracy follow-up provides after improving reporting or delivering selection information earlier, together with the computational and communication costs of that improvement.

Our analysis connects the timing of communication to the information a report must preserve. Before the relevant need is known, a report must support multiple possible needs. Earlier selection information permits focused disclosure, while accurate decisions also require the model to construct and interpret the report correctly. We separate information lost during reporting from unused decision potential and develop record-based bounds and controlled interventions to examine these opportunities. The experiments show that stronger reports can recover much of the estimated follow-up advantage, while further gains are available through better reading and renewed private-evidence access. Together, these findings provide practical guidance on where to invest additional computation and communication when designing a multi-agent system.

Key Rebuttal Findings

The supplementary comparisons strengthen the evidence that communication design matters for efficiency. After matching report preparation and first-report budgets, Follow-up still generates approximately 3,000 more tokens per instance than Sequential. Providing the selection clue before reporting therefore remains an effective way to achieve high accuracy with less generation. A focused DeepSeek-V4-Pro evaluation shows the same accuracy–cost pattern in another model family.

Further communication also has a clear practical benefit. Keeping the actual Follow-up reports and query decisions fixed, restoring private-evidence access raises accuracy from 84% to 96%. This extends the original diagnostic to the protocol's own queries and demonstrates the value of allowing the responder to consult its source evidence again.

The evidence-holder exchange addresses the dependence on fixed agent responsibilities. Both bidirectional exchange and message-based scheduling achieve high accuracy when evidence holders vary. Message-based scheduling forwards the selection clue to the agent holding the detailed evidence and uses fewer generated tokens than Follow-up. This extends the practical guidance to workflows where the useful communication direction can be identified from received reports.

Reviewer Feedback and Discussion Status

Reviewer 74Jt recognized the distinction between missing information and failures to use received information as useful for understanding and improving multi-agent systems. The review also highlighted the carefully controlled comparisons.

Reviewer vGC3 recognized the stronger baselines, fixed-record diagnostics, and careful resource accounting. After reviewing the rebuttal, the reviewer stated that our response addresses “most of my concerns about the reliability and interpretation of the results” and raised presentation and soundness to 3. The overall rating remains 2, and we are awaiting clarification of the concerns that continue to determine that recommendation.

Reviewer GcuQ's overall rating remains 1. We are still awaiting feedback on our detailed responses and follow-up message concerning additional insight, practical relevance, and presentation.

We respectfully ask that the final assessment consider the complete discussion record, including the supplementary findings and the reviewers' recognition. We believe the resulting guidance for designing and evaluating multi-agent communication merits a favorable assessment.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.