- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3(+1)
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies what adaptive follow-up communication adds when agents hold different private evidence and communicate under resource limits. It compares independent reporting, stronger reporting with local revision, a fixed A-to-B sequential exchange, and an optional follow-up protocol. A decision-theoretic analysis separates information lost during reporting from decision potential left unused by the model, and characterizes how learning a selection clue before reporting changes coverage requirements. Routing and synthetic document-selection experiments use development-selected reporting baselines, paired comparisons, fixed-record interventions, and deterministic decoders. Stronger reporting reduces the estimated follow-up advantage; with a known dependency direction, sequential exchange attains high accuracy with fewer generated tokens in the tested implementations. Further interventions show benefits from renewed private-evidence access and errors in using already transmitted information.
Critical strengths
-
The study addresses weak-baseline and mechanism confounds directly. The independent reporting baseline is selected from a richer pool on separate development worlds, substantially improving its test accuracy. Fixed-record comparisons then distinguish another reply based only on the report from renewed access to private evidence, while fixed decoders test what can be recovered from the identical decision record. Together these controls provide more convincing evidence than aggregate protocol accuracy alone.
-
The empirical accounting and uncertainty reporting are unusually careful. Communication includes transmitted copies and query/reply overhead, generation includes private revisions and unsuccessful work, and paired whole-world intervals accompany important comparisons. The paper reports that the main four-point Follow-up advantage over strengthened reporting has an interval crossing zero, while the fixed-record fresh-evidence and document-task gains are supported more clearly.
Critical weaknesses
The contribution's empirical scope is limited. Both routing and document selection are synthetic tasks with a known A-to-B selection dependency, and the tested models are two 14B members of the Qwen series. This is a useful setting for isolating clue timing, but it also favors a predetermined exchange order. It does not establish whether the same accuracy–cost relationship holds when information dependencies are unknown, bidirectional, or discovered over several rounds. The theoretical decomposition and coverage counts are largely foundational decision/communication arguments; the paper's added value lies mainly in the controlled empirical diagnosis. I view this as a limitation on significance and generality.
Questions for authors
-
Could you compare Sequential and Follow-up with and without local revision, using matched report budgets? Please report both accuracy and generation cost.
-
Could you repeat the private-evidence intervention on the main Follow-up runs, keeping the actual reports and query decisions fixed? How much does renewed evidence access improve accuracy?
-
How do the results change when the information-dependency direction is unknown in advance? Please compare Follow-up with both fixed orders and a fixed bidirectional exchange under the same budgets.
-
Could you further explore whether your findings also apply to powerful closed-source models, such as GPT-6-Sol and Claude-Opus-5?
8 comments
Response to Q2:
Restoring access to the responder's private evidence raises overall accuracy from 84% to 96% when the main Follow-up reports and query decisions are held fixed. We use this intervention to examine the benefit of renewed evidence access along the trajectories produced by the complete protocol.
The comparison uses the 200 main Permutation worlds at K=32 and b=64. Follow-up issues a query in 156 worlds and answers directly in 44 worlds. For each queried world, we retain the actual initial reports, query recipient, complete query text, and remaining reply allowance. The report-only responder receives the transmitted report, query, and common instructions. The fresh-evidence responder additionally receives the original private evidence.
Both conditions use the same responder template and non-private context. The private-evidence block is the controlled difference in the responder's input. The fresh-evidence condition retains the original reply and final answer, while the report-only condition uses a newly generated continuation under the same model version and sampling configuration. Each condition contributes one stochastic continuation per queried world. The final decision uses the same template and original decision context, together with the reply produced under the corresponding condition. The 44 direct-answer outcomes are retained identically in both conditions.
| Evaluation group | Report-only | Fresh evidence | Paired difference in percentage points |
|---|---|---|---|
| Queried worlds | 125/156, 80.13% | 149/156, 95.51% | +15.38 [8.97, 22.44] |
| Direct-answer worlds | 43/44, 97.73% | 43/44, 97.73% | 0 |
| All worlds | 168/200, 84.00% | 192/200, 96.00% | +12.00 [7.00, 17.50] |
Within the queried subset, both conditions answer correctly in 121 worlds. Fresh evidence alone succeeds in 28 worlds, report-only alone succeeds in four, and both fail in three. Renewed access therefore yields 24 additional correct answers across the fixed query trajectories.
The pairing is defined by each original world and its fixed query trajectory. We compute pointwise 95% percentile intervals using 100,000 paired bootstrap replicates, resampling 156 worlds for the queried-subset comparison and 200 worlds for the overall comparison. Each replicate retains the two corresponding outcomes within every sampled world. The reported intervals summarize variation across these paired world-level outcomes.
The queried-subset result measures the benefit of private-evidence access at the questions actually selected by Follow-up. The overall result incorporates that benefit while preserving the protocol's original decisions about whether to query. This extends the separate fixed-report diagnostic to the main Follow-up trajectories and provides direct evidence that renewed access improves the replies and subsequent decisions under the protocol's own reporting and query choices.
Response to Q3 (Part 1):
We agree that varying the evidence holders directly tests the dependence of Sequential on a known communication direction. We therefore exchange the clue and detailed-evidence packages between A and B within each base world while preserving the task and correct answer. The decision-maker receives the public task and transmitted messages, while the evidence allocation remains private to the agents.
We implement Sequential in both fixed orders, A-to-B and B-to-A. In each order, the first agent sends its report to the other agent and the decision-maker, followed by the second agent's conditional report. Fixed bidirectional exchange sends both initial reports to the other agent and the decision-maker, after which both agents prepare conditional reports.
We also include Rule as a deterministic scheduling comparison. When the received report labels identify one clue holder and one detailed-evidence holder, Rule forwards the clue report to the detailed-evidence holder and obtains a conditional reply. Other label combinations lead to a decision from the initial reports.
Each setting contains 200 base worlds evaluated under both evidence allocations, giving 400 outcomes and 200 independent sampling units. The two K=128 settings reuse the same worlds. All procedures use Qwen3-14B under the same total resource ceilings and initial evidence permissions. Every transmitted copy is charged. Each table entry gives accuracy in percent, mean communication tokens, and mean generation in thousands of tokens.
| Procedure | K=32, b=64 | K=128, b=64 | K=128, b=128 |
|---|---|---|---|
| Sequential A-to-B | 83.25 / 42.4 / 2.97 | 50.75 / 43.1 / 4.44 | 59.50 / 67.7 / 4.63 |
| Sequential B-to-A | 82.75 / 42.5 / 2.98 | 50.75 / 42.9 / 4.55 | 60.00 / 67.7 / 4.54 |
| Fixed bidirectional exchange | 94.50 / 61.8 / 5.16 | 91.75 / 62.5 / 8.24 | 97.00 / 121.7 / 8.48 |
| Rule | 95.75 / 52.4 / 4.18 | 93.25 / 53.1 / 6.25 | 96.25 / 74.3 / 6.59 |
| Follow-up | 95.25 / 55.4 / 4.76 | 94.00 / 60.4 / 7.98 | 96.75 / 89.2 / 8.05 |
Response to Q3 (Part 2):
The allocation-specific results show why the fixed orders lose accuracy. At K=128 and b=64, Sequential A-to-B achieves 92.0% when A holds the clue and 9.5% after the evidence packages are exchanged. Sequential B-to-A achieves 10.0% and 91.5% under the same two allocations. Each fixed order therefore averages to 50.75%. When the detailed-evidence holder reports first, its 24-token report must be prepared before the relevant route is known.
Under the original and exchanged allocations, bidirectional exchange achieves 92.5% and 91.0%, while Follow-up achieves 96.0% and 92.0%. Both procedures maintain high accuracy across the two allocations. Their pooled accuracies are 91.75% and 94.00%.
The Follow-up-minus-bidirectional accuracy differences across the three settings are +0.75 points [-1.00, 2.75], +2.25 points [0.25, 4.50], and -0.25 points [-2.25, 1.50]. These pointwise 95% intervals use 100,000 paired bootstrap replicates. Each replicate resamples 200 base worlds as clusters, retaining both evidence allocations and all compared procedures within each sampled world.
These results connect the useful communication schedule to the available dependency information. A fixed direction supports a single sequential exchange. When evidence holders vary, bidirectional exchange can cover both directions, while Rule and Follow-up select further communication using received messages. At K=128 and b=64, Rule reaches 93.25% with 6.25k generated tokens, compared with 94.00% and 7.98k for Follow-up. The paired accuracy difference is +0.75 points [-1.25, 2.75].
The extension therefore examines dependencies whose direction becomes identifiable from initial messages. Dependencies that emerge through several rounds of evidence interpretation remain a further setting for evaluating these communication procedures.
Response to Q4 (Part 1):
We agree that evaluating a stronger model from another family would help clarify how the findings depend on model capability. Our original study focused on understanding the sources of a follow-up improvement through controlled comparisons of reporting, clue timing, and evidence access. We held the model fixed within each comparison and examined these mechanisms using two Qwen models with different reasoning modes. Your suggestion adds an important question. A stronger model may construct more informative reports, use received evidence more effectively, or make better decisions about when and how to ask a follow-up question. Each improvement could change the accuracy and cost differences between the communication procedures.
We would therefore like to examine this question while preserving the controls that make the comparisons interpretable. However, our access to GPT-6-Sol and Claude-Opus-5 is currently constrained by the applicable API access policies. The financial requirements also affect the scale of the evaluation we can undertake. Each instance requires several complete communication procedures, with model calls for report preparation, revision, communication decisions, and final answers. Repeating these procedures across evaluation worlds and model families creates a cumulative requirement for API usage. These considerations led us to seek an accessible alternative that would allow us to retain the essential comparisons within our research budget.
We selected DeepSeek-V4-Pro for a focused supplementary evaluation because it introduces a flagship reasoning model from a different family. We retained the task definitions and controlled resource limits, and compared strengthened independent reporting, Sequential, and Follow-up. The two routing settings examine both a case where a complete representation fits the communication budget and a case with greater reporting demands.
The evaluation uses 50 worlds at each of two Permutation settings, K=32 and K=128, with communication budget b=64. All agents use DeepSeek-V4-Pro. The compared procedures use the same worlds, generation settings, and total resource ceilings. Reporting configurations are fixed using separate development worlds. Sequential and Follow-up both use private revision, with matched first-report ceilings of eight tokens for A and 28 for B.
Communication counts complete messages and every transmitted copy using the paper's reference tokenizer. Generation uses API-reported output usage across all calls, including reasoning and revision. The common reader is the same deterministic decoder used in the paper. API response fields are mapped to the existing record schema through predefined deterministic rules using only the recorded message fields. The decoder then applies its fixed parsing and recovery rules to the public task and messages available to D. Ground-truth labels are used for scoring.
Thank you for your careful assessment and detailed questions. We appreciate your recognition of the reporting baselines, controlled diagnostics, and resource accounting. We understand that your main concern is how far the resulting insights extend beyond the tested workflows and models. Your questions also help identify the controls needed to interpret the differences between complete communication procedures.
We organize our response around these concerns. We examine report preparation under matched message limits, assess private-evidence access while preserving the main Follow-up reports and query decisions, and compare communication procedures when evidence holders vary. We also address the stronger-model question through a focused comparison with an additional model family within our access and funding constraints. These comparisons serve our broader aim of helping designers identify which improvements require better reports, better use of received information, or further evidence access. The closing discussion connects this guidance to a concrete troubleshooting workflow and clarifies the scope of the empirical findings.
Response to Q1:
We agree that matching both report preparation and report budgets makes the generation-cost comparison easier to interpret. Our original comparison evaluated the complete procedures selected for each protocol. To examine the contribution of report preparation more directly, we compare Sequential and Follow-up under both Report and Revise while fixing the first-report allocations.
The comparison uses the same 200 Permutation worlds at K=32 and b=64, with the same Qwen3-14B model, report encoding, and generation ceilings. A's first report has an eight-token ceiling, and B's first report has a 28-token ceiling. For Sequential, Report or Revise applies to A's report and B's report conditioned on A's clue. For Follow-up, the preparation choice applies to the two initial independent reports. The subsequent query, reply, and decision rules remain fixed across its preparation conditions. Communication charges every transmitted copy, including the two copies of A's report in Sequential.
Follow-up uses an additional 2.96k generated tokens when both protocols use Report and 3.23k when both use Revise. Changing Report to Revise adds 0.55k tokens for Sequential and 0.82k for Follow-up.
The corresponding accuracy changes from revision are +0.5 percentage points with a pointwise 95% paired interval of [-1.5, 3.0] for Sequential and +1.5 points [-1.0, 4.0] for Follow-up. These intervals use 100,000 bootstrap replicates over the 200 paired worlds. The accuracy effects of revision remain uncertain at this sample size, while mean generation increases in both procedures.
The original manuscript reports Sequential at 97% accuracy and 1.20k generated tokens under its original conditional-report allowance. This supplementary comparison instead fixes B's first-report ceiling at 28 tokens in both protocols. A mean generation gap remains after preparation and first-report budgets are matched. The remaining difference concerns the complete procedures. Sequential prepares B's report after receiving the clue, while Follow-up prepares independent reports before deciding whether to request further evidence.