acceptodds
Under review as a conference paper at ICLR 2027

Right Answers, Wrong States: Hidden Information Failures in Multi-Agent Collaboration

Abstract

Multi-agent systems are often judged by whether they reach the correct answer. This can miss a distinct failure: collaboration may leave behind a corrupted information state even when the immediate decision is correct. We call this an off-query failure. To study this failure in collaborative decision support, we introduce OffQuery, which separately evaluates evidence verification (T1), shared-state reconstruction (T2), and task resolution (T3) in two representative high-stakes settings: healthcare and disaster response. Across GPT, Gemini, and Qwen models, standard collaboration shows much stronger task performance than state reliability. Averaged over 21 model–setting combinations, task resolution reaches 64.7%, while evidence verification and state reconstruction reach only 14.3% and 43.1%. We trace this gap to selective information use: current queries often bypass corrupted facts, which become consequential when later tasks require them. We further introduce ReGround, which resolves conflicting evidence, verifies shared facts, reconstructs a trusted state, and reasons over that state. Across seven models from three families, ReGround improves all three capabilities in every evaluated setting, with average relative gains of 309.0%, 82.9%, and 17.6% on T1, T2, and T3. Reliable collaboration therefore requires both a correct decision and a reliable shared state for future reasoning.

Primary area:
datasets and benchmarks
Keywords:
multi-agent systems, collaborative reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8YkeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper talks about how multi agent LLMs can get the answer correct for a given query for a collective decision making task, but the "state" might be erroneous, which the authors refer to as "off query failure". The authors introduces OffQuery, based on established datasets from medical domain and disaster preparedness domain, and evaluate three things:

  • T1: identify which agent is the one with misleading info
  • T2: reconstruct the shared state
  • T3: answer the query correctly Based on the evaluation and diagnosis, it proposed ReGround, a framework that improves multi-agent LLMs performance on all three metrics, especially targeting at T1 and T2.
Critical strengths
  • In general I like this problem the authors are targeting at, and I like how the authors are spending effort on building a evaluation framework to seperate state reliability from task accuracy. This is especially clear with the modeled shared state C*, where the benchmark can score what survives collaboration independently of the answer. And this is validated with LLM judges plus human annotators. But I also have some concerns about this part, see the weaknesses part below.
  • I also like the arc from problem formulation, to benchmark construction, to evaluation and diagnosis, and to mitigation. Especially the last two phaes on that arc, where we can see clear interventional evidence that state quality drives downstream accuracy.
Critical weaknesses
  • I'm quite iffy about how the benchmark is actually testing the failure. The paper’s main concern is errors that goes unnoticed in the shared state because the query never touches them. But in OffQuery, the false information is generated specifically to push the model toward the wrong answer on the query. So if I understand correctly, when the model still answers correctly, it has resisted a direct attack, and it's not just simply the misleading info is left unused. When I skim through the appendix, though, I do see one small experiment (E.2) actually checks whether errors show up on a different question later. I understand the benchmark tries to isolate what it wants to tests, which I value, but the setup also makes the false information quiet easy to catch, as there is always exactly one misleading agent among truthful ones, its claims are always contradicted by someone else, and it may even sound different from the others in style (e.g. the very example you show in the teaser figure in intro). Thus the main results do show how well models handle obvious, targeted misinformation, but not really whether hidden errors survive in realistic shared states, which is the paper’s central claim. I'm willing to hear more about the authors' thoughts here. Please correct me if I'm wrong.
  • The gains produced by ReGround is a bit "inflated" by weak baselines. I was surprised to see the massive gains of T1 by 309%. But the standard collaboration baseline scores about the same as random guessing on this task as you measured. This is likely because it only sees the agents’ opinions after discussion, by which point the misleading agent’s input has been mixed in with everyone else’s. T2 has a similar issue, where the baseline agents are told to share information “briefly”, so much of their low score comes from leaving out true facts, not from keeping false ones. Since the paper is about false information surviving in the shared state, these two problems make it quite hard to tell how much ReGround actually helps with that.
  • A minor issue: I understand ReGround is very specific to this task as it's informed directly by the failure diagnosed. Right now it's so closely coupled with this task setting, that I can barely see its generality to more general settings. I understand how the arc progressed, and thus I suggest the author frame ReGround as evidence of the benchmark, which is the main contribution, and the diagnosis are actionable, instead of framing this framework as a standalone main thing on its own.
  • And a even more minor issue: GPT-4o is doing a lot of work in this paper. I encourage the authors to switch to newer models, as we don't want model's capability as judges for example, to cause issues.
Questions for authors

These questions of mine are coupled with the first two weaknesses I pointed out:

  • On weakness 1: (a) As I talked about earlier, the false information in OffQuery is designed to change the answer to the current question, so it isn’t really “off-query.” Could you add an experiment where the false information is unrelated to the current question, and then test whether it causes errors on a later, different question? (b) In that E2 experiments I pointed out, 43% of plans were judged inconsistent. How often exactly are the two plans built from clean states judged inconsistent? Without that comparison, it’s hard to tell how much of the 43% is caused by the false information.
  • On weakness 2: (a) As I mentioned earlier, the standard baseline identifies the misleading agent at about the rate of random guessing. How does ReGround compare to a fairer baseline that sees the agents’ original information instead of their post-discussion opinions? Please report the absolute differences, not just percentages. (b) Could you split the T2 score into two parts: how many true facts are kept and how many false facts are removed? Only the second part measures the problem the paper is about.

If these questions are answered with good quality and we see the headline results still hold, I will consider raising my score.

5 comments
AuthorsOpenReview ↗

Reply (1/3)

Thank you for the constructive review and for recognizing the value of evaluating shared-state reliability independently of task accuracy. We have conducted additional experiments addressing your specific questions and clarify the existing evidence below.

W1 & Q1(a). Does OffQuery capture genuine off-query failures?

Could you add an experiment where the false information is unrelated to the current question, and then test whether it causes errors on a later, different question?

The original answer-targeted construction ensures that misleading information can affect downstream reasoning if accepted. Whether this information influences the current decision depends on which facts the model uses.

Our existing analyses provide evidence of this distinction:

  • Selective information use (§E.1 in submission): Misleading facts appear in only 13.9%/19.2% of correct-answer explanations, compared with 60.9%/71.7% of incorrect-answer explanations. This pattern is consistent with correct decisions relying on unaffected portions of the shared state.
  • Subsequent information reuse (§E.2 in submission): Among 100 instances with correct original answers and residual misleading information, 43% of broader clinical plans are judged inconsistent with plans generated using reliable reference states.

To further examine the role of answer-targeted corruption, we introduce an additional question-blind misinformation setting. We sample 100 instances from Healthcare-S and Healthcare-I and generate misleading information using GPT-4o conditioned on the reliable patient context, without providing the original downstream question. The generated statement is then incorporated into the patient context. We evaluate the same instances under the original answer-targeted setting for comparison.

We evaluate GPT-4o and Gemini-3.5-Flash under both settings:

Model / MethodH-S T1H-S T2H-S T3H-I T1H-I T2H-I T3
GPT Standard, answer-targeted.070.308.660.100.269.690
GPT Standard, question-blind.160.421.840.130.475.840
GPT ReGround, answer-targeted.560.719.770.550.750.780
GPT ReGround, question-blind.880.739.870.860.724.880
Gemini Standard, answer-targeted.140.442.820.140.463.840
Gemini Standard, question-blind.120.464.890.140.527.860
Gemini ReGround, answer-targeted.720.827.920.690.839.900
Gemini ReGround, question-blind.730.668.930.740.760.910

Two findings are particularly relevant:

  1. Current-task accuracy remains high despite unreliable reconstructed states. Under question-blind corruption, Standard achieves 84–89% T3 accuracy, while T2 remains at 42.1–52.7%. This demonstrates a substantial gap between current-task success and retained-state reliability without explicitly targeting an incorrect answer during misinformation generation.
  2. ReGround consistently improves state reconstruction. Under question-blind corruption, ReGround achieves higher T2 and T3 than Standard across both models and healthcare settings.

We additionally apply the broader clinical planning evaluation from §E.2 in submission to the question-blind setting. Using GPT-4o, the plan inconsistency rates are:

DatasetStandardReGroundReduction
Healthcare-S54%26%28 pp
Healthcare-I58%28%30 pp

These results show that unreliable retained information continues to affect broader clinical reasoning under question-blind corruption. ReGround substantially reduces these downstream inconsistencies. We add these results in §6 (lines 447-474) and Table 4 in updated manuscript.

W1 & Q1(b). Clean-clean control for broader clinical planning

How often are two plans generated from clean states judged inconsistent?

We conduct an additional clean-clean control to quantify natural variation in clinical plan generation. We sample 100 instances where T2 and T3 are correct and generate two plans using clean states under the same evaluation protocol.

The clean-clean inconsistency rate is 12%, compared with 43% for the corrupted-clean comparisons reported in §E.2 in submission.

The 12% result provides an empirical reference for natural plan-generation variability. The observed corrupted-clean inconsistency rate is 31 percentage points higher. These analyses use separately selected samples, so we interpret the comparison as supporting evidence. Our controlled state-repair experiments (§E.3 in submission) provide complementary interventional evidence that improving shared-state reliability improves downstream reasoning. We add these results and analysis in §6 (lines 441-446) in updated manuscript.

AuthorsOpenReview ↗

Reply (2/3)

W2 & Q2(a). Fairer verification baselines and original evidence access

How does REGROUND compare with a baseline that accesses agents' original information? Please report absolute differences.

Our submission already includes a centralized T1 verifier with direct access to all agents' original private observations (§D.3, Table 1 in submission).

For GPT-4o, the T1 results are:

DatasetStandardReGroundCentralized verifier
Healthcare-S15.2%56.7%65.3%
Healthcare-I14.0%54.5%68.0%

Relative to Standard, ReGround improves T1 by 41.5/40.5 percentage points on Healthcare-S/I. The centralized verifier achieves 65.3/68.0%, confirming that direct access to original private evidence can substantially improve source identification.

To further examine the impact of original evidence access on state reconstruction and downstream reasoning, we conduct additional comparisons with two variants:

  • Retain: Preserve all agents' original observations, including those of the source identified as misleading in T1.
  • Remove: Exclude the T1-identified misleading source before subsequent reasoning.

Both Standard and ReGround retain their respective reasoning procedures.

SettingH-S FRRH-S T3H-I FRRH-I T3
Standard, Original.211.669.242.627
Standard, Retain.174.495.166.476
Standard, Remove.196.612.199.574
ReGround, Original.657.766.668.661
ReGround, Retain.648.633.631.617
ReGround, Remove.683.888.705.792

The results reveal three patterns:

  1. Original evidence access alone provides limited benefits. Retaining all observations decreases T3 accuracy for both methods, indicating that unfiltered evidence can introduce additional interference.
  2. Source filtering improves downstream reasoning. For ReGround, Remove improves T3 over Retain by 25.5/17.5 percentage points on Healthcare-S/I, alongside FRR gains of 3.5/7.4 points.
  3. ReGround's advantages persist under matched evidence-access conditions. In the Remove setting, ReGround exceeds Standard by 48.7/50.6 points in FRR and 27.6/21.8 points in T3 on Healthcare-S/I.

These comparisons demonstrate that ReGround's improvements in misinformation removal and task resolution persist when both methods operate under the same original-evidence access protocol. They further highlight the importance of explicit verification and state reconstruction. We add these results and analysis in Table 14, §5.2 (lines 403-418), and §D.9 in updated manuscript.

W2 & Q2(b). Separating true-fact retention and false-fact removal

Could you split T2 into how many true facts are retained and how many false facts are removed?

Yes. Our existing atomic-fact annotations (§B.5 in submission) separately identify reliable and misleading facts, allowing us to decompose T2 into:

  • True-fact Retention Rate (TRR): TP / (TP + FN)
  • False-fact Removal Rate (FRR): TN / (TN + FP)

We report the results for GPT-4o and Gemini-3.5-Flash across all three datasets here and complete results in Table 11 in updated manuscript:

Model / MethodH-S TRRH-S FRRH-I TRRH-I FRRDisaster TRRDisaster FRR
GPT Standard.335.211.331.242.653.513
GPT ReGround.807.657.787.668.777.845
Gemini Standard.472.422.465.413.664.656
Gemini ReGround.840.794.825.801.799.910

ReGround consistently improves both true-fact retention and false-fact removal across the evaluated models and datasets.

On Healthcare-S/I, FRR improves by 42.6–44.6 percentage points for GPT-4o and 37.2–38.8 points for Gemini.

The improvements are also substantial in Disaster Response:

  • GPT-4o: FRR improves by 33.2 points (51.3% vs. 84.5%), alongside a 12.4-point TRR gain.
  • Gemini: FRR improves by 25.4 points (65.6% vs. 91.0%), alongside a 13.5-point TRR gain.

These results directly address the concern that low T2 scores may primarily reflect omitted reliable facts. ReGround achieves substantial improvements in removing misleading information while preserving more reliable facts. The gains therefore reflect improved retained-state reliability across both dimensions. We include the complete decomposition in §D.5 and Table 11 in updated manuscript.

AuthorsOpenReview ↗

Reply (3/3)

W3. Contribution and framing of ReGround

I suggest framing REGROUND as evidence that the benchmark identifies actionable failures.

We agree that OffQuery and its diagnosis are the central contributions. The benchmark provides an evaluation framework for identifying retained-information failures that current-task accuracy misses. ReGround demonstrates that the diagnosed failures can be mitigated through explicit evidence verification and shared-state reconstruction.

We will revise the introduction and contribution summary to emphasize this relationship and clarify ReGround's role as an actionable mitigation of the failure identified by OffQuery.

W4. Reliance on GPT-4o

I encourage the authors to switch to newer models.

Our main experiments already cover newer models, including GPT-5, GPT-5-mini, and Gemini-3.5-Flash (Table 1 in submission), alongside additional Qwen models.

For T2 evaluation, §B.6 independently validates GPT-4o-based judgments using Gemini-3.5-Flash and Qwen3-235B-A22B. All three judges produce identical aggregate rankings across the evaluated configurations. §B.7 in submission further reports human validation with 0.88/0.84 agreement between GPT-4o and human annotators.

GPT-4o is retained as a fixed model for controlled benchmark construction, while the cross-model and human evaluations assess the reliability of the resulting fact-level scoring.

Reviewer_zYxCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a failure mode in multi-agent systems where the final answer can be correct while the shared state still contains erroneous information. The authors construct OFFQUERY, decompose system capability into evidence identification (T1), state reconstruction (T2), and task resolution (T3), and build multi-agent tasks with synthetic misleading information based on MediQ and CrisisFact.The paper further proposes REGROUND, which repairs the shared state through conflict detection, candidate-source screening, evidence verification, fact filtering, and context reconstruction. The experiments report that standard collaboration performs substantially worse on T1 and T2 than on T3, whereas REGROUND improves all three metrics in the authors' constructed evaluation setting.

Critical strengths
  1. Meaningful evaluation perspective. The paper clearly distinguishes final-answer correctness from shared-state reliability, showing that task accuracy alone may hide erroneous information that can be reused in later tasks. This is a practically important issue.
  2. Clear diagnostic framework. The T1/T2/T3 decomposition supports separate analysis of evidence identification, state reconstruction, and task resolution, making failures at different stages easier to observe.
Critical weaknesses
  1. The distinction from prior work is insufficiently established. Error propagation, erroneous consensus, unreliable agents, multi-agent verification, and cases where the final answer is correct despite an unreliable process have already been studied extensively. The clearest contribution here is the T1/T2/T3 framing, while REGROUND largely combines existing steps such as conflict detection, source verification, fact filtering, and state reconstruction. The paper should define its conceptual and technical novelty relative to related work more precisely.
  2. The method closely mirrors the benchmark construction, so the current results do not establish generality. Although the appendix adds multiple unreliable sources or incorrect answers, these settings retain the same generation mechanism: first choose an incorrect answer, then generate misleading information that supports it and can be refuted by other evidence. REGROUND's conflict detection, source verification, removal, and reconstruction effectively reverse this pipeline. The multi-error experiments change the number of errors, not their structure. The results therefore establish effectiveness only for OFFQUERY's specific construction, rather than for natural retrieval or tool errors, mutually reinforcing errors, or errors without direct counterevidence.
  3. The necessity of the multi-agent design is not demonstrated. REGROUND assigns conflict detection, source verification, information filtering, and state reconstruction to multiple agents, but the same steps could be performed sequentially by a single agent. The paper does not include a single-agent baseline using the same input and model while matching the number of model calls, token budget, and reasoning rounds. It is therefore unclear whether the gains come from multi-agent collaboration or from staged prompting, additional computation, and repeated verification. A strictly budget-matched single-agent comparison is needed to establish any independent advantage of the multi-agent design.
Questions for authors
  1. Why does REGROUND require a multi-agent design? Compared with a single-agent baseline using the same input and model and matched for the number of calls and token budget, does REGROUND still provide a significant advantage?
  2. Is REGROUND effective only for the error patterns predefined by OFFQUERY? Can the authors evaluate other error structures, such as multiple sources jointly supporting the same false claim, a single source mixing correct and incorrect information, or misleading information that does not directly conflict with reliable evidence?
  3. What is REGROUND's substantive novelty relative to prior work on error detection, source verification, and multi-agent error correction? Beyond decomposing the problem into T1, T2, and T3, what new method or capability does the paper introduce?
4 comments

Thank you for recognizing the importance of evaluating shared-state reliability and the value of the T1/T2/T3 diagnostic framework. We clarify our contributions, robustness results, and methodological scope below.

W1 & Q3. Novelty and contributions

The distinction from prior work is insufficiently established. What new method or capability does REGROUND introduce beyond the T1/T2/T3 decomposition?

OffQuery contributes a distinct evaluation target: the reliability of information retained after a correct collaborative decision. Prior work has examined error propagation, unreliable consensus, and downstream task failures. OffQuery explicitly measures residual state corruption alongside current-task correctness, using a known reference state and separate T1/T2/T3 evaluations. Our analyses further examine how retained errors affect subsequent reasoning (§6, §E.1–E.3 in submission).

ReGround operationalizes explicit shared-state maintenance through conflict-guided source verification, fact-level filtering, and reconstruction of a reusable trusted state. Its design follows the principle of verify before sharing, repair before reasoning: private evidence is verified before potentially corrupted shared context is introduced, and downstream reasoning uses the reconstructed state (§4 in submission). Figure 4 in submission examines verification order, while Table 10 in submission demonstrates the importance of source-attribution quality.

We recognize that the individual verification operations are established techniques. ReGround provides an integrated mitigation targeting the shared-state failures identified by OffQuery.

W2 & Q2. Robustness across corruption structures

The method closely mirrors the benchmark construction. Can REGROUND handle other error structures, including multiple unreliable sources and mixed correct/incorrect information?

ReGround operates solely on distributed observations and potentially corrupted shared information at inference time. It has no access to the target incorrect answer, ground-truth state, or benchmark generation procedure.

Our submission already evaluates two more complex corruption structures (§6, §E.4 in submission):

  1. Multiple unreliable agents: Three agents contain misleading information. ReGround achieves 60.8% performance, compared with 45.2% for standard collaboration (+15.6 percentage points).
  2. Mixed reliable and misleading information within agents: Misleading facts are distributed across agents, requiring fact-level verification. ReGround achieves 49.7%, compared with 26.4% for standard collaboration (+23.3 points).

The second setting changes the verification structure by introducing reliable and misleading facts within the same agents. ReGround remains effective when identifying an unreliable source is insufficient and verification must operate at the fact level.

These results demonstrate robustness across different distributions of unreliable information. The experiments retain controlled misinformation generation, and we will clarify this scope. Correlated falsehoods, errors without direct counterevidence, and naturally occurring retrieval or tool failures remain important directions for broader validation.

W3 & Q1. Role of multi-agent collaboration

Why does REGROUND require a multi-agent design? Does it outperform a single-agent implementation under matched computational budgets?

ReGround targets collaborative settings where complementary and potentially conflicting information is initially distributed across agents (§3.2 in submission). Its verification and reconstruction stages explicitly maintain the reliability of information shared across participants.

Our experiments demonstrate that this state-maintenance procedure substantially improves T1/T2/T3 over standard collaboration across models and domains (Table 1 in submission), with consistent T3 improvements over representative collaboration strategies (Table 3 in submission).

The Oracle setting (§D.3 in submission) includes centralized single-model verification over original private evidence for T1. For T2/T3, it uses the complete reliable global state as an upper-bound reference. Table 11 additionally reports token consumption across collaboration frameworks. We will clarify the roles of these comparisons.

Our current results establish the benefits of explicit verification and state reconstruction in distributed-information collaboration. A strictly budget-matched single-agent implementation would further isolate the contribution of the multi-agent architecture. We will clarify the scope of our architectural claims accordingly.

AuthorsOpenReview ↗

Gentle reminder of the author-reviewer discussion deadline

Dear Reviewer zYxC:

We are eager to engage in further discussions with you! In our earlier responses, we have actively addressed your concerns by providing:

  1. Clarifications on the conceptual novelty of OFFQUERY and the contribution of REGROUND, emphasizing the evaluation of retained-state reliability and explicit state maintenance.
  2. Existing robustness experiments across different corruption structures, including multiple unreliable agents and mixed reliable/misleading information within agents, with absolute performance gains of 15.6 and 23.3 percentage points.
  3. Clarifications on the role of multi-agent collaboration and the scope of existing comparisons, including centralized evidence verification, representative collaboration baselines, and computational costs.

As the author-reviewer discussion deadline is approaching, please let us know if you have any additional questions or concerns about the paper. We would be delighted to continue the conversation with you.

We sincerely hope that our responses have effectively addressed your concerns and may encourage a more favorable reconsideration of our paper.

Best regards,

Submission 272 Authors

Thank you for the detailed responses. I think the rebuttal has substantially strengthened the paper and addressed most of my original concerns.

AuthorsOpenReview ↗

Glad our responses helped address your concerns

Thanks for your support! We are glad that our responses have helped address your concerns and strengthen the paper.

Your feedback prompted us to further clarify the central contribution of OFFQUERY in evaluating retained-state reliability, the role of REGROUND in mitigating these failures, and the robustness of our findings across different corruption structures. These discussions have helped us sharpen the paper's contributions and scope.

We sincerely appreciate your constructive feedback and engagement.

Reviewer_q4J8OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies an overlooked issue in multi-agent collaboration: cross-query failure, where a system produces a correct answer for the current task but leaves behind a corrupted or unreliable Shared Information State, potentially compromising future multi-turn reasoning and broader downstream tasks. To investigate this issue, the authors introduce OFFQUERY, a benchmark covering healthcare and disaster response that evaluates evidence verification, shared-state reconstruction, and task solving. Experiments reveal that mainstream models frequently exhibit a gap between task correctness and state reliability. The authors further propose REGROUND, a framework that improves information-state reliability through conflict-driven evidence verification and atomic fact reconstruction.

Critical strengths

The paper highlights a limitation of existing multi-agent research, which primarily focuses on final-answer accuracy, and exposes the hidden risks of unreliable intermediate states behind seemingly correct answers. OFFQUERY is clearly structured into three sequential tasks, T1–T3, and provides a comprehensive evaluation in high-stakes domains, including healthcare and disaster response.

Critical weaknesses
  1. The evaluation and mitigation of erroneous information states are relatively simplistic. REGROUND verifies private evidence to identify misleading sources, decomposes the initial shared context CC into atomic facts, validates each fact against trusted evidence, removes contradicted or unsupported facts, and consolidates the retained facts with trusted private evidence into a coherent natural-language state C^\hat{C} using an LLM. These operations closely resemble existing fact-checking and evidence-grounded reasoning methods. Moreover, the approach does not directly assess the internal states of different LLMs. An analysis incorporating model-internal representations and information-theoretic perspectives could offer deeper insights into information-state corruption and reliability.

  2. Potential evaluation bias and limited human validation undermine the robustness of the findings. GPT-4o is used both to generate misleading evidence and to evaluate T2, potentially introducing model-specific biases. Human validation covers only 50 instances (200 pairs), which is relatively limited in scale and does not include disaster-response scenarios.

  3. More directly comparable baselines are needed. Although the paper compares several collaboration strategies, it lacks simple verification-and-reconstruction baselines that are directly comparable to REGROUND, such as centralized evidence verification by a single model or independent fact filtering followed by direct reasoning.

  4. The source of REGROUND's performance gains remains unclear. REGROUND and the baselines differ in their reasoning pipelines and token consumption. Comparing only final performance is therefore insufficient to determine whether the improvements stem from the specific conflict-driven mechanism or simply from explicit evidence verification and state reconstruction.

Questions for authors

See Weakness.

4 comments
AuthorsOpenReview ↗

Reply (1/2)

Thank you for the detailed review and for recognizing the importance of evaluating shared-state reliability. We clarify our evaluation target, existing validation results, and methodological comparisons below.

W1. Evaluation target and novelty

These operations closely resemble existing fact-checking and evidence-grounded reasoning methods. Moreover, the approach does not directly assess the internal states of different LLMs.

Our shared information state refers to the explicit, reusable information retained after collaboration (§3.1 in submission). Model-internal representation analysis concerns a separate level of analysis outside our evaluation scope.

The central contribution of OffQuery is systematically evaluating cases where correct answers coexist with corrupted retained information. By separately measuring evidence verification (T1), shared-state reconstruction (T2), and task resolution (T3) against a known reference state, OffQuery identifies reliability failures that final-answer accuracy cannot capture. Our analyses further examine selective information use and the consequences of residual corruption for subsequent reasoning (§6, §E.1–E.3 in submission).

ReGround provides a targeted mitigation through evidence verification and state reconstruction. We emphasize the novelty of the evaluation framework and its diagnosis of shared-state failures, while recognizing that the mitigation builds on established verification operations.

W2. Evaluation reliability and human validation

GPT-4o is used both to generate misleading evidence and to evaluate T2, potentially introducing model-specific biases. Human validation covers only 50 instances (200 pairs) and does not include disaster-response scenarios.

Our submission already includes independent cross-model and human validation of T2 evaluation:

  • Cross-model validation (§B.6 in submission): Three heterogeneous judges, GPT-4o, Gemini-3.5-Flash, and Qwen3-235B-A22B, evaluate seven method configurations. All three produce identical aggregate rankings (pairwise Spearman ρ = 1.00), with instance-level pairwise MAEs of 0.126–0.139 (Tables 5–6 in submission).
  • Human validation (§B.7 in submission): Two annotators independently evaluate 200 blinded context–fact pairs. GPT-4o achieves 0.88/0.84 agreement with the annotators, compared with 0.92 human–human agreement (Table 7 in submission).

These results demonstrate consistent comparative findings across independent judges and substantial agreement with human assessments. The current human validation covers healthcare instances; we will clarify its scope and the remaining limitations in domain coverage.

AuthorsOpenReview ↗

Reply (2/2)

W3. Centralized verification and comparable baselines

The paper lacks simple verification-and-reconstruction baselines that are directly comparable to REGROUND, such as centralized evidence verification by a single model or independent fact filtering followed by direct reasoning.

Our submission already evaluates centralized single-model evidence verification for T1 through the Oracle setting (§D.3, Table 1 in submission). A single model receives all agents' original private observations and independently identifies the misleading source. On GPT-4o/Healthcare-S, this centralized verifier achieves 65.3% T1, compared with 56.7% for ReGround.

For T2/T3, Oracle uses the complete reliable global state as an upper-bound reference. We will clarify the distinct roles of centralized T1 verification and the clean-state Oracle, including their relationship to a matched end-to-end verification-and-reconstruction baseline.

OffQuery studies collaboration with distributed, potentially conflicting observations (§3.2 in submission). ReGround explicitly verifies evidence and reconstructs a reliable shared state before downstream reasoning. Tables 1 and 3 demonstrate substantial improvements over standard collaboration and representative collaboration strategies. Our contribution focuses on the effectiveness of explicit state maintenance in distributed-information settings. We make no claim of an inherent advantage of multi-agent execution over a compute-matched centralized implementation.

W4. Sources of improvement and computational cost

It remains unclear whether the improvements stem from the specific conflict-driven mechanism or simply from explicit evidence verification and state reconstruction.

Our submission includes analyses examining verification quality, information-sharing order, and computational cost:

  • Verification quality (Table 10, §D.6 in submission): Candidate-pool size affects T1/T2/T3 performance. Providing the correct misleading source improves T2 by 8.5 percentage points (0.728 to 0.813) and T3 by 17.6 points (0.661 to 0.837), demonstrating the downstream importance of accurate source attribution.
  • Information-sharing order (Figure 4, §6 in submission): Introducing corrupted shared context before private-evidence discussion consistently degrades performance. This supports ReGround's design of verifying private evidence before reconstructing the shared state.
  • Computational cost (Table 11, §D.7 in submission): On GPT-4o/Healthcare-S, ReGround consumes 19.4K tokens per instance, compared with 52.1K for standard collaboration. The improvement over this baseline therefore cannot be attributed to greater token consumption.

These analyses establish the importance of verification quality and information-sharing order, while characterizing the computational cost of ReGround. We will clarify that our results evaluate the integrated state-maintenance procedure, including evidence verification and reconstruction, without assigning the full performance gain to an individual component.

Our primary contribution is OffQuery's evaluation of retained-information failures that remain undetected by current-task accuracy. ReGround demonstrates that explicit verification and state repair can substantially improve reliability across models and domains.

AuthorsOpenReview ↗

Gentle reminder of the author-reviewer discussion deadline

Dear Reviewer q4J8:

We are eager to engage in further discussions with you! In our earlier responses, we have actively addressed your concerns by providing:

  1. Clarifications on the evaluation target and conceptual novelty of OFFQUERY, supported by analyses of residual state corruption and subsequent information reuse.
  2. Existing cross-model judge consistency and human validation results supporting the reliability of our evaluation.
  3. Clarifications on the centralized evidence verification baseline and the distinct roles of the Oracle settings.
  4. Existing analyses of verification quality, information-sharing order, and computational costs supporting REGROUND's design and effectiveness.

As the author-reviewer discussion deadline is approaching, please let us know if you have any additional questions or concerns about the paper. We would be delighted to continue the conversation with you.

We sincerely hope that our responses have effectively addressed your concerns and may encourage a more favorable reconsideration of our paper.

Best regards,

Submission 272 Authors

Thank you for the detailed responses and additional clarifications. The cross-model evaluation and human validation provide stronger support for the reliability of the T2 evaluation, and the token-consumption analysis helps address concerns regarding computational efficiency.

However, some of my concerns remain unresolved. First, although the authors clarify that the primary novelty lies in the OFFQUERY benchmark, the methodological contribution of REGROUND remains relatively limited, as its core operations largely build upon established evidence verification and fact reconstruction techniques. Second, the centralized verification experiment only addresses T1 and does not provide a directly comparable end-to-end baseline for shared-state reconstruction and downstream reasoning, as requested in W3. Third, the ablation studies do not sufficiently isolate the contributions of conflict-driven verification from those of general evidence verification and state reconstruction.

Although I still believe that the novelty of this paper is limited and that their response has not fully addressed all my concerns, I will increase my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the AC

Dear AC,

We appreciate your time and effort in overseeing the review process. We summarize the reviewers' shared assessment of our contributions, the positive feedback received during the rebuttal, and the additional evidence incorporated into our revised manuscript.

Consensus on Strengths

All three reviewers recognized the importance of evaluating shared-state reliability beyond final-answer accuracy. Their reviews highlighted two central strengths:

  1. Important Evaluation Perspective: OffQuery exposes a consequential failure in multi-agent collaboration, where correct task outcomes can conceal unreliable shared information. Reviewer q4J8 recognized the hidden risks of unreliable intermediate states, Reviewer zYxC described this as a "meaningful evaluation perspective," and Reviewer 8Yke emphasized the practical importance of the problem.
  2. Clear Diagnostic Framework: The T1/T2/T3 decomposition enables separate evaluation of evidence verification, shared-state reconstruction, and task correctness against a known reference state. All three reviewers acknowledged the value of this formulation. Reviewer 8Yke additionally highlighted our interventional evidence connecting state reliability to downstream outcomes.

Positive Feedback During Rebuttal

We are particularly encouraged by the positive feedback received during the author-reviewer discussion:

  • Reviewer 8Yke (score increased: 2 → 3) explicitly acknowledged that our new question-blind experiments, clean-clean control, and TRR/FRR decomposition addressed their main concerns. The reviewer concluded that "the main claims are better supported now" and raised their score to 3.
  • Reviewer zYxC (score: 3) explicitly acknowledged that "the rebuttal has substantially strengthened the paper and addressed most of my original concerns."

These responses provide direct confirmation that our rebuttal has strengthened the empirical evidence and clarified the contributions of OffQuery.

How We Addressed Each Reviewer's Concerns

We provided the following additional experiments and clarifications, with new results incorporated into the updated manuscript.

Reviewer 8Yke: Off-query validity and fair evaluation

  • Question-blind corruption: Standard achieves 84–89% current-task accuracy despite substantially lower shared-state reliability (42.1–52.7% T2). REGROUND further reduces downstream clinical-plan inconsistency by 28–30 percentage points.
  • Clean-clean control: A separate clean-state control yields 12% plan inconsistency, providing a reference for the original corrupted-clean analysis (43%).
  • Fairer evidence-access comparisons: Under the same original-evidence access protocol, REGROUND exceeds Standard by 48.7/50.6 pp in false-fact removal and 27.6/21.8 pp in task accuracy on Healthcare-S/I.
  • T2 decomposition: Separating true-fact retention (TRR) and false-fact removal (FRR) confirms substantial improvements in both dimensions, directly addressing the concern about residual misinformation.

Reviewer zYxC: Novelty, robustness, and multi-agent design

  • Contribution clarification: We established OffQuery's evaluation novelty in measuring retained-information failures despite correct task outcomes, with ReGround providing an actionable state-maintenance approach.
  • Robustness across corruption structures: We highlighted existing experiments demonstrating ReGround's effectiveness with multiple unreliable agents (+15.6 pp) and mixed reliable/misleading information within agents (+23.3 pp).
  • Methodological scope: We clarified the role of distributed information, the existing centralized verification setting, and the scope of our claims regarding multi-agent architecture.

Reviewer q4J8: Evaluation reliability, baselines, and mechanism analysis

  • Evaluation target: We clarified that shared information state refers to explicit, reusable information retained after collaboration, and explained OffQuery's conceptual distinction from model-internal representation studies.
  • Evaluation reliability: We highlighted existing cross-model judge consistency (Spearman ρ = 1.00 across aggregate method rankings) and human validation (0.88/0.84 agreement).
  • Baselines and ablations: We clarified the centralized T1 verifier and clean-state Oracle, alongside existing analyses of source-attribution quality, information-sharing order, and computational cost. On GPT-4o/Healthcare-S, ReGround uses 19.4K tokens, compared with 52.1K for Standard.

With the additional evidence and analyses, we are confident that OffQuery advances multi-agent evaluation by exposing retained-state failures hidden by correct task outcomes, while ReGround demonstrates an actionable mitigation. We appreciate the reviewers' constructive engagement and encouraging reassessments, and hope these strengthened contributions will receive favorable consideration in the final decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.