acceptodds
Under review as a conference paper at ICLR 2027

Agents Recognize Source Inconsistency but Rarely Act on It: Decision-Time Mediation for Grounded Resolution

Abstract

Agents routinely combine heterogeneous information, including user instructions, external web observations, and agent-maintained state, to make decisions. These sources can disagree in ways that leave no principled basis for privileging one source from the available evidence. Existing work typically assumes a predefined correct source, an authority hierarchy, or an adversarially injected signal; less studied is the broader case where multiple plausible sources support incompatible decision-relevant claims and the conflict remains unresolved at decision time. We term this problem **Source Inconsistency (SOIN)** and study whether agents can recognize it, how they behave when it remains unresolved, and whether explicit SOIN checking can improve grounded resolution. Across five models and controlled web-agent settings, we find a pronounced recognition–behavior gap: explicit SOIN detection reaches up to 100.00%, yet spontaneous resolution remains low, with agents instead showing systematic, source-dependent commitment. We introduce a training-free, model-agnostic pre-action mechanism that explicitly checks for SOIN and makes the checking result available to downstream mediation. In conflict cases, providing the full checking result raises grounded task success to 62.51–100.00%, reaching 99.13–100.00% for four of the five models, while success on no-conflict cases remains 98.82–100.00% and performance approaches the oracle condition for most models. The same recognition–behavior gap and intervention benefit persist when DOM-derived and visual observations disagree. Taken together, these results establish unresolved source inconsistency as a distinct reliability problem for agents and show that explicit SOIN checking provides a practical bridge from conflict recognition to grounded resolution.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Web Agents, Agent Reliability, Source Inconsistency, Conflicting Information, Grounded Decision Making, Agentic AI, Multi-Source Reasoning, Conflict Resolution

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 5 reviews, above 85% of the others.
Reviewer_i2zgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper studies Source Inconsistency (SOIN), a reliability problem that arises when web-based LLM agents encounter conflicting information from multiple sources, including user prompts, internal memory, and the external environment, without a clear basis for determining which source is correct. The authors find that although agents can readily recognize these inconsistencies, they often fail to resolve them during normal execution and instead tend to commit to one of the conflicting sources. To address this gap, the authors propose a mediation framework that builds on agents' strong inconsistency recognition capabilities. The framework first prompts agents to perform a diagnostic call to identify whether SOIN exists and which source pairs are in conflict, then uses this diagnosis to generate an intervention that guides the agent toward resolving the inconsistency. Experimental results on two adapted datasets demonstrate substantial improvements in grounded conflict resolution while preserving performance on non-conflicting tasks.

Critical strengths
  1. Important reliability problem supported by systematic empirical analysis. The paper identifies a meaningful recognition–behavior gap in LLM agents: models can reliably detect conflicting information when explicitly prompted, yet rarely initiate verification or clarification during normal execution. The controlled experimental design, including source-value swapping and matched no-conflict cases across five models, provides convincing evidence of systematic source-dependent behavior. Overall, it highlights a practical agent reliability issue that cannot be fully captured by conflict-detection accuracy alone.

  2. Substantial improvement through a simple, training-free intervention. The proposed decision-time mediation framework achieves considerable improvements in grounded conflict resolution while largely preserving performance on non-conflicting tasks. The comparisons between binary, full, and oracle inconsistency states, alongside component ablations, help demonstrate the importance of explicitly identifying and localizing conflicts before acting. Although the evaluation remains controlled, these results provide promising evidence for a practical mechanism to improve agent reliability.

The paper is also very well written and easy to follow and understand.

Critical weaknesses
  1. Limited comparison with existing or similar baselines. Although the paper provides extensive comparisons among different variants of SOIN-aware action mediation in Sections 5 and 6, along with some ablation studies in the appendix, it lacks comparisons against potentially relevant baselines. One of the most basic baselines, instead of the two-stage SOIN mediation, could be to add a single-line instruction to the original agent, prompting the model to check whether the user, webpage, and memory disagree, and to verify or ask before acting if necessary. Having such a baseline could clarify whether SOIN is even necessary beyond good prompting. I also found some baselines, including Ask-when-Needed (Wang et al., 2025) and CRITIC (Gou et al., 2024), that appear relevant, although they may need some adaptation. Given that the intervention achieves near-perfect performance in a controlled setting, such comparisons are important for understanding the method's actual contribution, practical advantages, and trade-offs compared with other existing approaches.

  2. The generalizability of the recognition–behavior gap and the mediation framework could be limited due to synthetic, single-decision-point evaluations. The paper is motivated by the need to address reliability challenges that do not arise in single-turn generation but instead exist in dynamically changing, complex environments in which agents are expected to act (lines 41–45 in the introduction). However, I found the experimental setting and tested datasets somewhat synthetic and not as complicated as the authors suggest in their motivation. Although the paper evaluates 1,500 conflict instances, these originate from only 100 base scene–field combinations, with different source-value assignments. Additionally, the experiments use frozen decision points, where the agent's internal state is simulated rather than accumulated through actual preceding interactions. While this design allows for controlled comparisons, it remains unclear whether the observed recognition–behavior gap and the intervention's effectiveness would persist in realistic, multi-step agent trajectories. In practice, conflicts may emerge gradually, involve implicit or partially contradictory information, and interact with previously accumulated errors.

  3. Simplified inconsistency detection. Somewhat related to my prior concern, the proposed definition of SOIN focuses on incompatible values assigned to the same decision-relevant variable. This makes the experimental inconsistencies relatively explicit, particularly because the sources follow controlled record structures. Real-world source disagreements, however, may involve semantic differences, temporal changes, incomplete statements, or contextual dependencies that are substantially harder to identify. This may be partly evident from the near-perfect detection scores in Tables 1 and 2.

  4. Insufficient analysis of natural source-arbitration behavior. Section 4 presents an interesting finding that agents rarely choose No Commitment (N.C.) despite being able to recognize source inconsistencies when explicitly prompted. However, I have two concerns regarding the interpretation of these results. First, I could not find the complete original actor prompt (without mediation) in either the main paper or the appendix. Given the consistently low N.C. rates in Tables 3 and 4, it is important to establish whether the agent's original instructions or action specification might explicitly or implicitly encourage commitment and task completion rather than verification, clarification, or deferral. Without this information, it is difficult to determine how much of the observed behavior reflects the models' natural tendencies versus the underlying agent scaffold. Second, although Tables 3 and 4 reveal systematic source preferences, particularly the strong dominance of user-provided information over external observations and internal memory, the paper provides limited analysis of why these preferences emerge. Section 7.2 suggests that the dominance of user information may be related to models' instruction-following tendencies and leaves this for future mechanistic studies, but this remains a hypothesis without direct experimental evidence. A quick analysis of the models' action rationales, where CoT is available, or controlled experiments examining the effects of source roles and instructions could help explain these patterns. I believe a deeper investigation of these behaviors would strengthen the paper's motivation by clarifying why agents fail to initiate conflict resolution despite their demonstrated detection capabilities. It could also provide more meaningful insights into the underlying causes of the recognition–behavior gap and how the proposed mediation mechanism addresses them.

  5. Limitations of the proposed approach and problem formulation. The experimental setting makes relatively strong assumptions about grounded conflict resolution. In particular, the success metric requires agents to explicitly obtain the correct value through verification or user clarification before completing the task. Consequently, even when a baseline agent selects the correct value and successfully completes the task, its action is counted as unsuccessful if no explicit verification was performed. While this criterion is consistent with the paper's objective of measuring grounded resolution, it does not distinguish incorrect decisions from correct but unverified decisions. Therefore, the substantial improvements reported for the proposed framework may partly reflect an increase in explicit verification rather than an equivalent improvement in actual task correctness. Additionally, the evaluation assumes that the correct value is always recoverable through verification or user clarification, which may not hold in realistic environments where verification can be unavailable, inconclusive, or contradictory. Although these assumptions are reasonable within the paper's controlled setting, they limit the generalizability of the near-perfect results. If the authors consider these scenarios within the scope of their approach, reporting final task correctness separately and evaluating imperfect verification would strengthen the empirical findings. Otherwise, I would encourage them to explicitly acknowledge these limitations in Section 7.2 and clarify the boundaries of the proposed framework's applicability.

Questions for authors
  1. How does the proposed framework compare against existing conflict-handling or clarification-seeking approaches? Would a simpler conflict-aware prompting strategy, or an LLM-based self-checking mechanism without the proposed structured representation, achieve comparable results?

  2. Do the recognition–behavior gap and intervention improvements persist in realistic, multi-step agent trajectories where inconsistencies arise naturally?

  3. Could the authors provide the complete original actor prompt and action specifications used in Section 4? Could these instructions influence the consistently low No Commitment (N.C.) rates?

  4. Is there experimental evidence explaining agents' strong preference for user-provided information, particularly the suggested instruction-following hypothesis? Could action rationales or controlled experiments help explain this behavior?

7 comments

Q1: Comparison with Existing Methods and Simple Prompting

We thank the reviewer for highlighting these important baselines. Ask-when-Needed [1] and CRITIC [2] make valuable contributions to clarification-seeking and tool-interactive verification. These works share SOIN's broader goal of improving agent reliability while investigating distinct research questions and intervention settings, as summarized below.

DimensionAsk-when-Needed [1]CRITIC [2]SOIN (Ours)
Research ObjectiveClarification under unclear user instructionsTool-interactive self-correctionSource-dependent arbitration and grounded resolution
Problem SettingIncomplete, ambiguous, or incorrect tool-use instructionsErrors in model-generated outputsUnresolved conflicts across user, external, and internal sources
Experimental DesignNoisyToolBench with imperfect user instructionsQuestion answering, mathematical program synthesis, and toxicity reductionMatched source configurations and carrier–value exchange
InterventionProactive clarification before API executionIterative tool-based verification and revisionExplicit SOIN diagnosis and pre-action mediation
EvaluationClarification behavior, tool-use accuracy, and efficiencyOutput correctness and correction effectivenessConflict detection, source arbitration, and grounded task success
Empirical FindingsImproved tool use under unclear instructionsImproved outputs through external feedbackSystematic source preferences and improved grounded resolution

Our distinct contribution is the controlled investigation of unresolved source inconsistency across live agent channels. Through matched carrier–value exchange, we isolate source identity and reveal systematic source-dependent commitment (Section 4). We further investigate how explicit conflict information supports downstream mediation and grounded task resolution (Section 5), extending the analysis to DOM–visual conflicts (Section 6). These findings complement the clarification and self-correction capabilities investigated in prior work.

Regarding the suggested simple prompting baseline, we conduct an additional experiment using a single pre-action instruction to check whether user, webpage, and memory information disagree, and to verify or ask before committing. This Simple Prompt baseline uses no separate diagnostic call or structured SOIN state. All other experimental settings remain unchanged.

ModelSRcSR_c (Simple Prompt)SRnSR_n (Simple Prompt)SRcSR_c (Full SOIN)SRnSR_n (Full SOIN)
GPT-5.6 Sol65.67±3.1100.00±0.0100.00±0.098.91±0.3
GPT-5.6 Luna48.00±4.0100.00±0.0100.00±0.098.82±0.1
Gemini 3.5 Flash Lite51.33±1.5100.00±0.099.13±0.499.62±0.3

The results demonstrate substantial benefits from explicit SOIN checking and mediation. Full SOIN improves grounded conflict-resolution success by 34.33–52.00 percentage points compared with Simple Prompt, reaching 99.13–100.00%. Both approaches maintain high no-conflict success, with Full SOIN showing a small reduction in SRnSR_n. These findings demonstrate that explicit conflict diagnosis and structured mediation enable more reliable grounded resolution than a generic pre-action instruction alone.

We appreciate the reviewer for emphasizing these important comparisons. Ask-when-Needed advances proactive clarification in tool-using agents, while CRITIC establishes the effectiveness of tool-interactive self-correction. Both provide important and complementary insights into agent reliability. We will incorporate the additional baseline results and expand our discussion of their contributions and connections to SOIN in the revised manuscript.

[1] Wang et al. Learning to Ask: When LLM Agents Meet Unclear Instruction. EMNLP 2025.

[2] Gou et al. CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing. ICLR 2024.

Q2: Generalizability to Multi-Step and Real-Web Agent Trajectories

We thank the reviewer for raising this important concern. To investigate SOIN's applicability beyond synthetic, single-decision settings, we extend our evaluation to multi-step web-agent trajectories and real-world websites, examining its effectiveness under dynamic execution conditions.

(1) Multi-Step SOIN Evaluation. We conduct experiments on 3-, 5-, and 10-step trajectories adapted from WebArena. For U–E and U–I conflicts, we modify user-provided information; for E–I conflicts, we introduce disagreements between information encountered on earlier and later webpages. SOIN performs consistency checking and mediation before each action, enabling agents to address conflicts as they emerge throughout execution.

ModelStepsTPstateTP_{state}SRcSR_c
GPT-5.6 Sol397.33±1.296.00±2.0
596.67±1.294.00±0.0
1095.33±1.290.67±1.2
GPT-5.6 Luna397.33±1.296.00±0.0
590.67±1.285.33±1.2
1084.00±2.076.67±1.2
Gemini 3.5 Flash Lite396.00±2.090.00±2.0
590.67±1.289.33±2.3
1088.67±2.384.67±2.3

The results demonstrate that SOIN checking and mediation remain effective across multi-step trajectories. Although performance gradually decreases with trajectory length, SOIN's per-action design enables continuous conflict assessment and mediation, supporting its applicability to longer agent workflows.

(2) Real-World Web Evaluation. We further evaluate SOIN on 50 real-world websites, using tasks adapted from our original benchmark. These tasks involve dynamic, multi-step interactions without predefined trajectory lengths. We introduce U–E and U–I conflicts through user-source modifications. For E–I conflicts, we introduce conflicting information into the internal state at an earlier step, which subsequently disagrees with external observations on later webpages.

ModelConflict PairTPstateTP_{state}SRcSR_c
GPT-5.6 SolU–E99.33±1.299.33±1.2
U–I99.33±1.299.33±1.2
E–I98.67±1.298.00±2.0
GPT-5.6 LunaU–E99.33±1.299.33±1.2
U–I99.33±1.298.00±0.0
E–I98.67±1.298.00±2.0
Gemini 3.5 Flash LiteU–E99.33±1.299.33±1.2
U–I99.33±1.298.00±0.0
E–I98.67±1.298.00±2.0

The results demonstrate that SOIN maintains high detection and grounded resolution performance in dynamic real-web environments, consistent with our original controlled findings. These experiments extend our evaluation to actual webpage interactions and evolving agent states, with controlled conflict injection ensuring reproducibility. The current evaluation covers 50 websites, and we will further expand its coverage in the final manuscript.

Together, these experiments provide additional evidence of SOIN's effectiveness across multi-step execution and realistic web environments. We thank the reviewer for motivating these extensions. We will incorporate the experimental setups, results, and corresponding discussion into the revised manuscript.

Q3: SOIN Detection Beyond Literal String Matching

We thank the reviewer for highlighting the challenges of detecting semantic source disagreements. Our main experiments deliberately use explicit field-value conflicts to isolate source identity and enable reproducible evaluation of conflict detection, natural source arbitration, and downstream mediation. These structured conflicts can be identified through deterministic field matching, providing a controlled setting for examining how agents respond to readily detectable inconsistencies.

To investigate more complex disagreements, we extend our evaluation to Semantic SOIN, where source compatibility requires semantic interpretation. We consider four types of semantic variation: semantic equivalence, contextual dependency, negation/modality, and numerical/unit conversion [3,4]. We evaluate three models across these four types and three pairwise source configurations (U–E, U–I, and E–I), reporting averaged results. We focus on cases with well-defined source compatibility, as temporal changes and incomplete statements can require additional context to establish whether a genuine conflict exists. The new Appendix F provides the definition of Semantic SOIN and representative examples.

ModelTPstateTP_{state}TNstateTN_{state}SRcSR_cSRnSR_n
GPT-5.6 Sol98.67±1.298.00±2.096.00±0.096.67±2.3
GPT-5.6 Luna84.67±1.276.67±2.383.33±2.372.67±1.2
Gemini 3.5 Flash Lite92.67±1.292.00±2.089.33±1.290.00±2.0

The results demonstrate that SOIN detection and mediation extend beyond literal field matching. GPT-5.6 Sol achieves 98.67% TP and 98.00% TN, alongside 96.00% grounded conflict-resolution success. Gemini 3.5 Flash Lite also maintains strong performance, while the lower scores of GPT-5.6 Luna highlight the additional challenges of semantic interpretation. These findings extend our evaluation to semantically expressed source conflicts and demonstrate how detection and mediation effectiveness depend on the underlying model's reasoning capabilities.

We thank the reviewer for motivating this important extension. We will incorporate these findings and their implications into the revised manuscript.

[3] Zhaochen Su et al. "CONFLICTBANK: A Benchmark for Evaluating Knowledge Conflicts in Large Language Models." NeurIPS 2024.

[4] Rongwu Xu et al. "Knowledge Conflicts for LLMs: A Survey." EMNLP 2024.

Q4: Original Actor Prompt and Scaffold Bias

We thank the reviewer for raising this important concern. The original actor prompt and action specifications are important for interpreting the observed recognition–behavior gap. We clarify three aspects of our experimental design.

(1) Source Preference Bias. We carefully reviewed the original browser-use prompt and removed instructions that could encourage prioritizing user-provided information. The resulting actor prompt does not prescribe an authority hierarchy among user (U), external (E), and internal (I) sources or favor any particular source when conflicts arise.

(2) Task-Completion Orientation. We intentionally preserve the original task-oriented instructions of browser-use, including its standard action and task-completion requirements, without adding explicit conflict-handling guidance. This design allows us to examine how agents handle source inconsistencies under a standard task-execution scaffold. Accordingly, our findings characterize unmediated agent behavior within this task-oriented environment.

(3) Prompt Transparency. To improve reproducibility, we now provide the complete original actor prompt in the new Appendix G of our updated manuscript, including its task-execution instructions and browser action rules. This enables direct examination of the scaffold used in our experiments.

We thank the reviewer again for emphasizing this important aspect of experimental transparency. We will further clarify these design choices and their implications for interpreting the recognition–behavior gap in the final manuscript.

Reviewer_Lv4yOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how LLM-based web agents handle conflicting information from user inputs, external observations, and internal states. The authors find that agents can identify such conflicts when explicitly prompted, but rarely attempt to resolve them during normal execution. They propose a training-free, decision-time mechanism that detects conflicts and guides agents to seek additional evidence before acting, leading to improvements in task success under controlled settings.

Critical strengths
  1. The paper addresses an important issue in agent reliability. The distinction between recognizing source conflicts and taking appropriate actions is meaningful and relevant to practical agent systems.

  2. Interesting empirical findings: The results reveal a clear recognition–behavior gap and consistent preferences for user-provided information. These findings provide useful insights into how agents make decisions under conflicting evidence.

Critical weaknesses
  1. Limited methodological novelty. The proposed framework mainly combines explicit conflict detection with structured prompting and predefined handling policies. While the ablation studies show that these components work well together, the paper does not establish whether this framework offers meaningful advantages over simpler verification prompts or existing self-reflection approaches. Stronger baselines are needed to demonstrate the value of the proposed design.

  2. The evaluation assumes that the correct value can always be recovered through verification or user clarification. This substantially simplifies the problem, as real-world source conflicts may involve unreliable evidence, unavailable information, or persistent disagreement. As a result, the near-perfect success rates may overstate the effectiveness of the proposed approach in practical settings.

Also, the experiments rely on synthetic conflicts introduced at isolated decision points rather than naturally occurring inconsistencies in complete agent trajectories. Real-world conflicts may involve ambiguous claims, multiple dependent decisions, and evolving information across steps. The paper also lacks an analysis of the additional inference cost and latency introduced by repeated conflict checking. These limitations make it difficult to assess the practical applicability of the approach.

Questions for authors
  1. The evaluation assumes that the correct value can be recovered through verification or user clarification. How would the proposed approach handle cases where neither source provides reliable evidence, or where the conflict remains unresolved after verification? How much of the reported improvement depends on this assumption?

  2. Comparison with simpler baselines: How does Full SOIN compare with a simple pre-action prompt that instructs the agent to check for conflicting information and verify it before proceeding? This comparison would help clarify whether the structured SOIN state provides benefits beyond explicit prompting.

  3. What are the additional token costs and latency introduced by the diagnostic calls? How do these costs compare with the improvements in task success, particularly when conflicts are rare?

4 comments

Q1: SOIN Mediation under Unavailable or Inconclusive Verification

We thank the reviewer for highlighting this important practical consideration. Our original grounded task success (SRcSR_c) results are measured under recoverable ground truth, where verification or user clarification provides the evidence needed for task completion. SOIN identifies unresolved source conflicts and guides subsequent mediation, while evidence availability depends on the underlying task and environment.

To further address cases where evidence acquisition fails, we strengthen the SOIN mediation policy. When verification or user clarification leaves a conflict unresolved, the agent seeks an alternative information channel if available. If sufficient evidence remains unavailable, it invokes defer, halts the pending action, and reports the reason, avoiding unsupported commitments.

We evaluate this extension under two settings, keeping all other experimental configurations unchanged. In Verify Failure, verification returns no confirmed record, requiring the agent to obtain the correct value through user clarification, measured by grounded task success (SRcSR_c). In Both Failure, neither verification nor user clarification provides usable evidence, and success requires deferral without committing to a conflicting value, measured by Safe Deferral Rate (SDRSDR).

ModelSRcSR_c (Original)SRcSR_c (Verify Failure)SDRSDR (Both Failure)
GPT-5.6 Sol100.00±0.099.33±1.298.67±1.2
GPT-5.6 Luna100.00±0.096.67±1.290.67±1.2
Gemini 3.5 Flash Lite99.33±1.294.67±1.296.00±2.0

The results clarify the role of the recoverability assumption and demonstrate SOIN's effectiveness under failed evidence acquisition. When verification fails, all three models maintain high grounded task success through user clarification. When both channels fail, agents achieve high safe deferral rates (90.67–98.67%). Thus, the original SRcSR_c improvements characterize recoverable conflicts, while the additional experiments demonstrate effective handling when grounded task completion is unavailable. Performance remains model-dependent, highlighting the importance of the underlying agent's capabilities.

We thank the reviewer for motivating this important extension. We will incorporate these experiments, clarify the scope of our original success results, and expand the discussion of unresolvable conflicts in the revised manuscript.

Q2: Full SOIN State vs. Simple Pre-Action Prompting

We thank the reviewer for this valuable suggestion. To assess the benefits of explicit SOIN checking, we introduce a Simple Prompt baseline that instructs the agent to check source consistency and verify or ask before committing. This baseline uses a single pre-action instruction, with no separate diagnostic call or structured SOIN state. All other experimental settings remain unchanged.

ModelSRcSR_c (Simple Prompt)SRnSR_n (Simple Prompt)SRcSR_c (Full SOIN)SRnSR_n (Full SOIN)
GPT-5.6 Sol65.67±3.1100.00±0.0100.00±0.098.91±0.3
GPT-5.6 Luna48.00±4.0100.00±0.0100.00±0.098.82±0.1
Gemini 3.5 Flash Lite51.33±1.5100.00±0.099.13±0.499.62±0.3

The results demonstrate substantial benefits from explicit SOIN checking and mediation. Simple Prompt achieves 48.00–65.67% grounded conflict-resolution success, while Full SOIN reaches 99.13–100.00%, improving SRcSR_c by 34.33–52.00 percentage points. Both approaches maintain high no-conflict success, with Full SOIN showing a small reduction in SRnSR_n. These findings demonstrate that explicit conflict diagnosis and structured mediation enable substantially more reliable grounded resolution compared with a generic pre-action instruction alone.

We thank the reviewer again for suggesting this important comparison. We will incorporate the baseline, experimental results, and corresponding discussion into the revised manuscript.

Q3: Token and Latency Overhead of SOIN Checking

We thank the reviewer for highlighting this important practical consideration. We now measure end-to-end token consumption and latency across 5- and 10-step trajectories, comparing Vanilla, Simple Prompt (a generic consistency-checking instruction without diagnostic calls), and SOIN under conflict and no-conflict conditions. All multipliers are calculated relative to Vanilla at the same trajectory length.

StepsMethodAvg. Token (k)Avg. Latency (s)
5Vanilla53.4 (×1.00)27.9 (×1.00)
Simple prompt (no conflicts)70.3 (×1.32)35.3 (×1.27)
Simple prompt (conflicts)77.7 (×1.46)36.5 (×1.31)
SOIN (no conflicts)62.8 (×1.18)27.6 (×0.99)
SOIN (conflicts)69.7 (×1.31)44.4 (×1.59)
10Vanilla112.9 (×1.00)51.2 (×1.00)
Simple prompt (no conflicts)118.4 (×1.05)51.5 (×1.01)
Simple prompt (conflicts)133.6 (×1.18)61.6 (×1.20)
SOIN (no conflicts)124.7 (×1.10)53.9 (×1.05)
SOIN (conflicts)138.6 (×1.23)76.2 (×1.49)

The results reveal a clear cost–performance trade-off. Without conflicts, SOIN incurs modest token overhead (1.10–1.18×) while maintaining latency close to Vanilla (0.99–1.05×). This suggests limited latency overhead in settings where conflicts are infrequent. When conflicts occur, SOIN's token consumption remains comparable to Simple Prompt, while latency increases to 1.49–1.59× that of Vanilla, reflecting the combined costs of diagnostic checking and conflict handling. Importantly, our separate Simple Prompt comparison shows that Full SOIN improves grounded conflict-resolution success by 34.33–52.00 percentage points, demonstrating substantial benefits alongside these additional costs.

We thank the reviewer for emphasizing this practical consideration. We will incorporate the overhead measurements and corresponding cost–performance discussion into the revised manuscript.

Reviewer_Lv4yOpenReview ↗

Thank you for the detailed and reasonable response

I have read your reply carefully. Overall, these experiments and clarifications have largely reduced my previous concerns. I raised my score to 3 points. Thank you for your efforts and reply!

We thank the reviewer for recognizing the importance of our work and providing us with solid suggestions. The constructive feedback massively helped us improve SOIN's practical applicability part in this work.

Thank you again for your support!

Reviewer_qhT3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Source Inconsistency (SOIN) in LLM web agents, finding that agents often recognize conflicting information but fail to resolve it, favoring user inputs. The authors propose a training-free, model-agnostic framework to detect conflicts and guide verification, clarification, or deferral.

Critical strengths
  1. Systematic experimental design and analysis. The authors test multiple models and sources, carefully separating the effects of source identity and content. The results show how agents prioritize different sources.

  2. Promising mitigation results. The proposed method clearly improves agent performance. The experiments show that detailed conflict information works better than simple warnings and that the method applies to different types of conflicts.

Critical weaknesses
  1. Lack of comparison with existing methods. The paper only compares with the base agent and its variants or ablations (Section 5, Table 5; Appendix E.2), without considering related methods (eg. [1-3] ). These comparisons are needed to better assess SOIN's contribution.

  2. Reliance on synthetic conflicts. The conflicts and decision contexts are simulated rather than collected from real agent executions (Section 3.1; Appendix B). The paper should examine how often such conflicts occur in practice and evaluate real conflict cases.

  3. Limited evaluation under realistic conditions. The evaluation assumes that conflicts always have ground truth or can be resolved through verification or user clarification (Section 5; Appendix B). However, this assumption may not always be true in practice. The paper should also evaluate cases where conflicts cannot be resolved.

[1] Structured Uncertainty guided Clarification for LLM Agents. Findings of ACL 2026.

[2] Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents. Findings of EMNLP 2026.

[3] CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing. ICLR 2024.

Questions for authors
  1. Comparison with existing methods: How does the proposed framework compare with existing verification and consistency-checking methods (e.g., [1–3])?

  2. Naturally occurring conflicts: Have the authors observed naturally occurring source conflicts in real web-agent tasks? If so, could they share examples or statistics? Evaluation on such conflicts would further strengthen the paper.

  3. Handling unresolved conflicts: How does the framework handle cases where verification or user clarification cannot resolve a conflict (i.e., no ground truth)?

  4. Computational overhead: What are the computational and latency costs of the diagnostic step compared with simpler prompting baselines, particularly for long-horizon tasks?

[1] Structured Uncertainty guided Clarification for LLM Agents. Findings of ACL 2026.

[2] Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents. Findings of EMNLP 2026.

[3] CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing. ICLR 2024.

8 comments

Q1: Comparison with Existing Clarification and Verification Methods

We thank the reviewer for highlighting these important studies. We agree that these works provide valuable approaches to agent clarification, uncertainty handling, and tool-based verification. SOIN differs from them regarding research objectives, problem formulations, and intervention mechanisms, as summarized below.

DimensionStructured Uncertainty [1]Ask or Assume? [2]CRITIC [3]SOIN (Ours)
Research ObjectiveStructured clarification under uncertain tool parametersClarification-seeking in coding agentsTool-interactive self-correctionSource-dependent arbitration and grounded resolution
Problem SettingAmbiguous or incomplete tool-call specificationsUnderspecified software tasksErrors in model-generated outputsUnresolved conflicts across user, external, and internal sources
Experimental DesignStructured tool-parameter uncertainty and ClarifyBenchUnderspecified SWE-bench Verified tasksTool-feedback evaluation across generation tasksMatched source configurations and carrier–value exchange
InterventionEVPI-guided question selection (SAGE-Agent)Uncertainty-aware multi-agent clarificationExternal tool validation and iterative revisionExplicit SOIN checking and pre-action mediation
EvaluationClarification coverage and question efficiencyTask resolution and clarification behaviorOutput correctness and correction effectivenessConflict detection, source arbitration, and grounded task success
Empirical FindingsImproved clarification efficiency and coverageImproved task resolution through clarificationImproved outputs through external feedbackSystematic source preferences and improved grounded resolution

Our distinct contribution is the controlled investigation of unresolved source inconsistency across the live channels of an executing agent. Through matched carrier–value exchange, we isolate source identity from value preference and reveal systematic source-dependent commitment (Section 4). We further study how explicit conflict information affects downstream mediation and grounded task completion (Section 5), extending the analysis to DOM–visual conflicts (Section 6). These findings complement the clarification and verification capabilities investigated in prior work.

We appreciate the reviewer for highlighting these three important studies. These works offer valuable and complementary insights into improving agent reliability. We will expand the discussion in the revised manuscript to more thoroughly acknowledge their contributions and clarify their connections to SOIN.

[1] Structured Uncertainty guided Clarification for LLM Agents. Findings of ACL 2026.

[2] Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents. Findings of EMNLP 2026.

[3] CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing. ICLR 2024.

Q2: Naturally Occurring Source Conflicts and Real-Web Evaluation

We thank the reviewer for raising this important practical concern. Source inconsistencies can naturally emerge during web-agent execution, and our controlled benchmark provides a systematic way to investigate their effects on agent behavior.

For U–E conflicts, users may provide outdated information, incorrect values, or typographical errors that disagree with current webpages. For U–I conflicts, user instructions may contradict previously stored preferences, records, or task context. For E–I conflicts, dynamic webpage updates or navigation may introduce discrepancies between current observations and earlier agent states. Additionally, DOM–visual inconsistencies can arise from rendering differences or asynchronous updates, as examined in Section 6.

To evaluate SOIN in realistic environments, we extend our experiments to 50 real-world websites, covering different tasks from our original benchmark. We construct U–E and U–I conflicts through user-source modifications. For E–I conflicts, we introduce information into the internal state at an earlier step that subsequently disagrees with observations on later webpages.

ModelConflict PairTPstateTP_{state}SRcSR_c
GPT-5.6 SolU–E99.33±1.299.33±1.2
U–I99.33±1.299.33±1.2
E–I98.67±1.298.00±2.0
GPT-5.6 LunaU–E99.33±1.299.33±1.2
U–I99.33±1.298.00±0.0
E–I98.67±1.298.00±2.0
Gemini 3.5 Flash LiteU–E99.33±1.299.33±1.2
U–I99.33±1.298.00±0.0
E–I98.67±1.298.00±2.0

The results demonstrate that SOIN maintains high detection and grounded resolution performance on real-world websites, consistent with our original controlled findings. These experiments provide initial evidence of SOIN's effectiveness in realistic web environments beyond structured record stubs. Conflicts are introduced to ensure reproducibility, while agent execution takes place on actual websites.

We thank the reviewer for emphasizing this important direction. We will expand the current real-web evaluation in the final manuscript and further investigate naturally occurring conflict examples and their prevalence in future work.

Q3: Handling Unresolved Conflicts without Ground Truth

We thank the reviewer for raising this important question. Our original evaluation assumes recoverable ground truth, enabling controlled measurement of grounded task success. To extend SOIN to unresolvable conflicts, we strengthen its mediation policy to explicitly handle failed evidence acquisition. When verification or user clarification cannot resolve a conflict, the agent seeks an alternative information channel if available. If the conflict remains unresolved, it invokes defer, halts the pending action, and reports the reason, avoiding unsupported commitments.

We evaluate this extension under two settings, keeping all other experimental configurations unchanged. In Verify Failure, verification returns no confirmed record, requiring the agent to obtain the correct value through user clarification, measured by grounded task success (SRcSR_c). In Both Failure, neither verification nor user clarification provides usable evidence, and success requires deferral without committing to any conflicting value, measured by Safe Deferral Rate (SDRSDR).

ModelSRcSR_c (Original)SRcSR_c (Verify Failure)SDRSDR (Both Failure)
GPT-5.6 Sol100.00±0.099.33±1.298.67±1.2
GPT-5.6 Luna100.00±0.096.67±1.290.67±1.2
Gemini 3.5 Flash Lite99.33±1.294.67±1.296.00±2.0

The results demonstrate that SOIN mediation remains effective under failed evidence acquisition. Under Verify Failure, all three models maintain high grounded task success through user clarification. When both channels fail, agents achieve high safe deferral rates (90.67–98.67%), demonstrating their ability to handle unresolved conflicts without unsupported commitments. Performance remains model-dependent, highlighting the importance of the underlying agent's capabilities.

These findings extend SOIN's demonstrated applicability from grounded resolution to appropriate handling of unresolvable conflicts. We thank the reviewer for motivating this important extension and will incorporate the additional experiments and corresponding discussion into the revised manuscript.

Q4: Computational Overhead in Multi-Step Tasks

We thank the reviewer for highlighting this important practical consideration. To assess SOIN's effectiveness and computational overhead in longer tasks, we extend our evaluation to 3-, 5-, and 10-step trajectories adapted from WebArena. U–E and U–I conflicts are introduced through user-source modifications, while E–I conflicts involve information encountered on earlier and later webpages. SOIN performs consistency checking and mediation before each action.

ModelStepsTPstateTP_{state}SRcSR_c
GPT-5.6 Sol397.33±1.296.00±2.0
596.67±1.294.00±0.0
1095.33±1.290.67±1.2
GPT-5.6 Luna397.33±1.296.00±0.0
590.67±1.285.33±1.2
1084.00±2.076.67±1.2
Gemini 3.5 Flash Lite396.00±2.090.00±2.0
590.67±1.289.33±2.3
1088.67±2.384.67±2.3

SOIN maintains high detection and grounded resolution performance across multi-step trajectories, although both metrics gradually decline as trajectory length increases. Its per-action design enables continuous conflict checking and mediation throughout execution.

We further measure end-to-end token consumption and latency on 5- and 10-step trajectories, comparing Vanilla (no additional checking), Simple Prompt (a generic consistency-checking instruction without diagnostic calls), and SOIN (explicit diagnosis and mediation). We evaluate both conflict and no-conflict conditions, with multipliers calculated relative to Vanilla at the same trajectory length.

StepsMethodAvg. Token (k)Avg. Latency (s)
5Vanilla53.4 (×1.00)27.9 (×1.00)
Simple prompt (no conflicts)70.3 (×1.32)35.3 (×1.27)
Simple prompt (conflicts)77.7 (×1.46)36.5 (×1.31)
SOIN (no conflicts)62.8 (×1.18)27.6 (×0.99)
SOIN (conflicts)69.7 (×1.31)44.4 (×1.59)
10Vanilla112.9 (×1.00)51.2 (×1.00)
Simple prompt (no conflicts)118.4 (×1.05)51.5 (×1.01)
Simple prompt (conflicts)133.6 (×1.18)61.6 (×1.20)
SOIN (no conflicts)124.7 (×1.10)53.9 (×1.05)
SOIN (conflicts)138.6 (×1.23)76.2 (×1.49)

The results reveal a clear cost–performance trade-off. Without conflicts, SOIN incurs modest token overhead (1.10–1.18×) while maintaining latency close to Vanilla (0.99–1.05×), indicating limited additional latency when conflicts are infrequent. Under conflicts, token consumption remains comparable to Simple Prompt, while latency increases by approximately 22–24% relative to Simple Prompt (1.49–1.59× relative to Vanilla). This reflects the combined overhead of diagnostic checking and subsequent conflict handling, which accompanies the improvements in grounded resolution demonstrated in our controlled experiments.

We thank the reviewer for emphasizing computational efficiency in long-horizon tasks. We will incorporate these experiments, overhead measurements, and corresponding cost–performance discussion into the revised manuscript.

Reviewer_CFXPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies disagreements between a web agent's user input, external observation, and internal state when the available evidence does not identify a correct source. It constructs 1,500 conflict instances and matched consistent controls from 100 scene–field combinations. Across five models, explicit prompting usually identifies the conflict, while ordinary execution rarely seeks clarification or verification. A separate detector supplies structured conflict information to an action policy. This substantially improves grounded completion in a controlled setting with a recoverable answer. Additional experiments consider disagreement between DOM and screenshot observations.

Critical strengths
  1. The controlled source comparisons are useful. Exchanging values across carriers, including matched consistent cases, and distinguishing conflict detection from subsequent action make the observed source preferences easier to interpret. The finding that a user-carried value can outweigh two agreeing sources is informative within this setting (Tables 3–4).

  2. The intervention evaluation measures more than a willingness to pause. Successful conflict handling requires obtaining new evidence and applying it to complete the task. Binary, full, and oracle states, together with the component ablations in Appendix E.2, help establish that both the supplied information and the handling policy matter. The gains across several models are substantial on these tasks.

Critical weaknesses
  1. The novelty of the general mediation mechanism is less clear than the value of the controlled study. TANGLE also separates conflict assessment from response generation, and the GUI source-reliance paper also tests a consistency check followed by corrective action. Sections 1 and 7.1 distinguish the settings, but provide limited discussion of these mechanism-level overlaps. This limits the method's positioning. The different source and authority assumptions prevent me from treating the earlier methods as equivalent or concluding that the controlled findings are redundant.

Sources: TANGLE, Section 4.5; Do GUI Agents Believe Their Eyes?, “Comparing mitigations on identical probes.” The latter has a different grounding assumption, which limits the comparison between the methods.

  1. The practical conclusion is limited to conflicts that can be resolved through an available information channel. Section 5 and Appendix B guarantee a recoverable answer through a targeted question or page check, and the paper explicitly acknowledges this condition (p. 8, lines 387–388). All comparison conditions have access to the same actions. The results therefore support the claimed improvement in this controlled setting, while leaving deployment benefit under unavailable or inconclusive verification, and the cost of additional checks, unestablished. This limits the scope of the practical conclusion; the controlled comparison remains valid.
Questions for authors

none

5 comments

Q1: Mechanism-Level Novelty and Distinction from Prior Work

We thank the reviewer for highlighting these important mechanism-level connections. We agree that TANGLE, GUI Source-Reliance, and SOIN share the general principle of assessing conflicts before corrective action. SOIN develops this principle for unresolved cross-channel source inconsistencies, with a distinct diagnostic formulation, mediation framework, and controlled empirical investigation. We summarize these connections and distinctions below.

DimensionTANGLEGUI Source-RelianceSOIN (Ours)
Research ObjectiveCalibrated handling of irreducible memory conflictsVisual state reliance and action groundingSource-dependent arbitration and grounded resolution
Conflict SettingConflicting personal-memory records (I)DOM–visual disagreement (E)Cross-channel disagreement (U, E, I), extended to DOM–visual conflicts
Authority AssumptionNo unique answer within memoryPixels define visible-state ground truthNo predefined source-authority hierarchy
Experimental DesignMemory-conflict taxonomy and oracle/pipeline evaluationPaired channel interventions and mitigation comparisonsMatched source configurations and carrier–value exchange
InterventionConflict-aware action selection followed by response generation (CAAP)Pixel-grounding consistency gate and UI repairIndependent SOIN diagnosis followed by pre-action mediation
EvaluationConflict recognition, calibration, and response qualityState-belief reliance and action errorsConflict detection, source arbitration, and grounded task success
Empirical FindingsRecognition–action gap in memory conflictsStructural reliance despite correct visual perceptionSystematic source preferences and improved grounded resolution

Our distinct contribution is to isolate source identity as a controlled variable through matched carrier–value exchange, revealing systematic source-dependent commitment across live agent channels (Section 4). In our framework, the diagnostic stage explicitly extracts source inconsistencies without prescribing an action, enabling controlled evaluation of how different levels of conflict information affect downstream mediation (Section 5). SOIN also supports evidence acquisition through verification or clarification under unresolved conflicts, without assuming a predefined authoritative source. We further extend this investigation to DOM–visual disagreements (Section 6).

We appreciate the reviewer for identifying these important connections. TANGLE contributes valuable insights into conflict-aware action selection, while GUI Source-Reliance establishes the importance of consistency checking and corrective grounding. Both provide complementary perspectives on agent reliability. We will expand Sections 1 and 7.1 in the revised manuscript to clarify these mechanism-level connections, distinct assumptions, and contributions.

Q2: SOIN Mediation under Unavailable or Inconclusive Verification

We thank the reviewer for highlighting this important practical consideration. SOIN is designed to identify unresolved source conflicts and mediate subsequent actions, but not guarantee the availability of external evidence. Whether verification or user clarification can provide a definitive answer depends on the underlying task and environment, instead of SOIN checking itself. Thus, unsuccessful evidence acquisition does not necessarily indicate a failure of conflict detection or mediation.

Nevertheless, we agree that handling failed or inconclusive verification is an important SOIN practical extension. To extend SOIN to cases where evidence acquisition fails, we strengthen its mediation policy to handle unavailable or inconclusive responses from verify and ask_user. When a conflict remains unresolved, the agent seeks evidence through an alternative information channel if available. If sufficient evidence cannot be obtained, it invokes defer, halts the pending action, and reports the unresolved conflict and the reason for deferral, avoiding unsupported commitments.

We evaluate this extension under two settings, keeping all other experimental configurations unchanged. In Verify Failure, verification returns no confirmed record, requiring the agent to obtain the correct value through user clarification, measured by grounded task success (SRcSR_c). In Both Failure, neither verification nor user clarification provides usable evidence, and success requires deferral without committing to a conflicting value, measured by Safe Deferral Rate (SDRSDR).

ModelSRcSR_c (Original)SRcSR_c (Verify Failure)SDRSDR (Both Failure)
GPT-5.6 Sol100.00±0.099.33±1.298.67±1.2
GPT-5.6 Luna100.00±0.096.67±1.290.67±1.2
Gemini 3.5 Flash Lite99.33±1.294.67±1.296.00±2.0

The results demonstrate that SOIN supports effective mediation under failed evidence acquisition. When verification fails, all three models maintain high grounded task success by obtaining the required information through user clarification. When both channels fail, agents achieve high safe deferral rates, demonstrating their ability to handle unresolved conflicts without unsupported commitments. Performance remains model-dependent, highlighting the importance of the underlying agent's capabilities. These findings extend SOIN's demonstrated applicability from grounded resolution under recoverable evidence to appropriate deferral when evidence is unavailable. We sincerely thank the reviewer for motivating this important extension. We will incorporate these experiments and the corresponding discussion into the revised manuscript.

Q3: Token and Latency Overhead of SOIN Checking

We thank the reviewer for highlighting this important practical consideration. To quantify the computational overhead of SOIN checking, we measure end-to-end token consumption and latency across 5- and 10-step trajectories. We compare Vanilla, Simple Prompt (a generic consistency-checking instruction without a diagnostic call), and SOIN under conflict and no-conflict conditions. All multipliers are calculated relative to Vanilla at the same trajectory length.

StepsMethodAvg. Token (k)Avg. Latency (s)
5Vanilla53.4 (×1.00)27.9 (×1.00)
Simple prompt (no conflicts)70.3 (×1.32)35.3 (×1.27)
Simple prompt (conflicts)77.7 (×1.46)36.5 (×1.31)
SOIN (no conflicts)62.8 (×1.18)27.6 (×0.99)
SOIN (conflicts)69.7 (×1.31)44.4 (×1.59)
10Vanilla112.9 (×1.00)51.2 (×1.00)
Simple prompt (no conflicts)118.4 (×1.05)51.5 (×1.01)
Simple prompt (conflicts)133.6 (×1.18)61.6 (×1.20)
SOIN (no conflicts)124.7 (×1.10)53.9 (×1.05)
SOIN (conflicts)138.6 (×1.23)76.2 (×1.49)

The results reveal a clear cost–performance trade-off. Without conflicts, SOIN incurs modest token overhead (1.10–1.18×) while maintaining latency close to Vanilla (0.99–1.05×). This is particularly relevant when conflicts are rare, as most executions incur only the no-conflict overhead. Under conflicts, latency increases more substantially (1.49–1.59×), reflecting the combined costs of diagnostic checking and subsequent conflict handling. These additional costs accompany the substantial improvements in grounded conflict resolution demonstrated in our experiments.

We thank the reviewer again for emphasizing the importance of computational efficiency. We will incorporate these measurements and the corresponding cost–performance analysis into the revised manuscript.

Reviewer_QPvTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies source inconsistency (SOIN) in web agents. At one decision point the user instruction, the current page observation and the agent's internal state can give different values for the same decision critical field, and the available evidence does not decide which one is right. The authors build a controlled setting on Browser Use with tasks adapted from WebVoyager and WebArena.

Critical strengths

The main text is dense and the notation for P(C), G(C) and the four field state takes effort to follow. Tables 3 and 4 pack many configurations into one page and are hard to parse. Some details that are essential for reading the numbers, especially that every instance is a single frozen decision point and that the recoverable value is injected synthetically, are mainly in Appendix B. I had to read Section 3 and Appendix B together before I understood what is actually measured.

Critical weaknesses
  1. The comparison with the closest prior work is only verbal. HiL-Bench and TANGLE are the two studies that come nearest to this problem, and both appear in one sentence of the introduction and one of related work, but there is no shared task, no shared metric and no empirical comparison. The claimed distinction, that SOIN concerns disagreement across the live user, external and internal channels with no authoritative source, is plausible, but a reader cannot tell from this paper whether it is a new problem or a renamed one. The detection result has the same issue in a different form. Because the conflicts are literal record values written into three carriers, a simple field matching script would reach 100% by construction, and no such non LLM baseline is reported. So the near ceiling TP numbers do not yet show that the models do anything beyond reading two strings. This matters because the whole argument in Section 4 and Section 5 takes the detection ability as the upper bound for the gap. If the detection task is trivial, the gap is less about source reasoning and more about instruction following.

  2. The experimental scale is small and the protocol is narrow in a way that limits what can be concluded. There are five API models and three runs, and the runs are not described well enough to tell whether the variance comes from instances, sampling or decoding. All data are record stubs in 20 adapted scenes rather than real websites, so there is no natural inconsistency, no partial or noisy conflict, and no case where verification returns nothing useful. The paper also does not go beyond three sources, does not study conflicts that appear at different steps of the same trajectory, and does not report an ablation on the number of conflicting sources or on conflict strength. There is no cost or latency accounting for the extra diagnostic call, which is central for anything described as a deployable pre action check. The residual failure of Claude Haiku 4.5 under the full state, 62.51%, is explained as under localization, but this is a single case study and not a systematic error analysis. This matters because the paper's claims are about agent reliability in general, while the evidence covers one decision step in one scaffold.

  3. The evaluation unit is a synthetic single step, and the ground truth is a value chosen at random from the conflicting values and delivered only when the agent asks for it. So SRc measures whether the agent follows an injected conflict protocol rather than whether it recovers a grounded fact inside a real trajectory. The abstract says grounded task success, which is stronger than what is measured. On top of that, there is no matched baseline that tests the mechanism without the extracted state, for example one generic instruction to check whether the sources agree and to verify or ask when they do not. The detection only and policy only ablations are helpful, but they do not answer this. As a result I cannot tell whether the gain comes from making source consistency explicit at decision time, which is the claimed principle, or from a prompt that names the conflicting field and tells the agent to verify it.

Questions for authors
  1. What happens with a single generic instruction such as "before acting, check whether the user request, the page and your memory agree, and if they do not, verify or ask before committing", with no separate diagnostic call and no structured state? Would SRc still be close to the full state condition?

  2. Would a non LLM checker that compares the three carriers field by field already reach the reported TP and TN rates on this data? If so, what does the diagnostic call add beyond parsing?

  3. How would the results change if the conflict were placed inside a real WebArena or WebVoyager trajectory that spans several steps, so that a wrong commitment propagates to later actions?


Update: 11/17

Authors have solved my concerns so I improve my rating from 2 to 3.

5 comments

Q1: Novelty and Distinction from HiL-Bench and TANGLE

We thank the reviewer for highlighting these closely related studies. We agree that HiL-Bench, TANGLE, and SOIN share the broader motivation of improving agent behavior under insufficient or conflicting information. SOIN addresses a distinct research question through its controlled experimental design and empirical investigation of source-dependent behavior. We clarify these differences below.

DimensionHiL-BenchTANGLESOIN (Ours)
Research ObjectiveSelective help-seeking under task uncertaintyRecognition and handling of irreducible memory conflictsSource-dependent arbitration (U, E, I) and grounded resolution
Conflict SettingIncomplete, ambiguous, or contradictory task informationConflicting personal-memory (I) recordsCross-channel disagreement among user, external, and internal sources (U, E, I)
Experimental DesignProgressive information blockersMemory-conflict taxonomy and evaluation scenariosMatched source configurations and carrier–value exchange
InterventionHelp-seeking policies and RLConflict-Aware Action Policy (CAAP)Explicit SOIN checking and pre-action mediation
EvaluationAsk-F1 and task completionConflict recognition, calibration, and response qualityConflict detection, source arbitration, and grounded task success
Empirical FindingsLimitations in selective help-seekingRecognition-to-action failures in memory conflictsSystematic source preferences and improved grounded resolution

Our distinct contribution is to isolate source identity as a controlled variable through matched carrier–value exchange, revealing systematic source-dependent commitment during natural execution (Section 4). Notably, agents frequently follow the user even when external and internal sources agree against it. Beyond identifying a recognition–behavior gap, we demonstrate that explicit SOIN checking enables grounded task completion, requiring agents to acquire and apply missing evidence rather than merely select a plausible response (Section 5). We extend this investigation to DOM–visual disagreement (Section 6).

Regarding empirical comparison, all three works are empirically motivated studies investigating distinct research questions and agent behaviors. Their different experimental settings, intervention mechanisms, and evaluation objectives make direct numerical comparisons difficult to interpret without a shared evaluation protocol. We recognize HiL-Bench and TANGLE as important and closely related contributions that provide complementary insights into agent reliability. We will include this more comprehensive comparison in the revised manuscript's appendix, highlighting their contributions, shared motivations, and distinctions from SOIN.

Q2: SOIN Detection Beyond Literal String Matching

We thank the reviewer for raising this important concern. Our use of literal field-value conflicts is an experimental choice that isolates source inconsistency from semantic ambiguity, enabling reproducible evaluation of conflict detection, natural source arbitration, and downstream mediation. We agree that a deterministic field-matching checker can identify conflicts in these structured cases. This controlled setting allows us to examine a central question: whether agents appropriately act on inconsistencies that are readily detectable.

The diagnostic call contributes by explicitly extracting decision-relevant source inconsistencies and providing a structured state for subsequent mediation. To further examine its capabilities beyond literal parsing, we extend our evaluation to Semantic SOIN, where source compatibility requires semantic interpretation. We consider four types of semantic variation: semantic equivalence, contextual dependency, negation/modality, and numerical/unit conversion [1,2]. We evaluate three models across these four types and three pairwise source configurations (U–E, U–I, and E–I), reporting averaged results. We keep the other setups as same as in our Section 5 of main paper. The new Appendix F provides the definition of Semantic SOIN and representative examples.

TPstateTP_{state}TNstateTN_{state}SRcSR_cSRnSR_n
GPT-5.6 Sol98.67±1.298.00±2.096.00±0.096.67±2.3
GPT-5.6 Luna84.67±1.276.67±2.383.33±2.372.67±1.2
Gemini 3.5 Flash Lite92.67±1.292.00±2.089.33±1.290.00±2.0

The results demonstrate that SOIN detection and mediation extend beyond literal field matching. GPT-5.6 Sol achieves 98.67% TP and 98.00% TN, with 96.00% grounded conflict-resolution success, while Gemini 3.5 Flash Lite also maintains strong performance. The lower scores of GPT-5.6 Luna highlight the additional challenges of semantic interpretation. These findings demonstrate the applicability of SOIN checking to semantically expressed conflicts, with effectiveness depending on the underlying model's capabilities.

We thank the reviewer for motivating this important extension. We will incorporate these findings and their implications into the final manuscript.

[1] Zhaochen Su et al. "CONFLICTBANK: A Benchmark for Evaluating Knowledge Conflicts in Large Language Models." NeurIPS 2024.

[2] Rongwu Xu et al. "Knowledge Conflicts for LLMs: A Survey." EMNLP 2024.

Q3: Real-World Web Evaluation and Multiple Concurrent Conflicts

We thank the reviewer for raising these important concerns. Our original evaluation already comprises 3,000 controlled instances across five API models and three runs, enabling systematic analysis of source-dependent behavior. To further assess SOIN in realistic environments and more complex conflict settings, we conduct two additional evaluations.

(1) Real-World Web Evaluation. We extend SOIN to 50 real-world websites, covering different tasks from our original benchmark. We introduce U–E and U–I conflicts by modifying user-provided information. For E–I conflicts, we introduce conflicting information into the internal state (I) at an earlier step, which subsequently disagrees with external observations (E) on later webpages. These conflicts reflect natural inconsistencies case in real world.

ModelConflict PairTPstateTP_{state}SRcSR_c
GPT-5.6 SolU–E99.33±1.299.33±1.2
U–I99.33±1.299.33±1.2
E–I98.67±1.298.00±2.0
GPT-5.6 LunaU–E99.33±1.299.33±1.2
U–I99.33±1.298.00±0.0
E–I98.67±1.298.00±2.0
Gemini 3.5 Flash LiteU–E99.33±1.299.33±1.2
U–I99.33±1.298.00±0.0
E–I98.67±1.298.00±2.0

The results show that SOIN maintains high conflict detection and grounded resolution performance on real-world websites, consistent with our original controlled findings. These results provide initial evidence of SOIN's effectiveness beyond structured record stubs. We evaluated 50 websites due to the limited period of rebuttal. We will further expand this evaluation size to more websites in the final manuscript.

(2) Multiple Concurrent Conflicts. We further evaluate 100 scenarios containing 2–4 simultaneous source inconsistencies, examining how SOIN performs as the number of conflicts increases. All other experimental settings remain unchanged as in Section 5 of main paper.

Model# ConflictsTPstateTP_{state}SRcSR_c
GPT-5.6 Sol298.33±0.697.67±1.5
397.00±1.796.67±0.6
495.00±1.095.00±1.0
GPT-5.6 Luna298.00±1.796.67±1.2
397.33±1.295.67±0.6
495.33±1.294.67±1.2
Gemini 3.5 Flash Lite295.33±1.594.33±0.6
395.67±2.194.67±1.2
495.00±0.093.67±1.5

SOIN remains effective as conflict complexity increases. Detection and grounded resolution performance generally decline with more concurrent inconsistencies, yet all three models maintain high success rates even with four simultaneous conflicts. These findings provide additional evidence of SOIN's applicability to more complex conflict configurations.

We thank the reviewer for motivating these valuable extensions. We will incorporate the additional experiments, clarify the three-run protocol and variability reporting, and refine our scopes in the revised manuscript.

Q4: Systematic Failure Analysis of Claude Haiku 4.5

We thank the reviewer for emphasizing the importance of systematic failure analysis. To better understand Claude Haiku 4.5's residual failures, we examine all unsuccessful cases under Full SOIN State in Section 5 and categorize them into detection, decision, and execution failures.

Our SOIN state consists of (G^,P^,E^,r^)(\hat G,\hat P,\hat{\mathcal E},\hat r). The existing detection metrics evaluate conflict identification (G^\hat G and P^\hat P), while the quality of evidence localization (E^\hat{\mathcal E}) and decision relevance (r^\hat r) also influences downstream execution. Analyzing these additional components is therefore important for understanding the gap between conflict detection and grounded resolution.

ModelDetection FailureDecision FailureExecution Failure
Claude Haiku 4.517.79%0.00%82.21%

The results show that 82.21% of Haiku's failures occur during execution, with 17.79% attributable to detection and none to handling decisions. This indicates that successful conflict recognition alone does not ensure grounded resolution: agents must also locate relevant evidence and execute the appropriate actions. These findings help explain the gap between Full and Oracle SOIN State and highlight evidence localization and downstream execution as important factors in model-dependent reliability.

We thank the reviewer for motivating this systematic analysis. We will incorporate the failure categorization, representative examples, and corresponding discussion into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.