- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies Source Inconsistency (SOIN), a reliability problem that arises when web-based LLM agents encounter conflicting information from multiple sources, including user prompts, internal memory, and the external environment, without a clear basis for determining which source is correct. The authors find that although agents can readily recognize these inconsistencies, they often fail to resolve them during normal execution and instead tend to commit to one of the conflicting sources. To address this gap, the authors propose a mediation framework that builds on agents' strong inconsistency recognition capabilities. The framework first prompts agents to perform a diagnostic call to identify whether SOIN exists and which source pairs are in conflict, then uses this diagnosis to generate an intervention that guides the agent toward resolving the inconsistency. Experimental results on two adapted datasets demonstrate substantial improvements in grounded conflict resolution while preserving performance on non-conflicting tasks.
Critical strengths
-
Important reliability problem supported by systematic empirical analysis. The paper identifies a meaningful recognition–behavior gap in LLM agents: models can reliably detect conflicting information when explicitly prompted, yet rarely initiate verification or clarification during normal execution. The controlled experimental design, including source-value swapping and matched no-conflict cases across five models, provides convincing evidence of systematic source-dependent behavior. Overall, it highlights a practical agent reliability issue that cannot be fully captured by conflict-detection accuracy alone.
-
Substantial improvement through a simple, training-free intervention. The proposed decision-time mediation framework achieves considerable improvements in grounded conflict resolution while largely preserving performance on non-conflicting tasks. The comparisons between binary, full, and oracle inconsistency states, alongside component ablations, help demonstrate the importance of explicitly identifying and localizing conflicts before acting. Although the evaluation remains controlled, these results provide promising evidence for a practical mechanism to improve agent reliability.
The paper is also very well written and easy to follow and understand.
Critical weaknesses
-
Limited comparison with existing or similar baselines. Although the paper provides extensive comparisons among different variants of SOIN-aware action mediation in Sections 5 and 6, along with some ablation studies in the appendix, it lacks comparisons against potentially relevant baselines. One of the most basic baselines, instead of the two-stage SOIN mediation, could be to add a single-line instruction to the original agent, prompting the model to check whether the user, webpage, and memory disagree, and to verify or ask before acting if necessary. Having such a baseline could clarify whether SOIN is even necessary beyond good prompting. I also found some baselines, including Ask-when-Needed (Wang et al., 2025) and CRITIC (Gou et al., 2024), that appear relevant, although they may need some adaptation. Given that the intervention achieves near-perfect performance in a controlled setting, such comparisons are important for understanding the method's actual contribution, practical advantages, and trade-offs compared with other existing approaches.
-
The generalizability of the recognition–behavior gap and the mediation framework could be limited due to synthetic, single-decision-point evaluations. The paper is motivated by the need to address reliability challenges that do not arise in single-turn generation but instead exist in dynamically changing, complex environments in which agents are expected to act (lines 41–45 in the introduction). However, I found the experimental setting and tested datasets somewhat synthetic and not as complicated as the authors suggest in their motivation. Although the paper evaluates 1,500 conflict instances, these originate from only 100 base scene–field combinations, with different source-value assignments. Additionally, the experiments use frozen decision points, where the agent's internal state is simulated rather than accumulated through actual preceding interactions. While this design allows for controlled comparisons, it remains unclear whether the observed recognition–behavior gap and the intervention's effectiveness would persist in realistic, multi-step agent trajectories. In practice, conflicts may emerge gradually, involve implicit or partially contradictory information, and interact with previously accumulated errors.
-
Simplified inconsistency detection. Somewhat related to my prior concern, the proposed definition of SOIN focuses on incompatible values assigned to the same decision-relevant variable. This makes the experimental inconsistencies relatively explicit, particularly because the sources follow controlled record structures. Real-world source disagreements, however, may involve semantic differences, temporal changes, incomplete statements, or contextual dependencies that are substantially harder to identify. This may be partly evident from the near-perfect detection scores in Tables 1 and 2.
-
Insufficient analysis of natural source-arbitration behavior. Section 4 presents an interesting finding that agents rarely choose No Commitment (N.C.) despite being able to recognize source inconsistencies when explicitly prompted. However, I have two concerns regarding the interpretation of these results. First, I could not find the complete original actor prompt (without mediation) in either the main paper or the appendix. Given the consistently low N.C. rates in Tables 3 and 4, it is important to establish whether the agent's original instructions or action specification might explicitly or implicitly encourage commitment and task completion rather than verification, clarification, or deferral. Without this information, it is difficult to determine how much of the observed behavior reflects the models' natural tendencies versus the underlying agent scaffold. Second, although Tables 3 and 4 reveal systematic source preferences, particularly the strong dominance of user-provided information over external observations and internal memory, the paper provides limited analysis of why these preferences emerge. Section 7.2 suggests that the dominance of user information may be related to models' instruction-following tendencies and leaves this for future mechanistic studies, but this remains a hypothesis without direct experimental evidence. A quick analysis of the models' action rationales, where CoT is available, or controlled experiments examining the effects of source roles and instructions could help explain these patterns. I believe a deeper investigation of these behaviors would strengthen the paper's motivation by clarifying why agents fail to initiate conflict resolution despite their demonstrated detection capabilities. It could also provide more meaningful insights into the underlying causes of the recognition–behavior gap and how the proposed mediation mechanism addresses them.
-
Limitations of the proposed approach and problem formulation. The experimental setting makes relatively strong assumptions about grounded conflict resolution. In particular, the success metric requires agents to explicitly obtain the correct value through verification or user clarification before completing the task. Consequently, even when a baseline agent selects the correct value and successfully completes the task, its action is counted as unsuccessful if no explicit verification was performed. While this criterion is consistent with the paper's objective of measuring grounded resolution, it does not distinguish incorrect decisions from correct but unverified decisions. Therefore, the substantial improvements reported for the proposed framework may partly reflect an increase in explicit verification rather than an equivalent improvement in actual task correctness. Additionally, the evaluation assumes that the correct value is always recoverable through verification or user clarification, which may not hold in realistic environments where verification can be unavailable, inconclusive, or contradictory. Although these assumptions are reasonable within the paper's controlled setting, they limit the generalizability of the near-perfect results. If the authors consider these scenarios within the scope of their approach, reporting final task correctness separately and evaluating imperfect verification would strengthen the empirical findings. Otherwise, I would encourage them to explicitly acknowledge these limitations in Section 7.2 and clarify the boundaries of the proposed framework's applicability.
Questions for authors
-
How does the proposed framework compare against existing conflict-handling or clarification-seeking approaches? Would a simpler conflict-aware prompting strategy, or an LLM-based self-checking mechanism without the proposed structured representation, achieve comparable results?
-
Do the recognition–behavior gap and intervention improvements persist in realistic, multi-step agent trajectories where inconsistencies arise naturally?
-
Could the authors provide the complete original actor prompt and action specifications used in Section 4? Could these instructions influence the consistently low No Commitment (N.C.) rates?
-
Is there experimental evidence explaining agents' strong preference for user-provided information, particularly the suggested instruction-following hypothesis? Could action rationales or controlled experiments help explain this behavior?
7 comments
Q2: Generalizability to Multi-Step and Real-Web Agent Trajectories
We thank the reviewer for raising this important concern. To investigate SOIN's applicability beyond synthetic, single-decision settings, we extend our evaluation to multi-step web-agent trajectories and real-world websites, examining its effectiveness under dynamic execution conditions.
(1) Multi-Step SOIN Evaluation. We conduct experiments on 3-, 5-, and 10-step trajectories adapted from WebArena. For U–E and U–I conflicts, we modify user-provided information; for E–I conflicts, we introduce disagreements between information encountered on earlier and later webpages. SOIN performs consistency checking and mediation before each action, enabling agents to address conflicts as they emerge throughout execution.
| Model | Steps | ||
|---|---|---|---|
| GPT-5.6 Sol | 3 | 97.33±1.2 | 96.00±2.0 |
| 5 | 96.67±1.2 | 94.00±0.0 | |
| 10 | 95.33±1.2 | 90.67±1.2 | |
| GPT-5.6 Luna | 3 | 97.33±1.2 | 96.00±0.0 |
| 5 | 90.67±1.2 | 85.33±1.2 | |
| 10 | 84.00±2.0 | 76.67±1.2 | |
| Gemini 3.5 Flash Lite | 3 | 96.00±2.0 | 90.00±2.0 |
| 5 | 90.67±1.2 | 89.33±2.3 | |
| 10 | 88.67±2.3 | 84.67±2.3 |
The results demonstrate that SOIN checking and mediation remain effective across multi-step trajectories. Although performance gradually decreases with trajectory length, SOIN's per-action design enables continuous conflict assessment and mediation, supporting its applicability to longer agent workflows.
(2) Real-World Web Evaluation. We further evaluate SOIN on 50 real-world websites, using tasks adapted from our original benchmark. These tasks involve dynamic, multi-step interactions without predefined trajectory lengths. We introduce U–E and U–I conflicts through user-source modifications. For E–I conflicts, we introduce conflicting information into the internal state at an earlier step, which subsequently disagrees with external observations on later webpages.
| Model | Conflict Pair | ||
|---|---|---|---|
| GPT-5.6 Sol | U–E | 99.33±1.2 | 99.33±1.2 |
| U–I | 99.33±1.2 | 99.33±1.2 | |
| E–I | 98.67±1.2 | 98.00±2.0 | |
| GPT-5.6 Luna | U–E | 99.33±1.2 | 99.33±1.2 |
| U–I | 99.33±1.2 | 98.00±0.0 | |
| E–I | 98.67±1.2 | 98.00±2.0 | |
| Gemini 3.5 Flash Lite | U–E | 99.33±1.2 | 99.33±1.2 |
| U–I | 99.33±1.2 | 98.00±0.0 | |
| E–I | 98.67±1.2 | 98.00±2.0 |
The results demonstrate that SOIN maintains high detection and grounded resolution performance in dynamic real-web environments, consistent with our original controlled findings. These experiments extend our evaluation to actual webpage interactions and evolving agent states, with controlled conflict injection ensuring reproducibility. The current evaluation covers 50 websites, and we will further expand its coverage in the final manuscript.
Together, these experiments provide additional evidence of SOIN's effectiveness across multi-step execution and realistic web environments. We thank the reviewer for motivating these extensions. We will incorporate the experimental setups, results, and corresponding discussion into the revised manuscript.
Q3: SOIN Detection Beyond Literal String Matching
We thank the reviewer for highlighting the challenges of detecting semantic source disagreements. Our main experiments deliberately use explicit field-value conflicts to isolate source identity and enable reproducible evaluation of conflict detection, natural source arbitration, and downstream mediation. These structured conflicts can be identified through deterministic field matching, providing a controlled setting for examining how agents respond to readily detectable inconsistencies.
To investigate more complex disagreements, we extend our evaluation to Semantic SOIN, where source compatibility requires semantic interpretation. We consider four types of semantic variation: semantic equivalence, contextual dependency, negation/modality, and numerical/unit conversion [3,4]. We evaluate three models across these four types and three pairwise source configurations (U–E, U–I, and E–I), reporting averaged results. We focus on cases with well-defined source compatibility, as temporal changes and incomplete statements can require additional context to establish whether a genuine conflict exists. The new Appendix F provides the definition of Semantic SOIN and representative examples.
| Model | ||||
|---|---|---|---|---|
| GPT-5.6 Sol | 98.67±1.2 | 98.00±2.0 | 96.00±0.0 | 96.67±2.3 |
| GPT-5.6 Luna | 84.67±1.2 | 76.67±2.3 | 83.33±2.3 | 72.67±1.2 |
| Gemini 3.5 Flash Lite | 92.67±1.2 | 92.00±2.0 | 89.33±1.2 | 90.00±2.0 |
The results demonstrate that SOIN detection and mediation extend beyond literal field matching. GPT-5.6 Sol achieves 98.67% TP and 98.00% TN, alongside 96.00% grounded conflict-resolution success. Gemini 3.5 Flash Lite also maintains strong performance, while the lower scores of GPT-5.6 Luna highlight the additional challenges of semantic interpretation. These findings extend our evaluation to semantically expressed source conflicts and demonstrate how detection and mediation effectiveness depend on the underlying model's reasoning capabilities.
We thank the reviewer for motivating this important extension. We will incorporate these findings and their implications into the revised manuscript.
[3] Zhaochen Su et al. "CONFLICTBANK: A Benchmark for Evaluating Knowledge Conflicts in Large Language Models." NeurIPS 2024.
[4] Rongwu Xu et al. "Knowledge Conflicts for LLMs: A Survey." EMNLP 2024.
Q4: Original Actor Prompt and Scaffold Bias
We thank the reviewer for raising this important concern. The original actor prompt and action specifications are important for interpreting the observed recognition–behavior gap. We clarify three aspects of our experimental design.
(1) Source Preference Bias. We carefully reviewed the original browser-use prompt and removed instructions that could encourage prioritizing user-provided information. The resulting actor prompt does not prescribe an authority hierarchy among user (U), external (E), and internal (I) sources or favor any particular source when conflicts arise.
(2) Task-Completion Orientation. We intentionally preserve the original task-oriented instructions of browser-use, including its standard action and task-completion requirements, without adding explicit conflict-handling guidance. This design allows us to examine how agents handle source inconsistencies under a standard task-execution scaffold. Accordingly, our findings characterize unmediated agent behavior within this task-oriented environment.
(3) Prompt Transparency. To improve reproducibility, we now provide the complete original actor prompt in the new Appendix G of our updated manuscript, including its task-execution instructions and browser action rules. This enables direct examination of the scaffold used in our experiments.
We thank the reviewer again for emphasizing this important aspect of experimental transparency. We will further clarify these design choices and their implications for interpreting the recognition–behavior gap in the final manuscript.
Q1: Comparison with Existing Methods and Simple Prompting
We thank the reviewer for highlighting these important baselines. Ask-when-Needed [1] and CRITIC [2] make valuable contributions to clarification-seeking and tool-interactive verification. These works share SOIN's broader goal of improving agent reliability while investigating distinct research questions and intervention settings, as summarized below.
Our distinct contribution is the controlled investigation of unresolved source inconsistency across live agent channels. Through matched carrier–value exchange, we isolate source identity and reveal systematic source-dependent commitment (Section 4). We further investigate how explicit conflict information supports downstream mediation and grounded task resolution (Section 5), extending the analysis to DOM–visual conflicts (Section 6). These findings complement the clarification and self-correction capabilities investigated in prior work.
Regarding the suggested simple prompting baseline, we conduct an additional experiment using a single pre-action instruction to check whether user, webpage, and memory information disagree, and to verify or ask before committing. This Simple Prompt baseline uses no separate diagnostic call or structured SOIN state. All other experimental settings remain unchanged.
The results demonstrate substantial benefits from explicit SOIN checking and mediation. Full SOIN improves grounded conflict-resolution success by 34.33–52.00 percentage points compared with Simple Prompt, reaching 99.13–100.00%. Both approaches maintain high no-conflict success, with Full SOIN showing a small reduction in SRn. These findings demonstrate that explicit conflict diagnosis and structured mediation enable more reliable grounded resolution than a generic pre-action instruction alone.
We appreciate the reviewer for emphasizing these important comparisons. Ask-when-Needed advances proactive clarification in tool-using agents, while CRITIC establishes the effectiveness of tool-interactive self-correction. Both provide important and complementary insights into agent reliability. We will incorporate the additional baseline results and expand our discussion of their contributions and connections to SOIN in the revised manuscript.
[1] Wang et al. Learning to Ask: When LLM Agents Meet Unclear Instruction. EMNLP 2025.
[2] Gou et al. CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing. ICLR 2024.