acceptodds
Under review as a conference paper at ICLR 2027

When Tools Agree for the Wrong Reasons: Risk-Controlled Evidence Acquisition under Source Dependence

Abstract

Tool-using agents rely on external evidence, yet different tools may repeat information from shared sources or return plausible but corrupted records. We formulate evidence acquisition and hierarchical answer release as a joint risk–coverage–cost problem with distinct factual, unsupported-claim, and over-specificity losses. EviWeave-Agent uses a shared source-dependence representation to coordinate reliability-weighted fusion, active querying, and answer specificity. We evaluate the framework on CorruptEvidenceBench, a controlled benchmark spanning open-domain investigation, scientific evidence analysis, and cybersecurity triage under six corruption and dependence conditions. At α = 0.05, policylevel calibration selects a frozen acquisition-and-release policy whose held-out 90-cell ledger has maximum-cell factual risk 0.038, minimum correct-task coverage 0.861, and mean query cost 2.31. Relative to selective risk control without dependency modeling, the selected policy raises correct-task coverage by 17.1 percentage points, lowers factual risk by 0.9 points, and reduces mean query cost by 13.8%. Matched diagnostics show that removing dependency modeling increases maximum factual risk to 0.106, while removing active acquisition raises cost to 2.84. These results support treating source dependence as a shared decision variable across evidence fusion, query selection, and calibrated answer release under the evaluated benchmark conditions.

Primary area:
reinforcement learning
Keywords:
Tool-Using Agents, Risk Control, Active Evidence Acquisition, Source Dependence, Evidence Provenance, Claim-Conditioned Evidence Fusion, Robust Aggregation, Nonmyopic Planning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_2DH3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary
  • This paper introduces EviWeave-Agent, a framework for risk-controlled evidence acquisition in tool-using agents, motivated by the concern that multiple tools may provide seemingly corroborating evidence originating from shared upstream sources.
  • The framework integrates provenance-aware evidence fusion, active tool querying, and hierarchical answer release, with policy-level calibration formulated as a joint risk–coverage–cost optimization problem.
  • The authors evaluate the approach on CorruptEvidenceBench, a synthetically constructed benchmark spanning three task domains and six corruption conditions.
  • Results report improvements in correct-task coverage, factual risk, and tool-query cost relative to several controlled baselines. Ablations further examine the contribution of dependency modeling, reliability estimation and calibration.
Critical strengths
  1. Relevant consideration of correlated evidence.
  • The paper raises a legitimate concern that multiple tools may provide seemingly independent corroboration while relying on common upstream information. But there is a lack of examples here which makes it difficult to follow why is this even an issue.
  • The distinction between independent evidence and duplicated support is conceptually relevant to the assurance of agentic systems.
  1. Joint treatment of risk, coverage, and acquisition cost.
  • The attempt to coordinate evidence fusion, acquisition, and answer specificity within a common risk-control formulation is potentially useful. - The distinction between factual errors, unsupported claims, and over-specificity is also sensible, although the broader value of this integration remains insufficiently demonstrated.
Critical weaknesses
  1. Weak problem motivation and questionable research significance.
  • My primary concern is that the paper appears solution-driven without adequately establishing the significance of the underlying problem. Yes, correlated evidence is a legitimate consideration, but the authors do not convincingly demonstrate its prevalence or consequential impact in real-world tool-using agents.
  • There is an elaborate framework to introduce substantial complexity through provenance modeling, active acquisition, hierarchical release, and risk calibration. However, this becomes a little hard to appreciate without clarity on the fundamental challenge that we face today with agents.
  • What substantive improvement does EviWeave-Agent provide over contemporary, simpler approaches such as RA-RAG (Hwang et al., EMNLP 2025)
  • Can the authors demonstrate these benefits using independently implemented baselines and realistic retrieval tasks, rather than predominantly synthetic evaluations and internal framework variants?
  • The overall contribution therefore feels disproportionate to the problem being addressed, and the paper does not establish a sufficiently compelling research impetus.
  1. Unconvincing experimental validation and baselines.
  • The evaluation relies almost entirely on a synthetic benchmark with predefined provenance, fixed answer hierarchies, and constrained corruption conditions.
  • More concerningly, the comparisons largely involve simplified or ablated variants of the proposed framework rather than independently implemented contemporary agentic systems (Provenance, Fixed fusion, Selective etc.).
  • The ReAct + self-consistency baseline is also described as a symbolic proxy rather than a verified LLM-agent implementation.

These limitations significantly weaken the credibility and generalizability of the empirical claims.

Questions for authors
  • Problem significance: Can the authors provide concrete empirical evidence that the problem formulation is indeed a significant and consequential failure mode in contemporary tool-use agents? Why is the proposed joint framework necessary over simpler provenance approaches such as tracking, de-duplication, or some form of weighted fusion?

  • Baseline validity: Why are the main comparisons largely limited to symbolic or internal framework variants? In particular, how can the ReAct + self-consistency results substantiate comparative claims without an independently implemented LLM-based agent?

  • Real-world validation: How representative is CorruptEvidenceBench of actual agentic workflows? Can the authors substantiate the reported improvements on realistic retrieval tasks with independently obtained experimental results?

Reviewer_zSEzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I read the paper 4 times, couldn't understand anything after first paragraph. Lots of jargons and unrelated sentences, very poor presentation. I believe they want to give a reliability weight to every evidence from tools, so in the final result, they can decide which of them to incorporate

Critical strengths

NA

Critical weaknesses
  • Very poor presentation, not readable

  • in paragraph 2, it says "An agent must decide which evidence applies to the current claim, which source domains remain informative after shared provenance is accounted for, which query justifies its cost, and how specific the final answer should be. A coarse answer can be useful when the evidence supports malicious command-and-control behavior but does not support a specific malware-family label."

  • why we have "cost" for query here? isn't the problem removing duplicate results?

  • what is "malicious command-and-control behavior"

  • what is " specific malware-family"

in paragraph 3 we have: "Second, at the α = 0.05 operating point, policy-level calibration selects a frozen EviWeave- Agent policy with maximum-cell factual risk 0.038, minimum correct-task coverage 0.861 ..."

what is alpha? what is cell factual risk?

all of them are not defined, I couldn't understand anything from this paper.

Questions for authors

NA

Reviewer_6YjNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EviWeave-Agent, a tool-based agent that focuses on corruption and redundancy in evidence from multiple tools. It combines several existing methods, such as data provenance, reliability-weighted evidence fusion, and fitted Q-iteration. It also uses hierarchical selective classification to control answer specificity. Experiments are conducted on a synthetic dataset called CorruptEvidenceBench.

Critical strengths

S1. The paper studies an important problem in tool-using agents. S2. The paper presents extensive experimental results, which demonstrates a substantial amount of experimental work.

Critical weaknesses

W1. Lack of novelty. Most parts of the proposed method have already been studied in previous work [1,2,3,4]. I do not see new components here beyond existing methods. W2. The presentation of the paper should be improved. There are a lot of confusing concepts that make the paper hard to understand. Some of these concepts appear suddenly without any explanation. And the rhythm of these papers is terrible. I have to keep going back and forth between main text and appendices to find the necessary information. W3. I have serious concerns about the experiments. In Table 1, the paper reports the results (0.038,0.021,0.024) in benchmark. An interesting fact is that the authors provide the evidence about how they conducted data to match reported results in Appendix H "The source constructs these rows from threat anchors with domain and seed offsets". Why did the authors not provide the original experimental results instead?

[1] J. Hwang et al. Retrieval-Augmented Generation with Estimation of Source Reliability. EMNLP'25. [2] D. Li et al. IMRRF: Integrating Multi-Source Retrieval and Redundancy Filtering for LLM-based Fake News Detection. NAACL'25. [3] W. Su et al. DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models. ACL'24. [4] S. Goren et al. Hierarchical Selective Classification. NeurIPS'24.

Questions for authors

See W1-W3

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.