acceptodds
Desk rejected at ICLR 2027

Mind the Tool Failures: Achieving Synergistic Tool Gains for Medical Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

For medical AI agents, disagreement among external tools can signal both a risk of error and the potential for better decisions: tools that fail on different instances may together outperform even the strongest individual tool. We define tool synergy as the ability to realize these complementary gains and characterize its potential through the Single-Oracle risk gap, which measures the performance gap between the best fixed single tool and an ideal instance-wise tool-use policy. Yet instance-level adaptation alone does not guarantee synergy, as standard outcome-based reinforcement learning and conventional aggregation methods do not explicitly exploit disagreement to learn when and how tools can complement one another. To address this challenge, we propose Collaborative Synergy Reinforcement Learning (CSRL), a GRPO-based framework that turns tool disagreement into an explicit learning signal for synergy. CSRL combines a Brier reward for probabilistic risk minimization, an override reward for disagreement-aware conflict resolution, and Entropy-Guided Sampling that prioritizes high-disagreement cases with stronger signals for learning complementary tool use. Across two tasks and seven medical benchmarks, CSRL consistently outperforms a broad range of baselines, with its advantage rising from 0.4% under tool consensus to 6.4% and 7.2% under moderate and high disagreement, respectively, highlighting its ability to turn tool disagreement into complementary gains.

Primary area:
reinforcement learning
Keywords:
Medical AI Agents, Tool-Augmented Agents, Tool Synergy, Tool Disagreement, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_KmtnOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the challenge of heterogeneous tool failures in medical AI agents by proposing Collaborative Synergy Reinforcement Learning (CSRL). The authors argue that disagreement among external diagnostic tools is not merely a source of uncertainty, but a valuable signal for exploiting case-level complementarity. This paper introduces the Single-Oracle risk gap, and guided by this metric, CSRL employs a GRPO-based optimization framework equipped with three key mechanisms: a Brier reward for probabilistic risk minimization, an override reward for disagreement-aware conflict resolution, and Entropy-Guided Sampling (EGS) to prioritize high-disagreement training cases.

Critical strengths
  • The paper is well-organized. This proposed method actively exploits the complementarity and disagreement among imperfect tools. This is highly relevant for high-stakes domains like healthcare.
  • The integration of the Brier reward, override reward, and EGS is logically sound. The authors provide theoretical propositions proving that maximizing these rewards aligns with minimizing prediction risk and enlarging the synergy risk gap.
  • The method is evaluated across diverse datasets, tasks, and policy model scales. The inclusion of detailed ablation studies and multi-seed robustness analysis strengthens the validity of the effectiveness of the proposed method.
Critical weaknesses
  • The effectiveness of CSRL fundamentally relies on the premise that the available tools have heterogeneous failure modes. If a predefined tool pool consists of models with highly correlated errors, the Single-Oracle risk gap shrinks, limiting the potential for synergistic gains. The paper assumes a well-curated tool pool but does not deeply address how to select or construct such a pool dynamically. In addition, many medical scenarios lacks the effective tools, which may also limit the applicability of this method.
  • As acknowledged in the Ethics Statement, the evaluation is strictly retrospective on existing benchmarks. The actual clinical safety, robustness compared to real-world data. The authors are encouraged to train on internal trajectories and test on external real-world clinical cases.
Questions for authors

NA

Reviewer_MnTZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the problem of heterogeneous failures in tool-augmented medical diagnosis agents. Instead of relying on a single best tool or naive aggregation (e.g., majority voting), the authors propose CSRL (Collaborative Synergy Reinforcement Learning), a GRPO-based framework that learns to exploit tool disagreement. CSRL introduces three mechanisms: (1) a Brier reward to minimize probabilistic diagnostic risk; (2) an override reward that explicitly supervises the policy to resolve conflicts by rewarding beneficial corrections of erroneous consensus; and (3) Entropy-Guided Sampling (EGS) to oversample high-disagreement cases, providing stronger learning signals. The method is validated on chest X-ray classification (6 benchmarks) and medical VQA, consistently outperforming individual tools, ensemble/aggregation baselines, and standard GRPO, with larger gains observed under high tool disagreement.

Critical strengths

Highly effective disagreement-focused sampling (EGS): The realization that tool consensus (all correct or all wrong) provides little learning signal for synergy, and that training should be dynamically reweighted towards high-entropy (moderate disagreement) cases, is highly practical and insightful. EGS elegantly solves the issue of sparse complementary signals in long-tailed medical datasets.

Explicit conflict resolution via override reward: Training the policy not just to be accurate, but specifically to override majority consensus when the majority is wrong, directly addresses the core failure mode of ensemble methods. The theoretical justification linking the override reward to maximizing the risk gap over the majority vote (Proposition 2.2) is an elegant way to frame this behavior.

Critical weaknesses

Theoretical contributions are somewhat trivial: Proposition 2.1 is simply the definition of the Brier score. Proposition 2.2 is a basic algebraic manipulation of expectations. Presenting these as formal "Propositions" overclaims the theoretical depth of the paper.

Narrow domain evaluation despite general claims: The paper positions itself as a general framework for "tool synergy under heterogeneous failures," yet all experiments are strictly confined to chest X-ray (CXR) diagnosis. Tool-use agents in other domains (e.g., web search, coding, API calls) exhibit different types of disagreements. It is unclear if the override reward and EGS would translate directly to non-medical, multi-modal or text-only environments where the "loss" is not a simple probability threshold.

Several important related studies are missing, particularly on tool feedback for agentic RL, including Tool-Star and Agentic Reinforced Policy Optimization (ARPO). A comparison with these methods would make the current contribution stronger.

Questions for authors

Regarding the theoretical claims: Since Propositions 2.1 and 2.2 are relatively direct derivations from the definitions of Brier loss and expectation, could you clarify what specific non-obvious insights these propositions provide for the learning dynamics that cannot be inferred from the empirical results?

Regarding EGS: By artificially oversampling high-disagreement cases during training, does the policy suffer from a calibration shift on the natural test distribution? Have you observed any degradation in performance on the "consensus" cases (e.g., all tools are correct) due to this distribution shift?

If a tool in the pool is universally bad (always wrong), how does the override reward and EGS prevent the policy from wasting capacity on trying to "learn" from its noise, and instead correctly learn to ignore it?

Reviewer_zPP2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether complementary failures among medical tools can be converted into gains beyond the best fixed individual tool. It introduces the Single–Oracle risk gap and proposes CSRL, a GRPO-based training framework for Qwen2.5-VL-7B that combines a Brier reward, an override reward relative to the majority of invoked tools, and Entropy-Guided Sampling (EGS) that oversamples cases with heterogeneous tool correctness. Experiments cover six chest X-ray classification benchmarks and a separately trained chest X-ray VQA setting. On the six classification benchmarks, CSRL reports an average Macro Accuracy/F1 of 82.2%/57.8%. A three-seed analysis on three datasets reports 78.0%/54.9% for CSRL versus 72.4%/52.5% for Vanilla GRPO. The empirical gains are meaningful, especially on disagreement cases, but I have concerns about the theoretical justification of the override objective, the attribution of gains to tool selection versus tool integration, and incomplete controls/details for the VQA experiment.

Critical strengths
  1. The paper studies a concrete failure mode of multi-tool medical agents: different tools fail on different instances. The disagreement-stratified evaluation, failure-recovery analysis, best-single preservation, and override precision provide more behavioral evidence than aggregate accuracy alone.

  2. The classification experiments include a reasonably broad set of comparisons: individual tools, majority voting, best-tool-per-pathology, logistic regression, several stacking methods, an image-aware MLP, MLLM aggregators, and Vanilla GRPO. The ablation also separates the effects of Brier reward, override reward, and EGS.

  3. The paper includes useful robustness checks, including three-seed experiments, multiple OOD benchmarks, the radiologist-adjudicated NIH-Google set, and demographic subgroup analyses with patient-level bootstrap intervals.

Critical weaknesses
  1. Proposition 2.2 is not established under the stated definition of the override reward. In Eq. (10), the majority prediction m is computed over the valid tools invoked by the policy, so its risk R(m) may depend on the policy parameters. The identity relating expected override reward to R(m) − R(π) still holds, but the proof in Appendix B.1 then treats R(m) as independent of the policy. Consequently, maximizing this reward does not necessarily imply reducing the policy’s own risk. The proposition needs an explicit policy-independent majority reference or a narrower conclusion. This issue concerns the theoretical guarantee and does not by itself invalidate the empirical gains.

  2. The current comparisons do not isolate whether adaptive tool selection adds value beyond learning to integrate tool outputs. CSRL learns both behaviors jointly, while the aggregation baselines differ in their model or training procedure. The tool-budget ablation does not compare the adaptive policy with an otherwise matched CSRL policy trained to integrate a fixed set of all available tools.

  3. Several details important for reproducibility and interpretation are missing. These include the effective number of classification training examples, pathology composition and uncertain-label handling, validation/model-selection protocol, tool-score calibration and binarization, and training/tuning details for learned aggregation baselines. In addition, some reported “OOD” sets may be OOD for the policy training distribution but not necessarily for every tool in the pool (e.g., a NIH-trained DenseNet is itself included), so the OOD claim should be qualified and tool pretraining overlap should be documented.

  4. The VQA study is useful as secondary evidence of applicability, but does not isolate the benefit of CSRL over a matched RL baseline. More immediately, the paper mentions both a 1,523-question test set and a 933-question non-empty-GT subset without clearly identifying which population Table 2 reports. The VQA-specific reward, correctness criterion for EGS, answer parsing/equivalence rules, and empty-answer handling are not described with sufficient detail. Since the VQA policy is separately trained on 5,000 VQA samples, this experiment supports applicability to another task rather than direct cross-task transfer of the classification policy.

Questions for authors
  1. In the implementation, is the override reference m computed from the fixed full tool pool or from the subset selected by the current policy? Please reconcile this implementation with Proposition 2.2. If the reference depends on the policy, please revise the claimed alignment result and its proof accordingly.

  2. In the existing three-dataset classification setting, can the authors compare CSRL with one control that always invokes all six tools and learns to integrate their outputs, using the same backbone, reward design, and training schedule? This would directly test the additional value of adaptive tool selection.

  3. Please clarify the classification data and evaluation protocol, particularly the effective training-set size, pathology labels and uncertain-label handling, checkpoint-selection procedure, tool-score thresholds, and training/tuning protocol for the learned aggregation baselines. Please also distinguish datasets that are OOD for the policy from those that are unseen by every tool in the pool.

  4. Does Table 2 report results on all 1,523 VQA questions or the 933-question non-empty-GT subset? Please specify the VQA training reward, correctness criterion used by EGS, and answer-matching and empty-answer rules. Given the separate VQA training, I interpret this experiment as evidence of task applicability rather than direct transfer of the classification policy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.