acceptodds
Under review as a conference paper at ICLR 2027

When Should an AI Agent Change Its Mind? Counterfactual Evaluation of Multimodal Evidence Reasoning with Contrast-Persistence Tests

Abstract

Multimodal agents increasingly make decisions by acquiring evidence from heterogeneous sources, yet they are typically evaluated by final-answer accuracy alone. This leaves a basic reliability question unanswered: does an agent change its reported prediction when the evidence warrants reconsideration, while remaining stable when it does not? We introduce a counterfactual evaluation framework for multimodal evidence reasoning based on relevant addition, countervailing contradiction, exact redundancy, and important removal. The framework separates fixed-input response, autonomous whole-system response, matched-input conditional integration, and deterministic post-processing. This separation enables a contrast-persistence test: a whole-system difference should not be attributed to evidence integration when it changes materially after the compared integrators receive the same prior state and newly added evidence. Apparent robustness should likewise not be attributed to model reasoning when deterministic post-processing alone recreates it. ADNI remains our primary study, comparing Direct, Tool Agent, and Evidence Ledger pipelines across three open-weight backbones. Under a common-input, common-prior addition replay, the Ledger–Tool contrast, averaged equally across the tested backbones for each subject, changes from autonomously to 0.012 conditionally, a gap of (95% CI to ). In IBDome, the corresponding four-backbone contrast falls from 0.131 to 0.014, a gap of 0.117 (0.089–0.144). Exact-duplicate stability also degrades sharply when the Ledger preservation guard is removed and is recovered by guard-only processing. Together, these results show that endpoint performance, autonomous response, conditional integration, and engineered invariance are distinct evaluation objects. Multimodal agents should therefore be evaluated not only by what they predict, but also by whether an attributed evidence-revision property survives controls on the pipeline components that can produce it.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Agents, Counterfactual Evaluation, Evidence Reasoning, Tool-Using LLM Agents, Biomedical AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UJBbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a framework for attributing behavior in LLM-based evidence-reasoning pipelines and applies it to two biomedical classification tasks (ADNI, 787 subjects; IBDome, 172 subjects) using three author-built systems (Direct, Tool, Ledger) across three or four backbones. Four objects are distinguished: fixed-input response, autonomous whole-system response, matched-input and matched-prior conditional replay, and component-controlled behavior. The central test asks whether the autonomous contrast between two systems persists when input and prior are held fixed. On ADNI the headline Ledger-vs-Tool contrast moves from -0.091 to 0.012 under conditional replay (gap -0.103, 95% CI [-0.162, -0.044]), reversing sign; on IBDome the gap is 0.117. The paper also argues that the Ledger's duplicate stability comes from a deterministic preservation guard rather than reasoning, and treats the Ledger as an instrumented control.

Critical strengths

The paper is unusually honest about what it does not show: it disclaims causal mediation, clinical validity, and native multimodal understanding, and reports a sign flip in its own headline result. The paired design is sound in construction, with subject-level pairing, equal-weight within-subject backbone averaging, 10,000-replicate subject bootstrap, and explicit exclusion accounting. Most reported arithmetic (Table 3 gaps, finite-panel averages, Eq. 5 cells, cohort counts) recomputes exactly.

Critical weaknesses
  • The endpoint table contradicts the paper's own fixed-order table: Direct is defined as receiving all allowed evidence, yet Tables 2/4 report Qwen 0.5566 and Gemma 0.6058, which are their +Genomic values in Table 6 rather than the full-evidence +MRI values (0.5551 and 0.5943), and the text itself describes 0.5566 as the post-genomics number.
  • The "no common winner across backbones" claim rests on these values, so this needs reconciliation.
  • There are no baselines, not even a constant classifier, and several reported cells coincide exactly with constant-classifier values (for example, Ministral clinical equals all-non-IBD at 16/172).
  • A substantial part of the evidence-response analysis therefore operates on degenerate predictors, which makes the reported effect sizes hard to interpret.
  • Several quantities central to the argument are never tabulated, including the neutral-control arm behind the 0.212 result, the per-backbone duplicate reductions, and the countervailing true-label log-scores.
  • The systems, prompts and guard thresholds are not described and the pipelines are attributed to an uncited "original paper", so the submission is neither self-contained nor reproducible.
Questions for authors
  1. Please reconcile the Direct values in Tables 2/4 with the +Genomic and +MRI rows of Table 6, and state which evidence condition Direct actually uses.
  2. Can you add a constant/majority baseline and a simple tabular model on the identical encoded evidence, so the reported contrasts have a reference point?
  3. Please quantify run-to-run variability (number of repeats, decoding settings) and report whether the headline confidence intervals survive it.
  4. Please justify the equivalence margins (m = 0.020/0.025 and +/-0.003) and show the persistence decisions as a function of m.
4 comments

We thank the reviewer for the careful assessment and independent numerical checks. We addressed the concerns through additional verification and analyses. The corrected Direct endpoints preserve the backbone-level rankings, and repeated full-cohort ADNI evaluations support the original nonpersistence conclusion. We respond to each point below.

W1 & Q1. Direct endpoint inconsistency

We acknowledge the discrepancy between the Direct endpoint values in Tables 2/4 and the fixed-order trajectories in Table 6.

We verified the released subject-level prediction artifacts and confirmed that the reported Direct Macro-F1 values for Qwen and Gemma correspond to the +Genomic condition rather than the final +MRI condition. The verified full-evidence values are:

BackbonePreviously reportedVerified full-evidence (+MRI)
Qwen0.55660.5551
Gemma0.60580.5943
Ministral0.51170.5117

The discrepancy arose from the endpoint mapping, which selected base_prediction for Qwen and Gemma but addition_prediction for Ministral. This was a reporting inconsistency rather than a change in the Direct system definition, as the full-evidence endpoint corresponds to the final +MRI condition.

The corrected +MRI values preserve the highest-Macro-F1 system for each backbone, with Direct leading for Qwen, Ledger for Gemma, and Tool for Ministral. The conclusion that no single architecture consistently outperforms the others across backbones therefore remains unchanged.

W2 & Q2. Predictive baselines and interpretation of evidence-response results

We computed constant-class baselines for both cohorts and a supervised tabular baseline on the same ADNI evaluation subjects.

Constant and majority-class baselines. For ADNI (787 subjects), always predicting the majority class (MCI) yields 55.53% accuracy and 0.2380 Macro-F1. For IBDome (172 subjects), always predicting Crohn disease yields 62.79% accuracy and 0.2571 Macro-F1. The empirical class-prior true-label log-score references are −0.9887 for ADNI and −0.8693 for IBDome; these characterize class-prevalence distributions rather than held-out predictive performance.

Five-fold out-of-fold logistic-regression baseline. We evaluated logistic regression on the same 787 ADNI subjects, using clinical/demographic features alone and with cognitive features added. Five-fold out-of-fold prediction ensured that each subject was evaluated by a model trained without that subject.

Adding cognitive evidence yielded a mean true-label log-score gain of 0.4577 (95% CI [0.4115, 0.5041]) for logistic regression, compared with 0.5161 (95% CI [0.4724, 0.5606]) for Qwen Direct. The subject-paired difference was 0.0583 (95% CI [0.0139, 0.1020]). Other LLM pipelines showed heterogeneous gains rather than a uniform advantage over the tabular baseline.

Predictive performance and evidence-revision attribution are distinct. The majority-class baselines contextualize endpoint performance, whereas paired interventions measure responses to evidence changes. An invariant prediction (TV = 0) is an observable response property, not evidence of successful reasoning by itself.

Hard-label concentration versus probabilistic revision. We also examined the frozen ADNI counterfactual predictions at both class-label and probability-distribution levels. With cognitive evidence added, Ledger's predicted class changed for 60.0% (Qwen), 74.7% (Gemma), and 87.4% (Ministral) of subjects, with corresponding mean TV distances of 0.3600, 0.4463, and 0.5568.

Even among subjects whose predicted class remained unchanged, mean TV distances were 0.2322, 0.4664, and 0.5556, respectively. Thus, unchanged class predictions can coexist with substantial probability-level revisions, demonstrating that hard-label concentration does not necessarily imply nonresponsiveness to evidence.

W3. Neutral-control, duplicate-ablation, and countervailing results

We report the intervention- and backbone-level estimates underlying these results.

Matched-neutral control. The ADNI semantic-minus-neutral true-label log-score contrast is 0.212 (95% CI [0.171, 0.254]), using subject-first, equal-weight aggregation across systems and backbones. The intervention preserves cognitive values and serialization while replacing clinically meaningful field names with neutral labels, separating the contribution of field semantics from that of the numerical values and format.

The backbone- and system-level contrasts are:

BackboneDirectToolLedger
Qwen0.2820.2850.258
Gemma0.2630.2800.232
Ministral0.1610.173−0.024

The positive panel-level contrast is supported by the Direct and Tool responses, with variation across systems and backbones.

Duplicate-preservation ablation. Removing the Ledger's duplicate-preservation guard increases median duplicate-induced TV from 0.000 to 0.097. The no-guard-minus-full mean TV differences are +0.0045 for Qwen, +0.1022 for Gemma, and +0.2584 for Ministral (787 subjects per backbone), with a pooled mean of +0.1217 (95% CI [0.1155, 0.1280]).

Guard-only processing restores median TV to 0.000, with a guard-only-minus-full mean difference of −0.000368 (95% CI [−0.000517, −0.000237]). These component-controlled results show that deterministic preservation can reproduce the observed duplicate stability without requiring model-level reasoning, supporting the distinction between whole-pipeline behavior and model-level evidence integration.

Countervailing evidence. Under the APOE4 intervention, the signed changes in AD probability are 0.257 (Direct), 0.190 (Tool), and 0.099 (Ledger). The corresponding true-label log-score changes are −0.154, −0.126, and 0.019, respectively. Thus, probability changes in the expected direction do not necessarily improve predictive performance.

W4. System specifications and reproducibility

Direct, Tool, and Ledger are the three operational pipelines evaluated in this study, with distinct evidence-processing procedures:

  • Direct predicts from supplied evidence without autonomous tool acquisition.
  • Tool autonomously acquires evidence before producing its predictive distribution.
  • Ledger adds explicit evidence-audit and reconciliation procedures to the acquisition-and-update process.

The released supplementary artifacts include prompts, structured-output schemas, model configurations, Ledger audit and reconciliation specifications, duplicate-preservation guard procedures, probability-processing policies, and frozen subject-level predictions.

The Evidence Ledger audit records four structured fields: Relation (supporting, contradicting, neutral, or unresolved), Novelty (new, partly redundant, or redundant), Relevance (low, medium, or high), and Reliability (low, medium, or high). The reconciliation stage updates the predictive distribution using the accumulated ledger and previous distribution, preserving predictions for redundant evidence and limiting revisions for low-information evidence.

Decision thresholds and implementation provenance. We inspected the original counterfactual implementations and recovery/resume code to verify how evidence-processing decisions are operationalized. Duplicate-handling branches use explicit existence checks (e.g., len(duplicate_rows) > 0) and categorical evidence judgments (e.g., profile["novelty"] == "redundant"), rather than a universal numerical relevance or reliability cutoff.

The Tool controller distinguishes admissible from selected actions through allow_stop and explicit STOP branches. Ledger processing records reconciliation outcomes including accept_candidate, redundant_freeze, and stop. The ADNI execution traces contain 1,066, 1,284, and 1,309 accept_candidate events for Qwen, Gemma, and Ministral, respectively, and 185 redundant_freeze events for Ministral. The trajectories retain decision-basis and reconciliation-rule fields, enabling inspection of individual acquisition and reconciliation decisions.

For ADNI, structured outputs specify probability vectors over CN, MCI, and AD. The analysis implementation normalizes nonnegative probabilities to sum to one, uses a uniform fallback for zero-total vectors, and applies a numerical floor of 1e-12 when computing true-label log scores. Recorded inference configurations specify do_sample=False for all three open-weight backbones, so temperature and top-p do not operate as sampling parameters.

The frozen subject-level predictions and analysis code support regeneration of the reported aggregate results and subject-paired bootstrap estimates. The accompanying prompts, schemas, guard procedures, and configurations document the implemented inference process independently of the experiments' development history.

Q3. Run-to-run variability and headline robustness

We assessed inference variability using same-condition neutral-control diagnostics and repeated ADNI counterfactual evaluations.

The neutral-control diagnostics covered 200 subjects per backbone and system. Mean absolute true-label log-score drift across Qwen, Gemma, and Ministral was 0.071, 0.101, and 0.227 for Direct; 0.106, 0.114, and 0.303 for Tool; and 0.565, 0.431, and 1.351 for Ledger, respectively. These results show that output-level variability can arise even with deterministic decoding (do_sample=False).

We also performed three full-cohort executions of the autonomous Tool/Ledger counterfactual evaluation and conditional integration replay for each ADNI backbone (787 subjects). Averaged across executions, the backbone-specific autonomous–conditional gaps were −0.1284 (Qwen), −0.0595 (Gemma), and −0.2134 (Ministral), yielding a backbone-averaged gap of −0.1338 (subject-paired bootstrap 95% CI [−0.1900, −0.0779]).

The original ADNI gap was −0.1031 (95% CI [−0.1623, −0.0440]). Both the original and re-executed confidence intervals lie entirely outside the ADNI equivalence region of ±0.020, preserving the nonpersistence classification under the evaluated configurations.

These confidence intervals capture subject-level uncertainty conditional on the evaluated outputs, while the additional executions assess stability under the recorded inference configuration.

Q4. Equivalence-margin rationale and sensitivity

The margins of ±0.020 (ADNI) and ±0.025 (IBDome) are operational tolerances for assessing whether autonomous and conditional log-score contrasts are sufficiently close to be considered persistent. They are not clinically calibrated minimum important differences, nor are these specific values uniquely determined by the data.

We recomputed the persistence classifications across alternative margins. Both ADNI and IBDome remain materially nonpersistent for margins from ±0.005 through ±0.040. At ±0.050, ADNI becomes inconclusive, while IBDome remains materially nonpersistent. Applying a common margin of either ±0.020 or ±0.025 to both cohorts also preserves the nonpersistence classification, showing that the conclusion does not depend on cohort-specific margin choices.

The tighter ±0.003 margin addresses a separate comparison of guard-only and full duplicate handling. The guard-only-minus-full mean TV difference is −0.000368 (95% CI [−0.000517, −0.000237]). Equivalence is supported at both the original ±0.003 margin and the stricter ±0.001 margin, while the result becomes inconclusive at ±0.0005.

These sensitivity analyses show that the nonpersistence conclusions are robust across a substantial range of margins, while the guard-attribution result supports equivalence under a tighter tolerance.

We hope these additional analyses address the reviewer's main concerns regarding numerical consistency, predictive baselines, reproducibility, and robustness. The corrected endpoints preserve the backbone-level rankings, while the full-cohort ADNI re-execution and component-controlled analyses provide further support for the paper's central findings.

Thanks for reconciling the Direct endpoint values and for the additional analyses on run-to-run variability, baselines, and equivalence margins. I have read the response and have no further questions.

Reviewer_CZWUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes "contrast-persistence evaluation" for agents that acquire evidence from several sources: an end-to-end (autonomous) difference between two pipelines should not be attributed to their evidence integrators unless the difference persists when both integrators receive the same prior state and the same added evidence, and an apparent robustness property should not be attributed to model reasoning if deterministic post-processing alone recreates it. The framework reuses four evidence interventions (relevant addition, countervailing replacement, exact duplication, important removal) and three pipelines (Direct, Tool Agent, Evidence Ledger) from what the text repeatedly calls "the original paper". On ADNI (787 subjects, three open-weight backbones) the autonomous Ledger-minus-Tool log-score contrast under addition is −0.091 while the matched-input replay contrast is 0.012 (gap −0.103, 95% CI −0.162 to −0.044); on IBDome (172 subjects, four backbones) the corresponding values are 0.131 and 0.014. Removing the Ledger's duplicate-preservation guard raises median TV under exact duplication from 0.000 to 0.097 and the guard alone restores it. A matched-neutral control shows that the response to MMSE/ADAS depends on the field names and not only on the numbers (0.212 log-score units).

Critical strengths

S1. The analysis is subject-paired on two controlled-access biomedical cohorts, uses a 10,000-replicate subject-level bootstrap that preserves the pairing, reports backbone-level estimates beside the panel estimate, and releases frozen subject-level outputs with the analysis code. The matched-neutral control (same numbers, placeholder field names) is a sound way to show that the addition effect is semantic rather than a formatting artifact.

S2. The authors state clearly what the replay does not identify (acquisition value, mediation, policy effects) and do not oversell the Ledger as an architecture.

Critical weaknesses

W1. The headline result is a consequence of the experimental design, and its most likely cause is visible in the paper's own tables. In the autonomous arm, "adding" cognition means making it available for acquisition; an agent that stops before acquiring it shows no gain. Table 4 shows that the Ledger acquires fewer modalities than the Tool Agent on every backbone (1.36 vs 2.75, 1.63 vs 2.48 and 1.90 vs 2.91 calls), so a smaller autonomous addition gain for the Ledger is expected before any experiment is run, and force-feeding both integrators the same payload must remove exactly that difference. The paper then presents the shrinkage of the contrast as the discovery that "whole-system response and conditional integration are different empirical objects". Holding more things fixed removes the effects of the things held fixed; that is the definition of a control, not a finding. The one analysis that would turn it into a finding, the autonomous contrast conditioned on whether the cognitive modality was actually acquired, or a decomposition of the gap into acquisition, ordering and stopping, is explicitly disclaimed ("nor does nonpersistence identify which upstream mechanism caused the autonomous excess"). The "sign reversal" is overstated for the same reason: the conditional contrast of 0.012 carries no confidence interval, two of three backbone-level gaps are "inconclusive" by the paper's own rule, and the panel figure is carried by Ministral (−0.231). Table 1, which lists what each design licenses one to conclude, is correct and is not a contribution.

W2. The IBDome "cross-domain stress test" is run on pipelines that do not work. With all five modalities available, three of the four backbones sit below the majority-class accuracy of 62.8 percent (Qwen 37.8, Gemma 48.8, GPT-5.6 52.9; Table 7), and with clinical evidence alone Ministral's 9.30 percent is exactly the non-IBD share (16/172), a constant prediction. Table 9 shows that Gemma's Direct and Tool systems have a TV of 0.0000 under every intervention, and Table 5 shows that the non-Ledger systems of Gemma, Qwen and GPT ignore the Olink edit on 91 to 100 percent of subjects. The autonomous Ledger–Tool contrast on IBDome therefore compares a scaffold that forces attention to a field against systems that do not react to the field at all; nothing about evidence revision can be learned from it. ADNI shows the same symptom at the start of every trajectory: 26.05 percent accuracy with clinical/demographic evidence is exactly the CN share (205/787), again a constant prediction. No majority-class, logistic-regression or any other non-LLM reference appears anywhere in the paper.

W3. The question in the title is never operationalized, and "multimodal" is nominal. "When should an agent change its mind" requires a normative reference for how much a prediction ought to move when MMSE/ADAS or APOE4 arrive; a logistic model on the same tabular fields would provide one and is standard on ADNI. The paper instead repeats that TV "answers whether the output changed, not whether the change was useful" and that proper scores "do not label every individual update as warranted", and stops there. All ADNI interventions act on tabular fields; Appendix G shows the MRI representation contributes nothing (0.006, CI −0.004 to 0.017), nor do RNA-seq or WES on IBDome; and the main text never says how MRI or H&E are encoded for a text model. What is evaluated is the response of language models to edits in serialized tabular text. W5. The manuscript is not a self-contained paper and does not describe its own systems. It refers throughout to "the original paper", "the original study", "the original question of this work", "legacy" metrics "retained only for continuity", "historical" outputs, "the revised confirmatory claims" and "the present analysis"; Table 3 is captioned "Acceptance-critical contrast-persistence and component results", which is language addressed to reviewers rather than readers. The methods a reader needs are absent: the Ledger's audit and reconciliation prompts and its four fields, the "cap" on low-information updates, how probability vectors are elicited and floored, whose trajectory supplies the "frozen pre-addition prior" u_i in the replay, and the size of same-condition rerun drift, which Appendix K admits exists but never quantifies against the −0.103 headline gap. The equivalence margins (0.020 and 0.025 log-score units) are asserted without justification and differ between datasets for no stated reason. Much of the text restates itself: the sign convention of the contrast is defined at least three times, the phrase "evaluation objects" or "empirical objects" recurs at least six times, and Sections 6 and 8 contain nothing that Section 5 does not.

Questions for authors

Q1. On each ADNI backbone, what fraction of autonomous Ledger and Tool trajectories acquired the cognitive modality, and what is the autonomous Ledger–Tool contrast restricted to subjects on which both did?

Q2. Whose trajectory supplies the pre-addition prior u_i in the conditional replay, and what is the confidence interval of the conditional contrast of 0.012?

4 comments

We thank the reviewer for the careful examination of our experimental design and released artifacts. We address the specific questions below and clarify the scope and interpretation of our attribution findings.

W1 & Q1. Acquisition behavior and contrast persistence

Table 4 reports total modality acquisitions, not the fraction of trajectories acquiring cognition. We verified the released ADNI autonomous trajectories and found that both Tool and Ledger acquired cognition for all 787 subjects on each backbone (100%). Restricting the analysis to subjects where both acquired cognition therefore retains the full cohort. The Ledger–Tool addition contrasts remain −0.0266 (Qwen), −0.0482 (Gemma), and −0.1990 (Ministral), with a finite-panel contrast of −0.0913.

The lower total acquisition counts for Ledger therefore cannot explain the addition gap through failure to acquire cognition. However, other differences in acquisition history, ordering, stopping, or context construction may contribute.

The autonomous trajectories show substantial differences in other acquisition behaviors. Genomic acquisition rates for Tool versus Ledger were 76.5% versus 0.0% (Qwen), 59.7% versus 2.9% (Gemma), and 97.2% versus 0.1% (Ministral). Explicit STOP rates, measured as the fraction of trajectories containing a recorded STOP event, were 23.5%, 40.3%, and 8.8% for Tool, compared with 100.0%, 97.1%, and 99.9% for Ledger, respectively, across Qwen, Gemma, and Ministral. Acquisition ordering also differed across systems.

These differences establish that the systems exhibit distinct acquisition and stopping behaviors, although they do not identify the causal contribution of each mechanism to the observed gap.

We agree that matching inputs is a standard control. The contribution is the attribution test built around this control, rather than the control operation itself. The autonomous–conditional gap of −0.103 (95% CI [−0.162, −0.044]) shows that the autonomous contrast is not preserved under matched prior and added evidence. This rules out attributing the whole-system contrast solely to conditional evidence integration, without identifying which upstream mechanism accounts for the difference.

The finite-panel sign difference does not establish a universal backbone-level reversal.

Q2. Prior provenance and conditional confidence interval

The replay uses two prior-origin conditions per subject: Tool-origin and Ledger-origin. Within each condition, both integrators receive the same pre-addition predictive distribution and identical cognitive evidence. Contrasts are averaged across origins at the subject level before paired bootstrap aggregation.

The conditional Ledger–Tool contrast is 0.0118 (95% CI [0.0047, 0.0187]). Backbone-specific estimates are Qwen −0.0034 [−0.0141, 0.0074], Gemma 0.0068 [−0.0001, 0.0136], and Ministral 0.0318 [0.0164, 0.0483].

The residual conditional effect is small and heterogeneous. Our principal finding concerns the gap between autonomous and conditional contrasts, not a consistent integration advantage for either system.

W2. Predictor degeneracy and the IBDome stress test

We acknowledge the weak endpoint performance and near-invariant outputs of several IBDome pipelines. These results cannot establish reliable clinical prediction or successful integration of every modality.

Predictive competence and evidence-revision attribution are distinct questions. A system's failure to respond to an evidence intervention is itself observable behavior, although it limits conclusions about integration quality.

We computed constant-class and majority-class baselines from the released evaluation labels. On ADNI (787 subjects), the majority-class predictor (MCI) achieves 55.53% accuracy and 0.2380 Macro-F1. On IBDome (172 subjects), the majority-class predictor (Crohn disease) achieves 62.79% accuracy and 0.2571 Macro-F1. The class distributions also confirm that the 26.05% ADNI clinical-only accuracy and 9.30% IBDome clinical-only accuracy identified by the reviewer match the respective CN and non-IBD class prevalences.

We also evaluated five-fold out-of-fold logistic regression on the same 787 ADNI subjects. Adding cognitive features yielded a mean true-label log-score gain of 0.4577 (95% CI [0.4115, 0.5041]), compared with 0.5161 (95% CI [0.4724, 0.5606]) for Qwen Direct. The subject-paired difference was 0.0583 (95% CI [0.0139, 0.1020]).

These baselines provide reference points for interpreting potentially degenerate predictions, but do not establish clinical predictive competence. The IBDome analysis characterizes evidence-response behavior in the evaluated pipelines rather than validating their predictive performance or establishing a general architecture ranking.

W3. Normative reference and multimodal scope

We agree that the experiments do not identify a uniquely warranted subject-level posterior. Our operational target is observable distributional revision and its predictive consequences, rather than normative Bayesian updating.

As reported in W2, we also evaluated a five-fold out-of-fold logistic-regression baseline on the same ADNI subjects, providing a conventional predictive reference for the value of cognitive evidence.

The evaluation nevertheless goes beyond measuring distributional movement alone. The ADNI matched-neutral control preserves cognitive values and serialization while replacing clinically meaningful field names with neutral labels. The semantic-minus-neutral true-label log-score contrast is 0.212 (95% CI [0.171, 0.254]), indicating that the observed response cannot be explained by numerical values and formatting alone. This provides outcome-grounded evidence of semantic sensitivity, not the rationality of individual updates.

The arm-level analysis reveals heterogeneity. Semantic-minus-neutral mean contrasts were positive for Direct and Tool across all three ADNI backbones (0.161–0.285), and for Ledger with Qwen (0.258) and Gemma (0.232). Ministral Ledger had a slightly negative mean (−0.024), despite a positive median (0.936) and positive contrasts for 61.8% of subjects.

We verified this discrepancy against the released ADNI neutral-control artifact. Across 787 subjects, the mean was −0.023826 and the median was 0.936093, with 486 positive (61.8%), 291 negative (37.0%), and 10 zero (1.3%) contrasts. Larger losses among negatively affected subjects account for the negative mean. These results illustrate why the pooled semantic contrast should not be interpreted as a uniform benefit across pipelines.

Modality-specific controls also show heterogeneous results. The ADNI MRI contrast was inconclusive (0.006, 95% CI [−0.004, 0.017]), whereas IBDome showed positive contrasts for H&E (0.084 [0.044, 0.126]) and Olink (0.031 [0.009, 0.054]). RNA-seq and WES remained inconclusive.

The evaluated systems consume encoded modality evidence rather than uniformly processing modalities in their native forms. ADNI structural evidence originates from T1-weighted MRI, while IBDome histopathological evidence originates from H&E whole-slide images. An information-matched H&E text summary recovered 71% of the production effect, indicating that representation influences the observed response. These controls support claims about the evaluated encoded representations, not native visual understanding, and serve as diagnostics rather than independent confirmation of the main attribution findings.

The scope of our conclusions is therefore evidence revision in encoded heterogeneous-evidence pipelines, not native multimodal perception or normative clinical belief updating.

W5. Self-contained methodology and reproducibility

The released Supplementary Material includes the system prompts, JSON schemas, probability-processing rules, and execution configurations. We clarify the implementation details below.

The Evidence Ledger separates evidence assessment from probability reconciliation. Its audit stage evaluates newly acquired evidence against the previous belief and accumulated evidence using four structured fields: relation (supporting, contradicting, neutral, or unresolved), novelty (new, partly redundant, or redundant), relevance (low, medium, or high), and reliability (low, medium, or high). The audit stage does not revise probabilities.

The reconciliation stage receives the previous probability distribution, updated ledger, and structured audit. Redundant evidence is instructed to preserve the previous distribution, while low-relevance neutral evidence permits only limited revision. Unresolved evidence discourages confident changes, whereas highly relevant and reliable supporting or contradicting evidence can strengthen or revise the current diagnosis. Reconciliation considers accumulated evidence rather than only the newest observation.

Probability elicitation uses structured-output schemas specifying class probabilities over CN, MCI, and AD. The ADNI analysis implementation clips probabilities to nonnegative values and renormalizes them to sum to one, using a uniform fallback for zero-total vectors. True-label log scores apply a numerical floor of 1e−12. These processing rules are separate from the Ledger's reconciliation instructions.

Conditional replay explicitly controls prior provenance. Both Tool-origin and Ledger-origin pre-addition distributions are evaluated. Within each origin condition, both integrators receive the same frozen prior and identical cognitive evidence. Contrasts are averaged within subject before paired bootstrap aggregation.

The released prompts, schemas, configurations, and frozen subject-level outputs allow inspection of these procedures, distinguishing model-driven audit and reconciliation from deterministic preservation.

W5. Same-condition rerun drift relative to the headline gap

We quantified same-condition rerun variability using the released neutral-control repeat diagnostics, covering 200 subjects per backbone and system.

Mean absolute true-label log-score drift across Qwen, Gemma, and Ministral was 0.071, 0.101, and 0.227 for Direct; 0.106, 0.114, and 0.303 for Tool; and 0.565, 0.431, and 1.351 for Ledger, respectively. Ledger exhibited greater rerun drift than Direct or Tool on every backbone, with the largest variability for Ministral Ledger.

These absolute same-condition differences cannot be directly compared with the signed −0.103 autonomous–conditional gap or treated as its uncertainty interval. The headline confidence interval is a subject-level bootstrap interval conditional on frozen inference outputs and does not incorporate inference-run uncertainty. The repeat diagnostics quantify this additional source of variability but do not provide a joint uncertainty estimate across inference runs.

We also performed three full-cohort ADNI autonomous and conditional executions across all 787 subjects per backbone. Averaged across executions, the backbone-averaged gap was −0.1338 (subject-paired bootstrap 95% CI [−0.1900, −0.0779]), compared with the original −0.1031 (95% CI [−0.1623, −0.0440]). Both intervals lie entirely outside the ±0.020 equivalence region, preserving the nonpersistence classification under the evaluated configurations. These intervals reflect subject-level uncertainty conditional on their respective inference outputs, rather than joint uncertainty across independent runs.

W5. Equivalence-margin sensitivity and manuscript terminology

The original equivalence margins were ±0.020 for ADNI and ±0.025 for IBDome. We evaluated the persistence classifications across a wider range of margins to assess their sensitivity to these choices.

The autonomous–conditional gaps were −0.1031 (95% CI [−0.1623, −0.0440]) for ADNI and +0.1172 (95% CI [0.0885, 0.1443]) for IBDome. Both remain materially nonpersistent for margins from ±0.005 through ±0.040. Applying a common margin of either ±0.020 or ±0.025 to both cohorts also preserves the nonpersistence classification, showing that the conclusions do not depend on dataset-specific margin choices.

We also examined backbone-composition sensitivity. On ADNI, excluding Qwen or Gemma preserves material nonpersistence, whereas excluding Ministral yields an inconclusive result (−0.0392, 95% CI [−0.0877, 0.0087]). On IBDome, material nonpersistence is preserved under every single-backbone exclusion. Thus, the full-panel findings are robust to margin choices, while the ADNI result remains sensitive to backbone composition.

Manuscript terminology and self-containedness. We acknowledge the concern that terms such as “the original paper,” “legacy,” and “historical outputs” may suggest dependence on an unspecified prior publication. These refer to earlier experimental outputs within the present study rather than an external publication. The system definitions, evaluation procedures, and released artifacts are part of the submitted work, with implementation and reproduction details provided in the Supplementary Material.

Central contribution

We agree that input matching and component ablation are standard experimental controls. Our contribution is not these operations individually, but their use as attribution tests for evidence-revising agents.

Autonomous intervention responses alone cannot distinguish conditional integration behavior from differences in acquisition, stopping, evidence history, or deterministic post-processing. Matched-prior replay tests whether the autonomous contrast persists under identical priors and added evidence. On ADNI, the Ledger–Tool contrast changes from −0.0913 to +0.0118, yielding a gap of −0.1031 (95% CI [−0.1623, −0.0440]). The corresponding IBDome gap is +0.1172 (95% CI [0.0885, 0.1443]). These results demonstrate that whole-system contrasts cannot automatically be attributed to conditional evidence integration.

The duplicate component control provides a complementary attribution test. Removing the preservation guard increases median duplicate TV from 0.000 to 0.097, while guard-only processing reproduces the full Ledger's stability within the prespecified ±0.003 equivalence margin.

The central finding is therefore not that matching inputs changes an estimate, but that observed agent behaviors can fail the attribution tests needed to support claims about their underlying mechanisms. These tests identify attribution failures without requiring a complete causal decomposition or presupposing a superior architecture.

Reviewer_CZWUOpenReview ↗

Official Comment by Reviewer CZWU

Thank you for the clarifications. I have no further questions and will maintain my original rating.

Reviewer_8SRzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how multimodal AI agents revise their predictions when new evidence is introduced, contradicted, repeated, or removed. The authors propose a counterfactual evaluation framework that distinguishes fixed-input responses, autonomous whole-system behavior, conditional evidence integration, and deterministic post-processing. The central contribution is a contrast-persistence test intended to determine whether differences observed between autonomous agents can genuinely be attributed to their evidence integration mechanisms.

The framework is evaluated primarily on ADNI, with IBDome serving as a cross-domain validation dataset. The authors compare Direct prediction, Tool Agent, and Evidence Ledger architectures across multiple language model backbones. The results show that differences between autonomous systems can diminish substantially under matched-input conditional replay. The paper also demonstrates that the Evidence Ledger’s apparent robustness to duplicated evidence can be reproduced by a deterministic preservation guard.

Critical strengths

The paper studies an important problem that is often overlooked in agent evaluation. Final-answer accuracy alone does not reveal whether an agent responds appropriately to changes in available evidence. The distinction between evidence sensitivity and evidence integration is meaningful, particularly for agents operating in medical and other high-stakes environments.

I appreciate the effort to distinguish autonomous system behavior from conditional integration under controlled inputs. The matched-input replay experiments provide useful evidence that performance differences between complete agent pipelines should not automatically be attributed to their integration mechanisms.

The experimental analysis is relatively careful. The authors report subject-level paired confidence intervals, evaluate several model backbones, and include an additional dataset to examine whether the main observations extend beyond ADNI. The component-level experiment involving the deterministic preservation guard is also useful for identifying the source of the Evidence Ledger’s duplicate stability.

Critical weaknesses

My primary concern is that the methodological novelty appears somewhat limited. The paper explicitly acknowledges that input perturbation, conditional replay, and component ablation are established evaluation techniques. The proposed contribution is therefore mainly the contrast-persistence criterion connecting these existing methods. While this is a reasonable evaluation perspective, I am not convinced that combining these controls into an attribution rule constitutes a sufficiently substantial methodological advance. The underlying principle that end-to-end system differences cannot be attributed to a specific component without controlling for other components is already well established in experimental methodology. The paper would benefit from demonstrating what new conclusions become possible specifically because of the proposed framework, rather than through conventional controlled comparisons and ablations.

I am also concerned about the interpretation of contrast persistence as a general attribution criterion. The autonomous and conditional experiments do not estimate the same quantity. The former includes evidence acquisition, ordering, stopping decisions, and context construction, whereas the latter deliberately controls the prior state and newly introduced evidence. Consequently, observing different results between these two settings is not particularly surprising. The authors appropriately acknowledge that conditional replay does not identify a causal mediation effect or the contribution of individual upstream components. However, this limitation also weakens the central methodological claim. The experiments establish that two different evaluation settings produce different outcomes, but they do not explain which mechanisms account for these differences. I would like to see a more systematic decomposition of acquisition, ordering, context construction, and integration to demonstrate the additional explanatory value of the framework.

The empirical results also raise questions about the strength and consistency of the main conclusions. In Table 3, the ADNI finite-panel contrast-persistence gap is statistically distinguishable from zero, but the backbone-specific results are inconclusive for Qwen and Gemma. The strongest evidence of nonpersistence comes from Ministral, which has a substantially larger gap than the other two models. Although the authors use equal backbone weighting and correctly report this heterogeneity, the generality of the ADNI result remains uncertain. IBDome provides stronger evidence across three backbones, but the GPT result is again inconclusive. These findings suggest that the proposed phenomenon may depend substantially on model characteristics, dataset properties, and pipeline implementation.

Another limitation is that the evidence interventions are relatively narrow compared with the broader motivation of multimodal evidence reasoning. On ADNI, the primary addition experiment introduces MMSE, ADAS11, and ADAS13, which are highly informative and directly related to the diagnostic labels. The positive predictive consequence of adding these variables is therefore expected. The contradiction experiment modifies APOE4, which is a probabilistic risk factor rather than a logically contradictory observation. The redundancy experiment duplicates exactly the same field, while the removal experiment reverses the cognitive addition. These interventions are useful controlled tests, but they do not adequately capture more challenging situations involving conflicting information from independent sources, uncertain evidence quality, misleading observations, or complex cross-modal interactions.

The paper’s claims about multimodal reasoning are further limited by its use of encoded evidence representations. Although ADNI includes structural MRI and IBDome includes histopathology and molecular modalities, the experiments evaluate evidence supplied through structured representations rather than directly assessing how multimodal models interpret raw images or integrate heterogeneous signals. It is therefore unclear whether the observed findings generalize to native multimodal agents that must perform perception, feature extraction, uncertainty estimation, and reasoning jointly. The distinction matters because errors introduced during perception may interact with evidence revision in ways that are not captured by the current evaluation.

I also find the Evidence Ledger comparison somewhat difficult to interpret. The Ledger introduces additional auditing, reconciliation, context organization, and deterministic guardrails relative to the Tool Agent. As a result, the architectures differ along several dimensions simultaneously. The conditional replay controls the prior and newly added evidence, but it does not isolate the contribution of individual Ledger mechanisms. The duplicate experiment convincingly demonstrates that exact-duplicate stability is imposed by the deterministic guard. However, this result is largely expected from the implementation, since the guard is explicitly designed to preserve predictions under exact redundancy. I am not convinced that reproducing this behavior with a guard-only configuration provides a sufficiently surprising or generalizable empirical finding.

Finally, the framework does not provide a clear operational definition of when an agent should rationally change its prediction. Total variation measures how much the predictive distribution changes, while log score and Brier score measure realized predictive consequences. Neither establishes whether an individual revision was justified by the available evidence. The authors acknowledge this limitation, but it creates a gap between the motivating question and the actual evaluation. A system could make a reasonable evidence-based update and still receive a worse realized score, or make an unjustified update that happens to improve the score. Without a normative reference, calibrated probabilistic model, or expert assessment of evidence-supported revisions, the proposed framework primarily measures behavioral sensitivity and attribution rather than whether the agent changes its mind appropriately.

Questions for authors

I would appreciate a clearer explanation of what makes contrast persistence fundamentally different from standard controlled ablation and replay-based evaluation. Can the authors provide an example where conventional component controls would lead to an incorrect conclusion, but the proposed criterion would identify a problem that those controls could not?

Regarding the attribution analysis, can the authors further decompose the autonomous–conditional gap into contributions from evidence acquisition, stopping behavior, ordering, and context construction? Since nonpersistence does not identify which upstream mechanism causes the discrepancy, it would be helpful to understand what actionable conclusions can be drawn from the test.

3 comments

We thank the reviewer for the careful assessment and for recognizing the importance of distinguishing evidence sensitivity from evidence integration. We address the concerns regarding methodological novelty, attribution, and the scope of our conclusions below.

W1 & Q1. Methodological novelty and distinction from conventional ablation/replay

We agree that perturbation, replay, and component ablation are established techniques. Our contribution is an explicit attribution-falsification criterion connecting these controls. A whole-system difference should not be attributed directly to conditional evidence integration when the corresponding contrast changes materially under matched input and prior. Likewise, robustness should not be attributed to model reasoning when deterministic post-processing alone reproduces it.

On ADNI, the autonomous Ledger–Tool addition contrast is −0.091, whereas the matched-prior conditional contrast is +0.012, yielding a paired gap of −0.103 (95% CI [−0.162, −0.044]). An autonomous comparison alone might suggest superior Tool evidence integration from its 0.091 larger log-score gain. Under matched prior and evidence, however, the conditional contrast instead favors Ledger by 0.012. This discrepancy prevents attributing the whole-system advantage directly to conditional integration without identifying which upstream mechanism accounts for the gap.

Conventional replay can reveal these contrasts, but reporting them separately does not specify whether an integration-specific attribution is supported. Contrast persistence provides an explicit decision criterion based on the paired gap, its confidence interval, and a prespecified practical equivalence region.

The contribution is therefore a testable rule for evaluating component-level interpretations of whole-system results, rather than a new control operation or a complete causal decomposition.

W2 & Q2. Decomposition of the autonomous–conditional gap

We analyzed the released ADNI autonomous trajectories to characterize modality acquisition, ordering, and stopping behavior.

Both Tool and Ledger acquired cognitive evidence for all 787 subjects on every backbone. Restricting the comparison to subjects where both systems acquired cognition therefore retains the full cohort and leaves the contrasts unchanged. The gap cannot be explained simply by differential cognitive non-acquisition.

Other acquisition behaviors differed substantially. Genomic acquisition rates for Tool versus Ledger were 76.5% versus 0.0% (Qwen), 59.7% versus 2.9% (Gemma), and 97.2% versus 0.1% (Ministral). MRI acquisition and ordering also differed. For example, 37.2% of Ministral Tool trajectories acquired genomic evidence before MRI, whereas 89.6% of Ministral Ledger trajectories acquired MRI after cognition and then stopped.

Stopping behavior also differed. Explicit STOP rates, defined as the fraction of trajectories containing a recorded STOP event, were 23.5%, 40.3%, and 8.8% for Tool, compared with 100.0%, 97.1%, and 99.9% for Ledger, respectively, across Qwen, Gemma, and Ministral. Tool also made more calls on average across all three backbones.

These results document concrete differences in acquisition and stopping behavior but do not identify their individual causal contributions to the gap. Acquisition decisions, stopping, evidence history, and context construction were not independently controlled. Matched-prior replay fixes the pre-addition distribution and added cognitive evidence without independently manipulating the preceding acquisition history or context.

The analysis therefore rules out differential cognitive non-acquisition as an explanation for the gap while identifying other upstream differences that may contribute to it. It supports the distinction between whole-system behavior and conditional integration without claiming a complete causal decomposition.

W3. Backbone heterogeneity and generality of nonpersistence

The ADNI backbone-specific gaps are −0.023 (95% CI [−0.073, 0.027]) for Qwen, −0.055 ([−0.114, 0.005]) for Gemma, and −0.231 ([−0.345, −0.119]) for Ministral. Only Ministral is individually classified as materially nonpersistent.

The finite-panel gap is −0.103 (95% CI [−0.162, −0.044]), calculated using equal backbone weights within each subject before bootstrapping. Thus, the panel result does not reflect unequal weighting, although Ministral contributes the largest effect.

On IBDome, the finite-panel gap is +0.117 (95% CI [0.089, 0.144]). Qwen, Gemma, and Ministral are materially nonpersistent, while GPT remains inconclusive.

We also performed leave-one-backbone-out analyses. On ADNI, material nonpersistence persists after excluding Qwen (−0.1430, 95% CI [−0.2167, −0.0702]) or Gemma (−0.1270 [−0.1966, −0.0564]). Excluding Ministral yields an inconclusive result (−0.0392 [−0.0877, 0.0087]), confirming its substantial contribution to the panel finding.

On IBDome, material nonpersistence persists under every single-backbone exclusion, with gaps ranging from 0.1019 to 0.1451. Thus, ADNI is particularly sensitive to Ministral, whereas the IBDome panel result does not depend on any single tested backbone.

These findings establish nonpersistence for the evaluated finite panels, not a universal effect across backbones.

W4. Limited scope of evidence interventions

The four interventions test distinct response properties rather than exhaustively representing real-world evidence conflicts. APOE4 replacement is a probabilistic countervailing intervention, not a logical contradiction; exact duplication tests repeated-field stability, not independent-source agreement; and cognitive removal is a dependent consistency check on addition.

The matched-neutral ADNI control provides outcome-grounded evidence. Adding diagnosis-relevant cognitive fields yields a semantic-minus-neutral true-label log-score contrast of 0.212 (95% CI [0.171, 0.254]), indicating that the predictive effect cannot be explained solely by similarly formatted numerical information.

Independent-source conflicts, uncertain reliability, and misleading observations remain outside the tested intervention family. Our conclusions concern the specific controlled evidence-response properties evaluated here, rather than a general guarantee of appropriate responses to arbitrary conflicting evidence.

W5. Encoded evidence versus native multimodal reasoning

Our experiments evaluate evidence revision over encoded representations derived from heterogeneous biomedical modalities, rather than native perception of raw images. ADNI structural evidence originates from T1-weighted MRI, while IBDome histopathological evidence originates from H&E whole-slide images. The evaluated pipelines do not uniformly process modalities in native form across backbones.

The representation controls show modality-specific differences. ADNI MRI has an inconclusive production-versus-shuffled contrast (0.006, 95% CI [−0.004, 0.017]), whereas IBDome H&E (0.084 [0.044, 0.126]) and Olink (0.031 [0.009, 0.054]) show positive production-versus-control log-score contrasts.

An information-matched H&E text summary recovered 71% of the production effect, indicating that the observed contribution depends on the evaluated evidence representation rather than establishing native visual understanding. These controls are diagnostic and are not treated as independent confirmation of the main attribution findings.

Our conclusions therefore concern evidence revision and attribution in encoded heterogeneous-evidence pipelines, not native multimodal perception.

W6. Interpretation of the Evidence Ledger and deterministic preservation guard

The Ledger combines auditing, reconciliation, context organization, and deterministic guardrails. It is therefore treated as an instrumented diagnostic system, not as an architecture whose individual mechanisms are isolated by conditional replay.

The duplicate-preservation experiment tests whether observed invariance requires the reasoning scaffold. Removing the guard increases median duplicate-induced TV from 0.000 to 0.097, with a mean no-guard-minus-full difference of 0.122 (95% CI [0.116, 0.128]). Guard-only processing restores median TV to 0.000, with a mean difference from the full Ledger of −0.0004 (95% CI [−0.0005, −0.0002]), within the prespecified ±0.003 equivalence margin.

The finding is not that a deterministic guard behaves unexpectedly, but that the observed duplicate stability can be reproduced without the full model-driven reasoning scaffold. This supports a component-level attribution to deterministic preservation without implying general Ledger superiority or ruling out contributions from model reasoning in other settings.

W7. Normative definition of appropriate evidence revision

“Changing its mind” refers to changes in the agent's reported predictive distribution, not an inferred internal belief. Total variation measures distributional movement, while log score and Brier score measure realized predictive consequences. Neither establishes a uniquely justified posterior for an individual subject.

The matched-neutral ADNI experiment provides a narrower outcome-grounded result. The semantic-minus-neutral true-label log-score contrast is 0.212 (95% CI [0.171, 0.254]), supporting greater average predictive benefit from diagnosis-relevant cognitive evidence than from a matched-format neutral edit.

The framework therefore evaluates observable evidence responses, predictive consequences, and attribution validity rather than prescribing a normative theory of rational belief revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.