- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a framework for attributing behavior in LLM-based evidence-reasoning pipelines and applies it to two biomedical classification tasks (ADNI, 787 subjects; IBDome, 172 subjects) using three author-built systems (Direct, Tool, Ledger) across three or four backbones. Four objects are distinguished: fixed-input response, autonomous whole-system response, matched-input and matched-prior conditional replay, and component-controlled behavior. The central test asks whether the autonomous contrast between two systems persists when input and prior are held fixed. On ADNI the headline Ledger-vs-Tool contrast moves from -0.091 to 0.012 under conditional replay (gap -0.103, 95% CI [-0.162, -0.044]), reversing sign; on IBDome the gap is 0.117. The paper also argues that the Ledger's duplicate stability comes from a deterministic preservation guard rather than reasoning, and treats the Ledger as an instrumented control.
Critical strengths
The paper is unusually honest about what it does not show: it disclaims causal mediation, clinical validity, and native multimodal understanding, and reports a sign flip in its own headline result. The paired design is sound in construction, with subject-level pairing, equal-weight within-subject backbone averaging, 10,000-replicate subject bootstrap, and explicit exclusion accounting. Most reported arithmetic (Table 3 gaps, finite-panel averages, Eq. 5 cells, cohort counts) recomputes exactly.
Critical weaknesses
- The endpoint table contradicts the paper's own fixed-order table: Direct is defined as receiving all allowed evidence, yet Tables 2/4 report Qwen 0.5566 and Gemma 0.6058, which are their +Genomic values in Table 6 rather than the full-evidence +MRI values (0.5551 and 0.5943), and the text itself describes 0.5566 as the post-genomics number.
- The "no common winner across backbones" claim rests on these values, so this needs reconciliation.
- There are no baselines, not even a constant classifier, and several reported cells coincide exactly with constant-classifier values (for example, Ministral clinical equals all-non-IBD at 16/172).
- A substantial part of the evidence-response analysis therefore operates on degenerate predictors, which makes the reported effect sizes hard to interpret.
- Several quantities central to the argument are never tabulated, including the neutral-control arm behind the 0.212 result, the per-backbone duplicate reductions, and the countervailing true-label log-scores.
- The systems, prompts and guard thresholds are not described and the pipelines are attributed to an uncited "original paper", so the submission is neither self-contained nor reproducible.
Questions for authors
- Please reconcile the Direct values in Tables 2/4 with the +Genomic and +MRI rows of Table 6, and state which evidence condition Direct actually uses.
- Can you add a constant/majority baseline and a simple tabular model on the identical encoded evidence, so the reported contrasts have a reference point?
- Please quantify run-to-run variability (number of repeats, decoding settings) and report whether the headline confidence intervals survive it.
- Please justify the equivalence margins (m = 0.020/0.025 and +/-0.003) and show the persistence decisions as a function of m.
4 comments
W3. Neutral-control, duplicate-ablation, and countervailing results
We report the intervention- and backbone-level estimates underlying these results.
Matched-neutral control. The ADNI semantic-minus-neutral true-label log-score contrast is 0.212 (95% CI [0.171, 0.254]), using subject-first, equal-weight aggregation across systems and backbones. The intervention preserves cognitive values and serialization while replacing clinically meaningful field names with neutral labels, separating the contribution of field semantics from that of the numerical values and format.
The backbone- and system-level contrasts are:
| Backbone | Direct | Tool | Ledger |
|---|---|---|---|
| Qwen | 0.282 | 0.285 | 0.258 |
| Gemma | 0.263 | 0.280 | 0.232 |
| Ministral | 0.161 | 0.173 | −0.024 |
The positive panel-level contrast is supported by the Direct and Tool responses, with variation across systems and backbones.
Duplicate-preservation ablation. Removing the Ledger's duplicate-preservation guard increases median duplicate-induced TV from 0.000 to 0.097. The no-guard-minus-full mean TV differences are +0.0045 for Qwen, +0.1022 for Gemma, and +0.2584 for Ministral (787 subjects per backbone), with a pooled mean of +0.1217 (95% CI [0.1155, 0.1280]).
Guard-only processing restores median TV to 0.000, with a guard-only-minus-full mean difference of −0.000368 (95% CI [−0.000517, −0.000237]). These component-controlled results show that deterministic preservation can reproduce the observed duplicate stability without requiring model-level reasoning, supporting the distinction between whole-pipeline behavior and model-level evidence integration.
Countervailing evidence. Under the APOE4 intervention, the signed changes in AD probability are 0.257 (Direct), 0.190 (Tool), and 0.099 (Ledger). The corresponding true-label log-score changes are −0.154, −0.126, and 0.019, respectively. Thus, probability changes in the expected direction do not necessarily improve predictive performance.
W4. System specifications and reproducibility
Direct, Tool, and Ledger are the three operational pipelines evaluated in this study, with distinct evidence-processing procedures:
- Direct predicts from supplied evidence without autonomous tool acquisition.
- Tool autonomously acquires evidence before producing its predictive distribution.
- Ledger adds explicit evidence-audit and reconciliation procedures to the acquisition-and-update process.
The released supplementary artifacts include prompts, structured-output schemas, model configurations, Ledger audit and reconciliation specifications, duplicate-preservation guard procedures, probability-processing policies, and frozen subject-level predictions.
The Evidence Ledger audit records four structured fields: Relation (supporting, contradicting, neutral, or unresolved), Novelty (new, partly redundant, or redundant), Relevance (low, medium, or high), and Reliability (low, medium, or high). The reconciliation stage updates the predictive distribution using the accumulated ledger and previous distribution, preserving predictions for redundant evidence and limiting revisions for low-information evidence.
Decision thresholds and implementation provenance. We inspected the original counterfactual implementations and recovery/resume code to verify how evidence-processing decisions are operationalized. Duplicate-handling branches use explicit existence checks (e.g., len(duplicate_rows) > 0) and categorical evidence judgments (e.g., profile["novelty"] == "redundant"), rather than a universal numerical relevance or reliability cutoff.
The Tool controller distinguishes admissible from selected actions through allow_stop and explicit STOP branches. Ledger processing records reconciliation outcomes including accept_candidate, redundant_freeze, and stop. The ADNI execution traces contain 1,066, 1,284, and 1,309 accept_candidate events for Qwen, Gemma, and Ministral, respectively, and 185 redundant_freeze events for Ministral. The trajectories retain decision-basis and reconciliation-rule fields, enabling inspection of individual acquisition and reconciliation decisions.
For ADNI, structured outputs specify probability vectors over CN, MCI, and AD. The analysis implementation normalizes nonnegative probabilities to sum to one, uses a uniform fallback for zero-total vectors, and applies a numerical floor of 1e-12 when computing true-label log scores. Recorded inference configurations specify do_sample=False for all three open-weight backbones, so temperature and top-p do not operate as sampling parameters.
The frozen subject-level predictions and analysis code support regeneration of the reported aggregate results and subject-paired bootstrap estimates. The accompanying prompts, schemas, guard procedures, and configurations document the implemented inference process independently of the experiments' development history.
Q3. Run-to-run variability and headline robustness
We assessed inference variability using same-condition neutral-control diagnostics and repeated ADNI counterfactual evaluations.
The neutral-control diagnostics covered 200 subjects per backbone and system. Mean absolute true-label log-score drift across Qwen, Gemma, and Ministral was 0.071, 0.101, and 0.227 for Direct; 0.106, 0.114, and 0.303 for Tool; and 0.565, 0.431, and 1.351 for Ledger, respectively. These results show that output-level variability can arise even with deterministic decoding (do_sample=False).
We also performed three full-cohort executions of the autonomous Tool/Ledger counterfactual evaluation and conditional integration replay for each ADNI backbone (787 subjects). Averaged across executions, the backbone-specific autonomous–conditional gaps were −0.1284 (Qwen), −0.0595 (Gemma), and −0.2134 (Ministral), yielding a backbone-averaged gap of −0.1338 (subject-paired bootstrap 95% CI [−0.1900, −0.0779]).
The original ADNI gap was −0.1031 (95% CI [−0.1623, −0.0440]). Both the original and re-executed confidence intervals lie entirely outside the ADNI equivalence region of ±0.020, preserving the nonpersistence classification under the evaluated configurations.
These confidence intervals capture subject-level uncertainty conditional on the evaluated outputs, while the additional executions assess stability under the recorded inference configuration.
Q4. Equivalence-margin rationale and sensitivity
The margins of ±0.020 (ADNI) and ±0.025 (IBDome) are operational tolerances for assessing whether autonomous and conditional log-score contrasts are sufficiently close to be considered persistent. They are not clinically calibrated minimum important differences, nor are these specific values uniquely determined by the data.
We recomputed the persistence classifications across alternative margins. Both ADNI and IBDome remain materially nonpersistent for margins from ±0.005 through ±0.040. At ±0.050, ADNI becomes inconclusive, while IBDome remains materially nonpersistent. Applying a common margin of either ±0.020 or ±0.025 to both cohorts also preserves the nonpersistence classification, showing that the conclusion does not depend on cohort-specific margin choices.
The tighter ±0.003 margin addresses a separate comparison of guard-only and full duplicate handling. The guard-only-minus-full mean TV difference is −0.000368 (95% CI [−0.000517, −0.000237]). Equivalence is supported at both the original ±0.003 margin and the stricter ±0.001 margin, while the result becomes inconclusive at ±0.0005.
These sensitivity analyses show that the nonpersistence conclusions are robust across a substantial range of margins, while the guard-attribution result supports equivalence under a tighter tolerance.
We hope these additional analyses address the reviewer's main concerns regarding numerical consistency, predictive baselines, reproducibility, and robustness. The corrected endpoints preserve the backbone-level rankings, while the full-cohort ADNI re-execution and component-controlled analyses provide further support for the paper's central findings.
Thanks for reconciling the Direct endpoint values and for the additional analyses on run-to-run variability, baselines, and equivalence margins. I have read the response and have no further questions.
We thank the reviewer for the careful assessment and independent numerical checks. We addressed the concerns through additional verification and analyses. The corrected Direct endpoints preserve the backbone-level rankings, and repeated full-cohort ADNI evaluations support the original nonpersistence conclusion. We respond to each point below.
W1 & Q1. Direct endpoint inconsistency
We acknowledge the discrepancy between the Direct endpoint values in Tables 2/4 and the fixed-order trajectories in Table 6.
We verified the released subject-level prediction artifacts and confirmed that the reported Direct Macro-F1 values for Qwen and Gemma correspond to the +Genomic condition rather than the final +MRI condition. The verified full-evidence values are:
The discrepancy arose from the endpoint mapping, which selected
base_predictionfor Qwen and Gemma butaddition_predictionfor Ministral. This was a reporting inconsistency rather than a change in the Direct system definition, as the full-evidence endpoint corresponds to the final +MRI condition.The corrected +MRI values preserve the highest-Macro-F1 system for each backbone, with Direct leading for Qwen, Ledger for Gemma, and Tool for Ministral. The conclusion that no single architecture consistently outperforms the others across backbones therefore remains unchanged.
W2 & Q2. Predictive baselines and interpretation of evidence-response results
We computed constant-class baselines for both cohorts and a supervised tabular baseline on the same ADNI evaluation subjects.
Constant and majority-class baselines. For ADNI (787 subjects), always predicting the majority class (MCI) yields 55.53% accuracy and 0.2380 Macro-F1. For IBDome (172 subjects), always predicting Crohn disease yields 62.79% accuracy and 0.2571 Macro-F1. The empirical class-prior true-label log-score references are −0.9887 for ADNI and −0.8693 for IBDome; these characterize class-prevalence distributions rather than held-out predictive performance.
Five-fold out-of-fold logistic-regression baseline. We evaluated logistic regression on the same 787 ADNI subjects, using clinical/demographic features alone and with cognitive features added. Five-fold out-of-fold prediction ensured that each subject was evaluated by a model trained without that subject.
Adding cognitive evidence yielded a mean true-label log-score gain of 0.4577 (95% CI [0.4115, 0.5041]) for logistic regression, compared with 0.5161 (95% CI [0.4724, 0.5606]) for Qwen Direct. The subject-paired difference was 0.0583 (95% CI [0.0139, 0.1020]). Other LLM pipelines showed heterogeneous gains rather than a uniform advantage over the tabular baseline.
Predictive performance and evidence-revision attribution are distinct. The majority-class baselines contextualize endpoint performance, whereas paired interventions measure responses to evidence changes. An invariant prediction (TV = 0) is an observable response property, not evidence of successful reasoning by itself.
Hard-label concentration versus probabilistic revision. We also examined the frozen ADNI counterfactual predictions at both class-label and probability-distribution levels. With cognitive evidence added, Ledger's predicted class changed for 60.0% (Qwen), 74.7% (Gemma), and 87.4% (Ministral) of subjects, with corresponding mean TV distances of 0.3600, 0.4463, and 0.5568.
Even among subjects whose predicted class remained unchanged, mean TV distances were 0.2322, 0.4664, and 0.5556, respectively. Thus, unchanged class predictions can coexist with substantial probability-level revisions, demonstrating that hard-label concentration does not necessarily imply nonresponsiveness to evidence.